Tracking Down Bandwidth Hogs: Netflow Home Edition

Recently Comcast announced they were going to be doing some testing of data caps in certain markets, including mine.  I’m a cord-cutter and I’ve been very happy with Comcast’s speed and reliability, but when you view a lot of streaming video (Netflix, Hulu, Amazon Prime, iTunes)  and do a lot of cloud backups of your data, 300GB suddenly doesn’t seem so much.

Fortunately, Comcast has graciously decided to give its customers a grace period before they actually start charging for overages, so I’ve been trying to manage my data use a little more proactively.  Looking at previous months, I saw that I was typically using 500-600GB of data.  After cutting my streaming usage way back, it seemed that I was still using a tremendous amount of data and exceeding my cap well before the end of the month, and I couldn’t really figure out why.

It so happens that the company I work for makes a great enterprise product for finding just these sort of things, using NetFlow.  Unfortunately, we don’t (currently) produce a home edition, so I decide to try the next best thing.

The first step was to configure my router, which is running  DD-WRT firmware, to export “RFlow”.  As best I can tell, RFlow is an implementation of some version of NetFlow.  Presumably NetFlow is trademarked, so they have to call it something else.   Add to the mix nprobe and ntopng, and I was able to find that one host was using a very large percentage of the total bandwidth.

 

ntop

So the host 192.168.1.103, which is my Macbook Pro, had used over 1GB of bandwidth during the reporting period, but what really surprised me was that it wasn’t predominantly downloading data, but sending data.  This set off all kinds of alarm bells and gave me a mild panic.  I’m thinking virus, botnet, who knows what evil malware I may have gotten, despite considering myself fairly savvy.

Unfortunately, I couldn’t tell much more about the traffic other than it was SSL.  But now that I had it narrowed down to a host, I remembered a handy utility I’d discovered awhile back called Little Snitch.  It’s basically a firewall which allows you to selectively allow and deny connections from your Mac.  Being required to whitelist each application that requests access to the network gets pretty tiresome after awhile, so I had stopped using it.  But turns out, the latest version has a “passive” mode, where it will monitor what’s going on but won’t actively block anything.  I let it run for awhile and was able to collect some interesting data.

snitch

Outlook?  What the hell are you doing?  After about an hour it had sent nearly half a gigabyte of data.  I’ll grant you that my emails may be overly wordy at time, but I hadn’t actually sent any.

Turns out, Outlook has a rather nasty little bug having to do with folder syncing that makes it use a metric crap-ton of bandwidth.  There doesn’t seem to be a fix that I could find, so for now, I’m just shutting down Outlook when I’m not actively using it.

And thus ends my bandwidth hog detective story.

OSX Lion: ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory

I keep running into this.  For various reasons I need to use password-based authentication on some test boxes that are regularly rebuilt which makes using key based authentication difficult.

Ordinarily I set up an alias like this:

alias ssh1="sshpass -p secretpassword ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

… which lets me use a command like:

sshl eric@devbox

But sometimes (and I swear, it doesn’t always happen, but I haven’t figured out the variable yet), it fails, and I get:

ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory

… and indeed, the ssh-askpass binary doesn’t exist.  After some flailing around and googling, I finally came up with this ugly hack.  I create a shell script with the following contents:

#!/bin/bash
echo "secretpassword"

Then, I set the following environment variable, which directs sshpass to use my script instead of looking for the missing ssh-askpass

export SSH_ASKPASS=/Users/eric/scripts/ssh-askpass.sh

It’s ugly, but it works.  Of course, it assumes you’re always using the same “secretpassword”.  Some day I need to figure out a better way of addressing this, but today is not that day.

See also:  http://apple.stackexchange.com/questions/18238/mac-os-x-lion-and-sshpass

Unattended Installation (aka Silent Install) of Sun JDK in Debian

Posting this short note to my blog in the hopes it may help someone else, as I had a hard time tracking this info down.

Debian provides a package (sun-java6-jdk) for installing the JDK, but when you have a situation where you need to do an unattended installation (aka “silent install”) of the JDK you are stuck, because the install insists on making you interactively accept Sun’s licensing agreement. Passing the -y option to apt-get has no effect. I was able to find a link on Sun’s site for doing silent installs on Windows, but nothing for Linux.

Fortunately, I eventually tracked down, all you have to do is the following before you run your apt-get install command:

echo sun-java6-jdk shared/accepted-sun-dlj-v1-1 boolean true | debconf-set-selections

This indicates to the installer hooks that you have already accepted the license agreement, and doesn’t give you the annoying EULA dialog.

UPDATE 11/3/09: I should note that all of the above research was in preparation for setting up an automated install with FAI. Unfortunately, I later found out, that while the above works great in the regular command line, due to the nature of the environment FAI installs run in, it doesn’t work for that. However, I also discovered there is a correct way of doing it with FAI.

Use “debconf preseeding”. Create a class file e.g. debconf/DEFAULT (I used debconf/easberry-vm-std-plt-01 for testing with my host class, and populate it with the following:

sun-java6-bin   shared/accepted-sun-dlj-v1-1    boolean true
sun-java6-jdk   shared/accepted-sun-dlj-v1-1    boolean true
sun-java6-jre   shared/accepted-sun-dlj-v1-1    boolean true

This accomplishes basically the same thing, but in the FAI environment. Success!

Matching multiline regular expressions in Java

Regular expressions are one of those things that I understand but don’t use often enough to have really mastered. I use them even less in my Java programming. Today I found myself banging my head into my cubicle wall trying to parse out a file that looked something like this:
—-START
something=foo
anotherthing=bar
—-END
—-START
something=baz
anotherthing=ran outta foo words
—-END
I needed to parse this out and create objects representing each section. Initially I started trying to read this in line by line, keeping track of where I was in relation to the markers, concatenating string buffers, etc. Then I realized how retarded that approach was, and how using a regular expression would make it a lot simpler.

Brushing aside the mental cobwebs I looked up a couple of references of the Java API and studied up on my friends Pattern and Matcher. I had trouble finding any examples for my specific case, where what I wanted to match spanned multiple lines. At first I thought I’d found the answer with the promising sounding Pattern.MULTILINE argument to Pattern.compile(), but that has to do with matching ^ and $. Without that option, those operators only match at the beginning or end of the text being parsed, with them it will allow them to work within the text at newline boundaries.

Turned out what I was looking for was the Pattern.DOTALL argument. By default, the dot operator does not match newlines, with this argument it does. An alternative is to prefix the regex pattern with (?s). It has the same effect, and the mnemonic stands for “single-line” mode, which is what its called in Perl.

So, to extract the relevant sections from the input above, you can do the following:

Pattern pattern = Pattern.compile(“—-START\n(.*?)\n—-END\n”, Pattern.DOTALL);
Matcher matcher = pattern.matcher(theInput);

while(matcher.find()) {
System.out.println(matcher.group(1) + “\n”);
}

That will print out:

something=foo
anotherthing=bar

something=baz
anotherthing=ran outta foo words

You can get the same result with the alternate method, using the embedded (?s) operator in the Pattern declaration:

Pattern pattern = Pattern.compile(“(?s)—-START(.*?)—-END”);

For further reference, have a look here.