Bandwidth ain’t everything

I maintain an Ubuntu server for my work. The machine serves up a git repo (for my private projects), a wiki, several development versions of apps in progress, and a CI system. I noticed recently that connection times seemed much slower than they should based on the cable speed of the connection. I decided to investigate.

First, I checked out my connection speed directly via speedtest.net. The recorded speed looked good (~20Mbps in both directions) though watching it in the browser made the numbers seem wrong. The page rendering was slow and clunky.

Next I checked a traceroute to anywhere. Each hop was quick (10s of ms) though the command itself took many seconds:

% time traceroute www.ucsd.edu
traceroute to www.ucsd.edu (132.239.180.101), 30 hops max, 60 byte packets
 1  wp.comcast.net (10.1.10.1)  0.518 ms  1.619 ms  1.950 ms
 2  73.69.138.1 (73.69.138.1)  13.784 ms  13.773 ms  31.279 ms
 3  te-4-3-ur01.sffolsom.ca.sfba.comcast.net (68.85.100.9)  12.753 ms  16.303 ms  16.777 ms
 4  te-1-2-0-0-ar01.sfsutro.ca.sfba.comcast.net (68.86.143.50)  20.858 ms  20.849 ms  20.838 ms
 ...
17  ucsd.edu (132.239.180.101)  38.968 ms  39.270 ms  39.260 ms

real	1m30.760s
user	0m0.000s
sys	0m0.004s

How could 17hops each at ~30ms or less take 1.5 minutes to execute? I started to do a little reading and came across this article. The basic message is that your ISP DNS servers may not be the best way to go. This reminded me that we recently upgraded to a static IP and at that time, we updated the nameserver for that machine to use Comcast’s recommended DNS.

I looked into checking out that connection with dig.

% time dig +trace www.ucsd.edu

; <<>> DiG 9.7.3 <<>> +trace www.ucsd.edu
;; global options: +cmd
.			43734	IN	NS	k.root-servers.net.
.			43734	IN	NS	l.root-servers.net.
.			43734	IN	NS	m.root-servers.net.
.			43734	IN	NS	a.root-servers.net.
.			43734	IN	NS	b.root-servers.net.
.			43734	IN	NS	c.root-servers.net.
.			43734	IN	NS	d.root-servers.net.
.			43734	IN	NS	e.root-servers.net.
.			43734	IN	NS	f.root-servers.net.
.			43734	IN	NS	g.root-servers.net.
.			43734	IN	NS	h.root-servers.net.
.			43734	IN	NS	i.root-servers.net.
.			43734	IN	NS	j.root-servers.net.
;; Received 512 bytes from 75.75.75.75#53(75.75.75.75) in 13 ms

edu.			172800	IN	NS	f.edu-servers.net.
edu.			172800	IN	NS	d.edu-servers.net.
edu.			172800	IN	NS	g.edu-servers.net.
edu.			172800	IN	NS	c.edu-servers.net.
edu.			172800	IN	NS	a.edu-servers.net.
edu.			172800	IN	NS	l.edu-servers.net.
;; Received 265 bytes from 192.112.36.4#53(g.root-servers.net) in 10030 ms

ucsd.edu.		172800	IN	NS	ns1.ucsd.edu.
ucsd.edu.		172800	IN	NS	ns2.ucsd.edu.
ucsd.edu.		172800	IN	NS	ns0.ucsd.edu.
;; Received 160 bytes from 192.5.6.30#53(a.edu-servers.net) in 10035 ms

www.ucsd.edu.		43200	IN	CNAME	www.dr-link.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	lucifer.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	ns-ucop-alt.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	act-wcs2-old2.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	devilbunny.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	game.ucsd.edu.
;; Received 256 bytes from 128.54.16.2#53(ns1.ucsd.edu) in 10055 ms

real	0m46.218s
user	0m0.008s
sys	0m0.000s

Wow. Check out the highlighted lines and notice that in 3 cases it took 10+ *seconds* to get back an answer from the nameservers. This is unacceptable. I immediately switched my nameservers to my OpenDNS nameservers. What a huge improvement!

% time dig +trace www.ucsd.edu

; <<>> DiG 9.7.3 <<>> +trace www.ucsd.edu
;; global options: +cmd
... SNIPPPED FOR BREVITY ... 
edu.			172800	IN	NS	c.edu-servers.net.
edu.			172800	IN	NS	d.edu-servers.net.
edu.			172800	IN	NS	g.edu-servers.net.
edu.			172800	IN	NS	l.edu-servers.net.
edu.			172800	IN	NS	f.edu-servers.net.
edu.			172800	IN	NS	a.edu-servers.net.
;; Received 265 bytes from 192.112.36.4#53(g.root-servers.net) in 68 ms

ucsd.edu.		172800	IN	NS	ns1.ucsd.edu.
ucsd.edu.		172800	IN	NS	ns2.ucsd.edu.
ucsd.edu.		172800	IN	NS	ns0.ucsd.edu.
;; Received 160 bytes from 192.41.162.30#53(l.edu-servers.net) in 85 ms

www.ucsd.edu.		43200	IN	CNAME	www.dr-link.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	game.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	ns-ucop-alt.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	devilbunny.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	act-wcs2-old2.ucsd.edu.
dr-link.ucsd.edu.	43200	IN	NS	lucifer.ucsd.edu.
;; Received 256 bytes from 128.54.16.2#53(ns1.ucsd.edu) in 33 ms


real	0m0.270s
user	0m0.004s
sys	0m0.000s

With this very simple change, the server connection now feels like it should. If you find you’re having issues with network connectivity that doesn’t seem to agree with your expected connection speed, look into your DNS servers. You may be able to improve your overall network performance with this very simple change.

Note: Though I haven’t had any issues with OpenDNS going down, it’s a good idea to keep your ISP’s nameserver on record since you are paying for it and it will work as a backup. If you want to try out a switch, you can check out Google’s public DNS. I found that it gave comparable results to OpenDNS.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s