Miles per Millisecond: A Look at the WonderProxy Network

Update: This post has been updated to account for pings being round trip times while distances are only one way. The original post failed to account for this in the last two tables, thanks to Steve for pointing this out.

Update 2: You can now view updates data with ping time between cities at our new WonderNetwork site.

Running a global network of servers for GeoIP application testing leaves you with a lot of servers and some interesting questions, occasionally an interesting combination. I found myself asking if we could compare ping times to physical distances, to see how efficient the internet was, and to confirm my suspicion that transferring data between Sydney Australia and Fremont California would be faster, mile per mile, than transferring between Boston and Fremont. My reasoning was that Australia → United States is one long cable, whereas within the US it would be switched frequently, which is slower.

First we generated a script to ping every city in our network from every other city in our network (Boston → New York and New York → Boston would both be executed). This took something like ten hours (we’ve since smartened up and now execute the test in parallel). Our results look something like this:

Pings Between Cities

This table shows us what we expected. Sydney is far away from most of our other servers, so the ping time is high. Our Fremont server is incredibly well connected with a top tier provider so it has good routes. The Baltimore → New York ping may raise your eyebrows, it certainly caught our attention. A quick look at the traceroute shows:

traceroute to newyork.wonderproxy.com (69.147.239.239), 30 hops max, 60 byte packets
 1  173.246.103.253 (173.246.103.253)  0.551 ms  0.557 ms  0.604 ms
 2  4xe-pc400.vcore1-dc1.balt.gandi.net (173.246.96.33)  0.606 ms  0.661 ms  0.693 ms
 3  xe3-4-core4-d.paris.gandi.net (217.70.176.233)  97.342 ms  97.325 ms  97.272 ms
 4  p251-core3-d.paris.gandi.net (217.70.176.253)  123.541 ms  123.547 ms  123.528 ms
 5  linx.ge1-0.cr01.lhr01.mzima.net (195.66.225.15)  119.101 ms  119.085 ms  119.017 ms
 6  te0-5.cr1.nyc2.us.packetexchange.net (69.174.120.89)  181.881 ms  176.377 ms  176.349 ms

Our provider in Baltimore seems to be routing all of their traffic through their central datacenter in Paris, rather sub-optimal (we’ve opened a ticket).

Next we needed to determine how far apart these cities were, that being the other half of the equation. We worked with the city center when more specific information on location wasn’t available, and generated the approximate latitude and longitude for every server in our network using publicly available sources. We then used the excel calculation script from Movable Type Scripts to determine distances, and came up with this:

Distance between cities

Finally, we merged the two tables, did some math and came up with this:

Miles Per Millisecond

Here things start to look a bit better for several of the connections. In the first chart which looked at raw ping times Sydney faired poorly. Here, accounting for the extreme distance between Sydney and the majority of our network we see that, mile for mile, it’s actually doing quite well. Other links like Paris → Milan that were looking quite good previously are now exposed as being rather inefficient.

While we’re examining the efficiency of our network, this is the chart we’ll use. Simple ping times tell a story (we use smoke ping to monitor consistency and packet loss), but not the whole story. From this we get a more realistic view of hour our connections are performing.

One Last Experiment

Networks are fast, but just how fast? In a vacuum Light travels roughly 186,000 miles/second, or 186 miles/millisecond. Light travels slower through fiber optics, on average about 35% slower, which gives us ~120.9 miles/millisecond. Let’s look at the speed of pings across our network, as a percentage of the theoretical maximum:

Network Speed As A Percentage Of The Speed Of Light

For this comparison to be fair cables would need to be run directly between cities, which is clearly not the case. We still think it’s interesting. Hitting 68% of the theoretical best speed is quite remarkable.