Miles per Millisecond: A Look at the WonderProxy Network

by Paul Reinheimer on

Update: This post has been updated to account for pings being round trip times while distances are only one way. The original post failed to account for this in the last two tables, thanks to Steve for pointing this out.

Update 2: You can now view updates data with ping time between cities at our new WonderNetwork site.

Running a global network of servers for GeoIP application testing leaves you with a lot of servers and some interesting questions, occasionally an interesting combination. I found myself asking if we could compare ping times to physical distances, to see how efficient the internet was, and to confirm my suspicion that transferring data between Sydney Australia and Fremont California would be faster, mile per mile, than transferring between Boston and Fremont. My reasoning was that Australia → United States is one long cable, whereas within the US it would be switched frequently, which is slower.

First we generated a script to ping every city in our network from every other city in our network (Boston → New York and New York → Boston would both be executed). This took something like ten hours (we’ve since smartened up and now execute the test in parallel). Our results look something like this:

Pings Between Cities

Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 239.38 35.64 79.68 104.70 141.15 174.71 97.71 251.07 114.11
Brisbane 238.32 221.60 174.38 350.12 357.21 245.89 335.79 33.45 346.92
Dallas 34.53 220.14 44.00 140.00 168.00 36.80 130.94 221.72 127.74
Fremont 79.39 176.07 44.57 167.85 214.59 78.43 176.22 184.91 167.16
Milan 104.84 339.54 139.86 170.73 67.08 112.17 21.61 344.70 11.17
Moscow 140.82 366.42 169.04 209.00 67.07 131.29 56.74 387.09 60.29
New York 174.61 246.87 39.46 77.49 111.60 131.34 78.33 319.33 101.03
Paris 97.75 337.47 131.79 175.19 21.29 56.13 77.31 348.84 108.06
Sydney 262.50 42.98 222.25 191.96 345.21 376.57 261.24 354.16 358.75
Zurich 102.18 346.71 127.59 173.61 11.25 60.32 100.24 108.25 345.05

This table shows us what we expected. Sydney is far away from most of our other servers, so the ping time is high. Our Fremont server is incredibly well connected with a top tier provider so it has good routes. The Baltimore → New York ping may raise your eyebrows, it certainly caught our attention. A quick look at the traceroute shows:

traceroute to newyork.wonderproxy.com (69.147.239.239), 30 hops max, 60 byte packets  
 1  173.246.103.253 (173.246.103.253)  0.551 ms  0.557 ms  0.604 ms
 2  4xe-pc400.vcore1-dc1.balt.gandi.net (173.246.96.33)  0.606 ms  0.661 ms  0.693 ms
 3  xe3-4-core4-d.paris.gandi.net (217.70.176.233)  97.342 ms  97.325 ms  97.272 ms
 4  p251-core3-d.paris.gandi.net (217.70.176.253)  123.541 ms  123.547 ms  123.528 ms
 5  linx.ge1-0.cr01.lhr01.mzima.net (195.66.225.15)  119.101 ms  119.085 ms  119.017 ms
 6  te0-5.cr1.nyc2.us.packetexchange.net (69.174.120.89)  181.881 ms  176.377 ms  176.349 ms

Our provider in Baltimore seems to be routing all of their traffic through their central datacenter in Paris, rather sub-optimal (we’ve opened a ticket).

Next we needed to determine how far apart these cities were, that being the other half of the equation. We worked with the city center when more specific information on location wasn’t available, and generated the approximate latitude and longitude for every server in our network using publicly available sources. We then used the excel calculation script from Movable Type Scripts to determine distances, and came up with this:

Distance between cities

Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 9553 1216 2448 4211 4853 171 3819 9845 4122
Brisbane 9553 8375 7138 10162 8795 9688 10349 457 10143
Dallas 1216 8375 1466 5355 5788 1376 4956 8638 5255
Fremont 2448 7138 1466 5983 5914 2564 5596 7481 5857
Milan 4211 10162 5355 5983 1429 4040 400 10348 137
Moscow 4853 8795 5788 5914 1429 4694 1554 9060 1371
New York 171 9688 1376 2564 4040 4694 3648 9993 3952
Paris 3819 10349 4956 5596 400 1554 3648 10600 304
Sydney 9845 457 8638 7481 10348 9060 9993 10600 10355
Zurich 4122 10143 5255 5857 137 1371 3952 304 10355

Finally, we merged the two tables, did some math and came up with this:

Miles Per Millisecond

  Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 79.81 68.24 61.44 80.44 68.76 1.96 78.17 78.43 72.25
Brisbane 80.17 75.59 81.87 58.05 49.24 78.80 61.64 27.33 58.48
Dallas 70.42 76.09 66.63 76.50 68.90 74.78 75.70 77.92 82.28
Fremont 61.67 81.08 65.78 71.29 55.12 65.38 63.51 80.92 70.08
Milan 80.33 59.86 76.58 70.09 42.61 72.03 37.02 60.04 24.52
Moscow 68.92 48.00 68.48 56.59 42.61 71.51 54.78 46.81 45.48
New York 1.96 78.49 69.75 66.18 72.40 71.48 93.14 62.59 78.24
Paris 78.14 61.33 75.21 63.88 37.58 55.37 94.38 60.77 5.63
Sydney 75.01 21.26 77.73 77.94 59.95 48.12 76.50 59.86 57.73
Zurich 80.68 58.51 82.37 67.47 24.36 45.46 78.85 5.62 60.02

Here things start to look a bit better for several of the connections. In the first chart which looked at raw ping times Sydney faired poorly. Here, accounting for the extreme distance between Sydney and the majority of our network we see that, mile for mile, it’s actually doing quite well. Other links like Paris → Milan that were looking quite good previously are now exposed as being rather inefficient.

While we’re examining the efficiency of our network, this is the chart we’ll use. Simple ping times tell a story (we use smoke ping to monitor consistency and packet loss), but not the whole story. From this we get a more realistic view of hour our connections are performing.

One Last Experiment

Networks are fast, but just how fast? In a vacuum Light travels roughly 186,000 miles/second, or 186 miles/millisecond. Light travels slower through fiber optics, on average about 35% slower, which gives us ~120.9 miles/millisecond. Let’s look at the speed of pings across our network, as a percentage of the theoretical maximum:

Network Speed As A Percentage Of The Speed Of Light

  Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 66.02 56.44 50.82 66.53 56.88 1.62 64.66 64.87 59.76
Brisbane 66.31 62.52 67.72 48.01 40.73 65.18 50.98 22.60 48.37
Dallas 58.25 62.93 55.12 63.27 56.99 61.85 62.61 64.45 68.05
Fremont 51.01 67.06 54.41 58.96 45.59 54.08 52.53 66.93 57.96
Milan 66.44 49.51 63.34 57.97 35.24 59.58 30.62 49.66 20.28
Moscow 57.01 39.71 56.64 46.81 35.25 59.15 45.31 38.72 37.62
New York 1.62 64.92 57.69 54.74 59.88 59.12 77.04 51.77 64.71
Paris 64.63 50.73 62.21 52.84 31.08 45.80 78.06 50.27 4.65
Sydney 62.04 17.59 64.29 64.47 49.59 39.80 63.28 49.51 47.75
Zurich 66.74 48.40 68.13 55.81 20.15 37.60 65.22 4.65 49.65

For this comparison to be fair cables would need to be run directly between cities, which is clearly not the case. We still think it’s interesting. Hitting 68% of the theoretical best speed is quite remarkable.