Tier 1 carrier performance: May, 2017 report

The presented analysis is based on more than 636 million successful probes that span the entire month. All data is aggregated per carrier on a daily basis and accounts for many thousands of successful probes. A control group (labeled C) is used as a base of comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers.

Fig. 1. Average Loss and Latency in May 2017
The numbers include a control group C (gray) to allow cross comparison.

The values for April 2017 are included for cross comparison.

Fig. 2. Average Loss and Latency in April 2017
The charts include a control group C (gray) to allow cross comparison.

Average packet loss analysis:

• NTT and Level 3 have obtained the leading positions in May, close followed by Zayo. If based on the control group level, better result shows Centurylink as well. Cogent, GTT, Telia, XO and Hurricane Electric have registered higher level of average packet loss than the control group level;
• All Tier 1s maintained relatively the same average packet loss as in April. Significant differences have not been registered.

Average latency analysis:

• In May, Centurylink, Level 3 and NTT have registered lower average latency than the control group level value;
• Hurricane Electric showed the worst results, followed by Cogent, GTT, XO, Zayo and Telia;
• A significant change in average latency for the month of May, compared to April, hasn’t been observed.

The charts below illustrate the performance of each carrier in comparison to the control group.

Fig. 3. Better or worse Loss and Latency in May 2017
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in April 2017
The numbers are differences from average control group.

In comparison with the control group for the month of May, 2017:

• NTT, Level 3, Zayo and Centurylink showed better results in terms of Loss. NTT’s registered average packet loss was less than the control group level by 0.8%, Level 3 by 0.5%, Zayo by 0.3% and Centurylink by 0.1% respectively;
• Centurylink presented better results in terms of Latency than the control group level by 13 ms, NTT by 7ms and Level 3 by 5 ms.

In comparison with the data from April, 2017:

• All Tier 1s show similar trends as in April for both packet loss and latency statistics;
• NTT, Level 3, Zayo and Centurylink maintained their leader positions in term of loss. Telia keeps showing the poorest results followed by Hurricane Electric;
• Centurylink, NTT and Level 3 have slightly improved their results in term of latency. Hurricane Electric has substantially worsened its average latency result.

Loss

For the Loss analysis we use a scatter plot, where average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All data-points below the diagonal represent the better performing carriers and vice versa.

Abnormally large losses are still registered for a large number of datapoints. We consider excessive an average above 4.5% packet loss. Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions are deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption. More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier (May 2017)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier (April 2017)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

NTT, XO and Zayo show different tendency of average packet loss (either gain or loss) when comparing the results determined for ALL data-points to the results with the applied cut off at 4.5% control group level. Stable results are registered for GTT (bad) and Centurylink (good). Meanwhile, XO and NTT showed better results both compared to: May’s cut off at 4.5% control group level as well as the April registered level.

As in April, the worst results showed Hurricane and Cogent, followed by Telia and GTT, both compared to: ALL registered datapoints as well as for cut off at 4.5% control group level. Zayo, left the group of better performing carriers after the data set restructuring (Fig. 7).

Latency

For Latency analysis we use a similar scatter plot to the one we used for Loss. It displays control group values on the diagonal while highlighting individual carrier measurements on the horizontal and on the vertical axis. Datapoints placed significantly and consistently below the average highlight better performing carriers while datapoints above the average highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

Based on data points presented in Fig.9 we can conclude the following:

• NTT and Level 3 remain present in a wide latency diapazon, showing lower and higher than 200 ms latency values;
• Cogent, Telia, Hurricane and Zayo have mostly been present within the latency diapazon of 80 – 200ms while XO and GTT registered latency between 100 – 180 ms;
• Centurylink was the only carrier with registered latency that didn’t exceed 120 ms.

Fig. 10. Average latency gains/losses by carrier Values averaged for the difference between carrier performance and the average group in that network.

The differences in latency above from the control group are averaged with the expectation that better or worse performance will cancel out if the differences are caused by measurement noise.

The results show that during the month of May in comparison with April:

• Level 3 moved to the positive category, improving its average latency for each packet by ~0.5 ms;
• Hurricane maintained its leading position, reducing its average RTT by ~3ms.Hurricane is followed by XO ~2 ms , Centurylink ~1.8 ms and GTT with ~1,5 ms;
• Cogent and NTT remain in the negative category showing little progress in their performances.

Appendix. Carrier Latency (highlighted)

Latency spread chart highlighting Centurylink.

Latency spread chart highlighting Cogent.

Latency spread chart highlighting GTT.

Latency spread chart highlighting Level 3.

Latency spread chart highlighting NTT.

Latency spread chart highlighting Telia.

Latency spread chart highlighting XO.

Latency spread chart highlighting Zayo.

Latency spread chart highlighting Huricane Electric.

Loss improvement/worsening highlighting Centurylink datapoints.

Loss improvement/worsening highlighting Cogent datapoints.

Loss improvement/worsening highlighting GTT datapoints.

Loss improvement/worsening highlighting Level 3 datapoints.

Loss improvement/worsening highlighting NTT datapoints.

Loss improvement/worsening highlighting Telia datapoints.

Loss improvement/worsening highlighting XO datapoints.

Loss improvement/worsening highlighting Zayo datapoints.

Loss improvement/worsening highlighting Huricane Electric.

Disclaimer*: The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the Intelligent Routing Platform Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document. For privacy protection, the exact location and number of IRP Lite instances are not provided.