Tier 1 carrier performance report: September, 2017

Disclaimer*:
The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the IRP Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document.

The report covers broadly used Tier 1 carriers performance results in the US and the general spread for the month of September 2017. For the US results, rare datapoints from distant locations e.g. Alaska have been omitted.

The presented analysis is based on more than 790 million successful probes that span the entire month. All data is aggregated per carrier on a daily basis and accounts for many thousands of successful probes. A control group (labeled C) is used as a base of comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers.

Fig. 1. Average Loss and Latency in September 2017
The charts include a control group C (gray) to allow cross comparison.

The values for August 2017 are included for cross comparison.

Fig. 2. Average Loss and Latency in August 2017
The charts include a control group C (gray) to allow cross comparison.

Average packet loss analysis:

• Based on the control group level, better results in term of loss show NTT, Level 3, XO and Zayo. In September, just as in August, the worst results of average loss have been registered for Hurricane Electric, GTT and Telia;
• Longitudinal analysis of average packet loss by Tier 1s in the last two months showed insignificant differences between the registered structures. Nevertheless, Centurylink and XO improved their results, visibly diminishing the average packet loss.

When generalizing all registered results, the following can be concluded: average packet loss in September was lower than in August for most of the analyzed Tier 1 carriers

Average latency analysis:

• As in August, higher average latency has been registered for: Hurricane Electric, Cogent, GTT, Zayo and Telia. Closely to the control group level is XO;
• Lower average latency has been registered for: Centurylink, NTT and Level 3;
• A significant change in average latency for the month of September, compared to August, has not been registered. Almost all Tier 1s showed roughly similar results to the previous month.

The charts below illustrate the performance of each carrier in comparison to the control group.

Fig. 3. Better or worse Loss and Latency in September 2017
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in August 2017
The numbers are differences from average control group.

In comparison with the packet loss average of the control group for the month of September, 2017 (2.4%) :

• NTT, Level 3 and XO have shown good results. The packet loss average of NTT and Level 3 was lower than the packet loss average of the control group by ~ 0.5%, while XO’s was lower by ~0.2%.
• Hurricane Electric and GTT average packet loss results are higher than the packet loss average of the control group by ~ 0.7% and 0.6% respectively. Telia, Centurylink and Cogent have shown better results than the two Tier 1s characterized above, however, the average packet loss results these carriers are still being higher than the control group level ones by ~0.4%, ~0.3% and 0.2% respectively.

In comparison with the control group average latency for the month of September, 2017 (123.08 ms) :

• Centurylink, NTT and Level 3 have presented good results in terms of Latency. Centurylink has registered a lower average latency than the control group’s one by ~16ms, NTT – by~10ms, Level 3 – by~9ms. XO has shown a very rougly results with control group level. The worst results have been presented by Hurricane Electric with the average latency being ~20 ms higher than the control group level’s one.

In comparison with the data from August, 2017:

• In the month of September, the structure of underperforming Tier 1s remained approximately the same asin the previous month. Nevertheless, almost all of them have shown a somewhat improvement in their performances. XO was the only carrier with the significantly improved results,, which allowed this carrier to migrate to the group of Tier 1s with the lower average loss than the control group level;
• In term of latency, NTT was the only carrier which results have somewhat improved.

 

Loss

For the Loss analysis, based on the general spread data, we use a scatter plot. Here the average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All data-points below the diagonal represent the better performing carriers and vice versa.

Abnormally large losses are still registered for a large number of datapoints. We consider excessive an average above 4.5% packet loss. Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions are deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption. More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier September 2017
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier August 2017
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

For September 2017, the graphic of “Average packet loss gain/loss by carrier” (Fig. 7), based on ALL data-points, shows that the worst positions were held by Telia, GTT, NTT, Zayo and Cogent, while the better positions were obtained by Centurylink, Hurricane Electric and Level 3. After the cut-off of 4.5% has been applied to the control group level, the structure of Tier 1s has changed, with XO migrating to the winners group.

As in August, when comparing the results determined for ALL data-points to the results with the 4.5% cut-off applied for the control group level, average packet loss registered by Centurylink remains the same in both representations. Stable results suggest that Centurylink has registered preponderantly lower value of loss for packets which transit its network, below the 4.5% cut-off.

Hurricane Electric which has registered a good result for ALL data-points, went to the group with worse results after the cut-off at the 4.5% has been applied to the control group level. This migration shows the unstable nature of the registered loss packages for this carrier. In more cases, packet loss was higher than the 4.5%.

Latency

For Latency analysis we use a similar scatter plot to the one we used for Loss. It displays control group values on the diagonal while highlighting individual carrier measurements on the horizontal and the vertical axis. Datapoints placed significantly and consistently below the average highlight better performing carriers while datapoints above the average highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

The results above linger around the diagonal with the following observations:

• Results of NTT are spotted on the far right of the graph that depict very long haul traffic;
• Cogent, and XO have been mostly present within the latency diapazon of 70- 190 ms;
• Zayo,Telia and GTT are spotted within the latency diapazon of 50-200 ms;
• Level 3 has been registered within the 50-220 ms latency range;
• Centurylink has been mostly registered within the latency diapazon of less than 130ms.

Fig. 10. Average latency gains/losses by carrier Values averaged for the difference between carrier performance and the average group in that network.

The differences in latency above from the control group are averaged with the expectation that better or worse performance will cancel out if the differences are caused by measurement noise.

The results show that during the month of September in comparison with August:

• Cogent moved to the negative category,increasing its average latency for each packet by ~0.5 ms in comparison with the control group level;
• XO maintained the leading position, reducing its average RTT for each packet in comparison to the registered control group level by ~8.9 ms.XO is followed by Hurricane Electric with the ~ 5 ms RTT reduction. Telia and GTT improved the average latency for each packet by ~1.3 ms. Centurylink and Level 3 show modest ~0.5ms reduction of their average RTT.

Appendix. Carrier Loss (highlighted)

Loss improvement/worsening highlighting Centurylink datapoints.

Loss improvement/worsening highlighting Cogent datapoints.

Loss improvement/worsening highlighting GTT datapoints.

Loss improvement/worsening highlighting Level 3 datapoints.

Loss improvement/worsening highlighting NTT datapoints.

Loss improvement/worsening highlighting Telia datapoints.

Loss improvement/worsening highlighting XO datapoints.

Loss improvement/worsening highlighting Zayo datapoints.

Loss improvement/worsening highlighting Huricane Electric.

Appendix. Carrier Latency (highlighted)

Latency spread chart highlighting Centurylink.

Latency spread chart highlighting Cogent.

Latency spread chart highlighting GTT.

Latency spread chart highlighting Level 3.

Latency spread chart highlighting NTT.

Latency spread chart highlighting Telia.

Latency spread chart highlighting XO.

Latency spread chart highlighting Zayo.

Latency spread chart highlighting Huricane Electric.