Tier 1 carrier performance report: August, 2017

Sep 7, 2017

The current report covers widely used Tier 1 carriers performance results in the US for the month of August 2017. Rare datapoints from distant locations have been omitted.

Disclaimer*:
The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the IRP Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document.

The data collection method comprises of a Netflow collector looking at destinations an installed network is speaking to and the explorer component then performing periodic probing via ICMP, UDP and TCP_Syn, checking the performance metrics. The presented data is well sampled, as most of the IRP Lite enhanced networks are typically speaking to around a third of the routing table each day. For privacy protection, the exact location and number of IRP Lite instances are not provided.

The current report covers widely used Tier 1 carriers performance results in the US for the month of August 2017. Rare datapoints from distant locations have been omitted.

The presented analysis is based on more than 809 million successful probes that span the entire month. All data is aggregated per carrier on a daily basis and accounts for many thousands of successful probes. A control group (labeled C) is used as a base of comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers.

Fig. 1. Average Loss and Latency in August 2017
The numbers include a control group C (gray) to allow cross comparison.

The values for July 2017 are included for cross comparison.

Fig. 2. Average Loss and Latency in July 2017
The charts include a control group C (gray) to allow cross comparison.

Average packet loss analysis:

Results presented in Figure 1 suggest that NTT, Level 3 and Zayo are the best performers for the analysed group in terms of packet loss. XO and Centurylink slightly exceed the average for the control group level. Hurricane Electric,Telia and GTT have registered the worst results.
Dynamic analysis of average packet loss did not show a big difference between the registered structure of Tier 1s in August and the registered structure of Tier 1s in July.

Average latency analysis:

In August, Centurylink has registered the lowest average latency for the analysed group, followed by Level 3 and NTT. XO can be named a good performer as well. Its results are placed near the control group level.
As in July, the maximum average latency was registered by Hurricane Electric, followed by Cogent. GTT, Telia and Zayo, which have registered approximately the same average latency, are positioned between the poor performing Tier 1s and the control group level. The control group has kept the same average latency level for both months.

The charts below illustrate the performance of each carrier in comparison to the control group.

Fig. 3. Better or worse Loss and Latency in August 2017
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in July 2017
The numbers are differences from average control group.

In comparison with the control group for the month of August, 2017:

Centurylink, Level 3, NTT and XO showed better results in terms of Latency.
Level 3, NTT and Zayo showed better results in terms of Loss.

In comparison with the data from July, 2017:

Tier 1s didn’t show a radical change in their performances. In term of latency, XO was the only carrier which results have somewhat improved. In term of loss, Centurylink showed worse results than in the previous month, migrating to the group of Tier 1s with the higher average loss than the control group level.

Loss

For the Loss analysis we use a scatter plot, where average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All data-points below the diagonal represent the better performing carriers and vice versa.

Abnormally large losses are still registered for a large number of datapoints. We consider excessive an average above 4.5% packet loss. Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions are deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption. More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier August 2017
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier July 2017
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Comparing the results determined for ALL data-points to the results with applied cut off at 4.5% control group level, we observed that average packet loss registered by Centurylink is the same in both representations. Stable results, suggest that Centurylink has registered preponderantly lower value of loss for packets which transit it, below 4.5%.

Zayo which has registered a good result for ALL data-points, went to the group with worse results after we applied the cut off at the 4.5% control group level. This migration shows non-stability in the registered loss packages for this carrier. In more cases, packet loss was higher than the 4.5%.

As in July, worst position in both graphics are held by Hurricane Electric, Cogent and Telia. Level 3 has left the winners group both for ALL data-points and for the applied cut off at the 4.5% control group. Other Tier 1s have not registered a significant change in average Loss for the month of August, compared with July.

Latency

For Latency analysis we use a similar scatter plot to the one we used for Loss. It displays control group values on the diagonal while highlighting individual carrier measurements on the horizontal and on the vertical axis. Datapoints placed significantly and consistently below the average highlight better performing carriers while datapoints above the average highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

The results above linger around the diagonal with the following observations:

NTT and Level 3 are spotted on the far right of the graph that depict very long haul traffic;
Cogent and GTT have been mostly present within the latency diapazon of 75 – 200 ms;
Zayo, Hurrycane and XO are spotted within the latency diapazon of 60-180 ms;
Telia has been registered within the 60-220 ms latency range;
Centurylink has been mostly registered within the latency diapazon of less than 130ms.

Fig. 10. Average latency gains/losses by carrier Values averaged for the difference between carrier performance and the average group in that network.

The differences in latency above from the control group are averaged with the expectation that better or worse performance will cancel out if the differences are caused by measurement noise.

The results show that during the month of August in comparison with July:

Hurricane Electric moved to the positive category, improving its average latency for each packet by ~4.7 ms;
XO maintained the leading position, reducing its average RTT by ~5,8 ms. Telia improved its average latency for each packet by ~1.2 ms, Centurylink and GTT reduced average RTT by ~1ms. Cogent, Level 3 and Zayo show modest ~0,2 ms average latency improvement results.