Tier 1 carrier performance: March, 2017 snapshot

This analysis covers widely used Tier 1 carriers performance results in the US for the month of March 2017. A new carrier, Hurricane Electric, has been added to the report starting this month. Rare datapoints such as data from Alaska were excluded from the final results.

Current analysis is based on more than 646 million successful probes that span the entire month. All values are aggregated per carrier on a daily basis and each accounts for many thousands of successful probes. A control group (labeled C) is used as a base of comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers.

Averages

The values for February 2017 are included for cross comparison.

Fig. 1. Average Loss and Latency. (March 2017)
The numbers include a control group C (gray) to allow cross comparison.

Fig. 2. Average Loss and Latency. (February 2017)
The charts include a control group C (gray) to allow cross comparison.

In comparison with the control group for the month of March, 2017:

• higher level of average loss has been registered for: Hurricane, Telia, Cogent and Century;
• the results close to average loss have been registered for: XO and GTT;  
• lower level of average loss have been registered for: Level 3, NTT and Zayo.
• higher average latency has been registered for: Hurricane, Zayo, XO, GTT and Cogent;
• lower average latency has been registered for: Century, Level 3 and NTT;

Average packet loss and latency increased in March when compared to February, for most carriers:

• A considerable increase of average loss has been registered for Telia, followed by Cogent. A moderate increase of average loss has been spotted for GTT, Level 3 and Zayo.
• average latency: decreased for Century, while remaining at approximately the same level for NTT and Level 3;

The charts below illustrate the performance of each carrier in comparison to the control group.

Fig. 3. Better or worse Loss and Latency in March 2017
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in February 2017
The numbers are differences from average control group.

In comparison with the control group for the month of March, 2017:

• GTT, NTT, Level 3 and Zayo showed better results in terms of Loss
• Century, NTT, Level 3 demonstrated better results in terms of Latency

In comparison with the data from February, 2017:

• Level 3 keeps its leading position, with the best results in terms of Loss. NTT, Zayo and GTT show considerable improvement. Telia demonstrates poor results;
• Century shows considerable improvement in terms of Latency (the results for Century are close to 0 therefore are not visible on the right hand chart of Fig. 4), followed by XO, meanwhile Telia is underperforming

Loss

A scatter plot is used for Loss analysis, where average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All datapoints below the diagonal are better performing carriers and vice versa.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Abnormally large losses are still registered for a large number of datapoints. As was mentioned in previous reports we consider excessive an average above 4.5% packet loss.

Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service Or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions is deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption.

More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier (March 2017)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier (February 2017)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

There are little differences between the averages determined for ALL datapoints and the ones that were cut off at the applied 4.5% control group registered in March, with the exception of Zayo which went from gains to losses. The worst position shows Hurricane, followed by Telia. As a result of the applied 4.5% control group to datapoints averages for packet loss, Level 3 position visibly decreased and its leadership became less evident. (Fig. 7)

Based on averages determined for ALL datapoints registered in February and compared to March, Level 3 and Zayo keep winning, while Century and GTT passed in the category of losses. As per the averages determined for datapoints cut off at 4.5% control group registered in February and compared to March results, all Tier 1 carriers showed negative evolution. Level 3  remains in the category of gains. Other Tier 1 carriers are in the category of losses.

Latency

For Latency analysis we use a similar scatter plot to the one we used for Loss. It displays control group values on the diagonal while highlighting individual carrier measurements on the horizontal and on the vertical axis. Datapoints placed significantly and consistently below the average highlight better performing carriers while datapoints above the average highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

Based on data points presented in Fig.9:

• NTT and Level 3 are  present over the entire diagonal of graph;
• Cogent and Hurricane data points are more scattered than average latency.

Fig. 10. Average latency gains/losses by carrier Values averaged for the difference between carrier performance and the average group in that network.

 

The differences in latency above from the control group are averaged with the expectation that better or worse performance will cancel out if the differences are caused by measurement noise.

The results show that during March of 2017:

• Cogent and NTT remain in the negative category, adding  ~1.5 ms and  ~1 ms respectively to RTT of each of packets a network forwards through these carriers.
• Hurricane showed better than average RTT reducing it by ~2.2 ms, followed by Century ~1,4 ms and GTT ~1,2 ms.

Appendix. Carrier Latency (highlighted)

Latency spread chart highlighting Centurylink.

Latency spread chart highlighting Cogent.

Latency spread chart highlighting GTT.

Latency spread chart highlighting Level 3.

Latency spread chart highlighting NTT.

Latency spread chart highlighting Telia.

Latency spread chart highlighting XO.

Latency spread chart highlighting Zayo.

Latency spread chart highlighting Huricane.

Loss improvement/worsening highlighting Centurylink datapoints.

Loss improvement/worsening highlighting Cogent datapoints.

Loss improvement/worsening highlighting GTT datapoints.

Loss improvement/worsening highlighting Level 3 datapoints.

Loss improvement/worsening highlighting NTT datapoints.

Loss improvement/worsening highlighting Telia datapoints.

Loss improvement/worsening highlighting XO datapoints.

Loss improvement/worsening highlighting Zayo datapoints.

Loss improvement/worsening highlighting Huricane.

 

---

Disclaimer*: The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the Intelligent Routing Platform Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document. For privacy protection, the exact location and number of IRP Lite instances are not provided.