Tier 1 carrier performance: January, 2017 snapshot

The analysis covers Tier 1 carriers widely used by US networks. Datapoints that appear rarely, such as data from Alaska, or a single use of Verizon and/or AT&T have been left out as potentially misleading outliers.

The results are based on more than 430 million successful probes, spanning the month of January 2017. All values are aggregated per carrier on a daily basis and each accounts for many thousands of successful probes. An ‘average’ control group (labeled C) is identified. It aggregates probing data from all transit providers in the network, including other carriers, not just Tier 1 providers.

Averages

The analysis of average values takes into account that there is a likelihood for some small and large networks to skew average values in favour of carriers that offer a significant fraction of services to smaller network. E.g. if only one carrier is present in a small network, the average values for this carrier would be skewed towards lower and thus better average values. To avoid this, the averages analysis is based on a homogenous range of networks referred to as “US wide networks”, where most of the traffic is concentrated in continental US. Nevertheless, even after this homogenization there is still a risk that final results might be skewed somewhat and as such should be considered carefully.
The values for December 2016 are included for cross comparison.

Fig. 1. Average Loss and Latency. (January 2017)
The numbers include a control group C (gray) to allow cross comparison.

Fig. 2. Average Loss and Latency. (December 2016)
The charts include a control group C (gray) to allow cross comparison.

In January 2017 compared with December 2016 the latency data stayed consistent while loss improved with the overall averages going down.

The charts above are difficult to use to spot the better and worse performers and their corresponding lead against other carriers or the control group. In order to highlight the differences the following figures depict how each carrier compared with the control group.

Fig. 3. Better or worse Loss and Latency in January 2017
The numbers are differences from average control group.

Fig. 4. Better or worse Loss and Latency in December 2016
The numbers are differences from average control group.

The averages above suggest that:

1. NTT and Level 3 are still better performers on both Loss and Latency metrics as was the case in December 2016. In January 2017, which had the overall poorer results, we see a turnaround situation for GTT.
2. Cogent and Telia continue to linger in the underperformers group, with mixed results for other carriers.

Refer to subsequent section for supporting and disclaiming confirmations of the above hypothesis.

Loss

Loss analysis continues to rely on a scatter of datapoints by displaying control group average values on the horizontal axis with carrier metrics using the vertical axis. This type of sc atter chart has the side-effect that average values represent the diagonal while individual carrier datapoints are placed above or below it depending on their corresponding cross-performance. Datapoints significantly and consistently below the average highlight better performing carriers while datapoints above highlight worse than average performance.

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

The main observation continues to be that a significant number of datapoints are for abnormally large losses which we consider to be an average above 4.5% packet loss.
Given the fact that Tier 1 carriers are characterized by both low loss values for some datapoints and abnormally high losses for other networks the conclusion is that high loss values are not caused by the carriers themselves and instead are caused by networks they service. Whether the true cause is poor design, over-provisioned links or deficiencies in governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line

A different representation of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by increasing average loss. The chart depicts gains or worsening on a network based on average control group’s performance – values are shown from left to right following better to worse loss values.
The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group.
Given this assumption, the chart above highlights situations that while a network registered a loss value of 7% across all its providers (control group), a tier 1 carrier might have contributed positively or negatively to this overall value by registering either a much better 4% or a much worse 9% loss itself.
The tendency of a carrier to stay on the left or right sides of the chart, below or above zero line suggest whether a carrier performs better or worse compared to other carriers. We will not dive deeper into these values and let you visually inspect the data and make your own hypothesis to investigate further.

Finally we try to spot if by adding a specific carrier an average network tends to improve or worsen its outlook on packet loss.

Fig. 7. Average packet loss gains/losses by carrier (January 2017)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

Fig. 8. Average packet loss gains/losses by carrier (December 2016)
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.

For this we average the gains and losses offered by different carriers with the expectation that a large number of datapoints will cancel out the noise in measurement. We also cut off the averages at 4.5% average loss on a right-side chart in order to exclude very large loss datapoints as outliers (right-side graph).
The results show that bringing Level 3 to the table has the tendency to decrease packet loss on your netwоrk.
Constraining the data to well performing networks has the tendency to flatten the right-side chart since all the loss values included are lower. Unexpectedly for Centurylink – in better performing networks their loss numbers fell dramatically for worse magnifying the already significant penalty of using this carrier.

Latency

Latency analysis uses the same scatter chart by displaying control group values on the horizontal axis while individual carrier measurements are charted on the vertical axis. Again the diagonal on this chart represents the cut-off line depicting better or worse carrier performance. Datapoints significantly and consistently below the average highlight better performing carriers while datapoints above highlight worse than average performance.

Fig. 9. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance

The results above linger around the diagonal with at least the following observations:
● GTT while consistently below the diagonal shines on regional traffic
● Zayo and Telia are spotted on the far right of the graph that depict long-haul traffic and their average latency is significantly below control group’s values.
● Cogent datapoints consistently linger above the average line implying lower performance.

Same as for loss we try to put a number next to each carrier’s performance by averaging all the differences whether positive or negative and expecting that noise cancels out while consistent gains or worsening are magnified.

Fig. 10. Average latency gains/losses by carrier (January 2017)
Values averaged for the difference between carrier performance and the average group in that network.

Fig. 11. Average latency gains/losses by carrier (December 2016)
Values averaged for the difference between carrier performance and the average group in that network.

Of note is that the scale for January 2017 is larger than December 2016 and this indicates a magnified effect to what has been spotted previously.
The results from January 2017 mainly mimic the findings of December 2016:
● Cogent adds ~4ms to the RTT of each of the packets a network forwards through this carrier. Almost 1.5ms are added by NTT.
● Zayo lost the December crown to GTT that consistently offers ~2ms gains to the average RTT with XO following close at almost 1.5ms gains.

Appendix. Carrier loss (highlighted)

Loss improvement/worsening highlighting Centurylink datapoints.

Loss improvement/worsening highlighting Cogent datapoints.

Loss improvement/worsening highlighting GTT datapoints.

Loss improvement/worsening highlighting Level 3 datapoints.

Loss improvement/worsening highlighting NTT datapoints.

Loss improvement/worsening highlighting Telia datapoints.

Loss improvement/worsening highlighting XO datapoints.

Loss improvement/worsening highlighting Zayo datapoints.

Appendix. Carrier latency (highlighted)

Latency spread chart highlighting Centurylink.

Latency spread chart highlighting Cogent.

Latency spread chart highlighting GTT.

Latency spread chart highlighting Level 3.

Latency spread chart highlighting NTT.

Latency spread chart highlighting Telia.

Latency spread chart highlighting XO.

Latency spread chart highlighting Zayo.

Disclaimer*: The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the Intelligent Routing Platform Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document. For privacy protection, the exact location and number of IRP Lite instances are not provided.