Around 200 million successful probes performed by IRP instances over the previous weeks offered us plenty of data for analysis. The current report card covers the performance of major providers within the continental United States.
As with our previous report, all values are aggregated per provider per day and each accounts for thousands of successful probes. An average control group is identified and it aggregates probing data from all transit providers, including Tier 1 carriers.
We’ll keep on sharing our insights with the community on a monthly basis and we truly welcome your comments or questions in regards to the presented report card.
Let us start with overall averages. The analysis is based on US wide networks – networks where most of the traffic is concentrated in continental US. These networks are characterized by similar latency values and using only this segment of networks allows the analysis to avoid the tendency by smaller or very large networks to skew average values in favor of those carriers that offer a significant fraction of their services to small networks.
Fig. 1. Average Loss and Latency.
The numbers include an Average (gray) control group to allow cross comparison.
Data is based on probing results for US networks as measured during December 2016.
The averages above suggest that NTT and Level 3 are best performers on both Loss and Latency metrics. It is harder to spot the worst offenders and there is a variation considering Loss and Latency, but the data is fairly self explanatory.
Fig. 2. Better or worse Loss and Latency
The numbers are differences from average control group.
We can observe that during the month of December 2016 on average:
1. NTT and Level 3 offered over the board performance gains
2. XO, Cogent but also GTT and Telia bring to the table worse than average latency and loss
By displaying average value on the diagonal we can spread and cross-compare performance. Datapoints significantly and consistently below the average highlight better performing carriers while datapoints above highlight worse than average performance.
Fig. 3. Loss values spread on average diagonal
Datapoints comparison with diagonal.
The preliminary finding is that a significant number of datapoints are for abnormally large losses. There is no industry definition of what to consider abnormally high packet loss for a network. Given our data spread and that these values are averages across many thousands of probed destinations per day, for the purposes of this analysis we will choose 4.5% packet loss.
Given the fact that Tier 1 carriers are characterized by both low loss values for some datapoints and abnormally high losses for other networks the conclusion is that high loss values are not caused by the carriers themselves and instead are caused by networks they service. Whether the true cause is poor design, over-provisioned links or deficiencies in governance – we cannot tell. What we can mention is that for many networks whether permanently or sporadically there is definitely an opportunity to improve things.
Fig. 4. Better or worse carrier loss (%)
Average placed on the zero line
A different representation of the above data around the average (zero line) with gain values by carrier. Values shown from left to right by average loss. Bigger dots are used so that carrier colors dominate one or another area of the chart.
Fig. 5. Average packet loss gains/losses by carrier
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied.
Finally we try to spot if by adding a specific carrier we improve or worsen a networks outlook on packet loss. For this we average the gains and losses offered by different carriers with the expectation that due to a large number of datapoints the gains will cancel out the losses (leftside graph). We also cut off the averages at 4.5% average loss and excluded those datapoints as outliers (rightside graph).
The results show that bringing Level 3 to the table has the tendency to decrease packet loss on your netwоrk.
While cutting off datapoints has the tendency to flatten the right-side graph, this is not the case for Zayo, which transformed from a winner into a loser, nor NTT, which being bad became worse. This lets us hypothesize that the loss in very bad networks has a tendency to cancel out the loss that is characteristic for these carriers. This finding is even more strange given the fact that both NTT and Zayo have been assumed to be net winners in the overall averages analysis before. This requires further investigation and is something we will expand on in future reports.
By displaying average values on the diagonal we can review and identify outliers. Datapoints significantly and consistently below the average highlight better performing carriers while datapoints above highlight worse than average performance.
Fig. 6. Carrier latency with average group on the diagonal
Clusters of datapoints below diagonal highlight better performance
The results above linger around the diagonal with at least the following observations:
● Zayo and Telia are spotted on the far right of the graph that depict very long haul traffic and their average latency is significantly below the average.
● Cogent datapoints consistently linger above the average line implying lower than average performance.
● Zayo and this time XO display each a patch of better performance with NTT having a case of bad performance.
Fig. 7. Average latency gains/losses by carrier
Values averaged for the difference between carrier performance and the average group in that network.
The differences in latency above from the control group are averaged with the expectation that better or worse performance will cancel out if the differences are caused by measurement noise.
The results show that during December of 2016:
● Cogent adds more than 2ms to the RTT of each of the packets a network generates. Almost 2ms are added by NTT too.
● Zayo reduces average RTT by almost 3ms with GTT and NTT following close by at almost 2ms gains.
Disclaimer*: The data presented in this report card is intended for information purposes only and is not to be interpreted as any form of promotion or debasement for carriers herein named. Information is obtained from the Intelligent Routing Platform Lite instances, where the compulsory consent of the legal entities for collection of such information is part of the Terms and Conditions document. For privacy protection, the exact location and number of IRP Lite instances are not provided.
