Tier 1 carrier performance report: February, 2018

The report covers broadly used Tier 1 carriers performance results in the US for the month of February 2018. Rare datapoints from distant locations e.g. Alaska or Hawaii have been omitted.

The report is presented in a slightly different form from the previous ones. Besides the general analysis we have complemented it with data reflecting the Tier 1 Carriers Performance results at the the following levels:

Worldwide – The min RTT to the probed prefixes is over 110 ms;
National – The min RTT to the probed prefixes is between 80 and 110 ms;
Regional – The min RTT to the probed prefixes is between 40 and 80 ms;
Local – The min RTT to the probed prefixes is between 0 and 40 ms.

The analysis is based on more than 663 million successful probes that span the entire month. All data is aggregated per carrier on a daily basis and accounts for many thousands of successful probes. A control group (labeled C) is used as a base for comparison. The control group aggregates the average for all transit providers in a network, including Tier 1 carriers.

General statistics

Fig. 1. Average Loss and Latency in February 2018
The charts include a control group C (gray) to allow cross comparison.

The values for January 2018 are included for cross comparison.

Fig. 2. Average Loss and Latency in January 2018
The charts include a control group C (gray) to allow cross comparison.

Average packet loss analysis:

• Centurylink obtained the leading position in February 2018, and is closely followed by Level 3, NTT and XO;
• Cogent and Telia results for the month of February are close to the control group level;
• A higher level of average loss has been registered for Hurricane Electric;
• All analyzed Tier 1 carriers bring to the table worse results in terms of packet loss when compared to January data. The biggest decline has been registered for Hurricane Electric.

Average latency analysis:

• Based on the control group level in February, 2018, the lower average latency has been registered for: Centurylink, Level 3, Zayo and Telia. Other Tier 1s showed poorer results than the control group. The worst results have been reported by GTT and NTT;
• Compared with January’s results, a significant change in average latency has been noted for the majority of the analysed Tier 1s. The control group has increased its latency level, dividing the analysed group into two categories: the better performing and worse performing carriers. Centurylink, Level 3, Zayo and Telia have been placed in the “better performing” category. The rest of the carriers are in the “worse performing” category. We must mention, however, that Zayo showed the same average latency as in the month of January. GTT registered the worst result in terms of latency among the analyzed carriers in the last two months.

The charts below illustrate the performance of each carrier in comparison to the control group.

Fig. 3. Better or worse Loss and Latency in February 2018
The numbers are differences from average control group.

The values for January 2018 are included for cross comparison.

Fig. 4. Better or worse Loss and Latency in January 2018
The numbers are the differences from the average control group.

In comparison with the control group for the month of January, 2018:

• Centurylink, Level 3 and Zayo showed better results in terms of Latency;
• All Tier 1s, with the exception of Hurricane Electric, showed better results in terms of Loss.

In comparison with the data from January, 2018:

• Tier 1s had a significant change in the registered performances. Hurricane Electric was the only one to keep worse result than the control group level one in terms of Loss;
• Better results in terms of latency showed Level 3 joined by Centurylink and Zayo. NTT has migrated into the “worse performing” carriers group.

Loss

For the Loss analysis we use a scatter plot, where average values by control group are assumed on the diagonal while the horizontal and the vertical axis highlight carrier metrics. All datapoints below the diagonal represent the better performing carriers and vice versa (the same approach is used for the Loss analysis on the Worldwide, National, Regional and Local levels).

Fig. 5. Loss values spread on average diagonal
Datapoints comparison with diagonal.

Abnormally large losses are still registered for a large number of datapoints. As was mentioned in previous reports we consider excessive an average above 4.5% packet loss.
Given the fact that Tier 1 carriers are characterized by both low loss values for some networks and abnormally high losses for the other networks, the conclusion is that high loss values are not caused by the carriers themselves but rather are caused by the networks they service Or the networks they peer with. Whether the true cause is poor design, over-provisioned links or deficiencies in the peering governance – this report cannot tell. What we can mention is that for many networks, whether permanently or sporadically, there is definitely an opportunity to improve things.

Fig. 6. Better or worse carrier loss (%)
Average placed on the zero line.

A different representation (Fig. 6) of the above data places it around the control group (zero line) with gain values by carrier. Values are sorted and charted from left to right by the increasing average loss. The chart depicts gains or worsening on a network based on the average control group’s performance – values are shown from left to right following better to worse loss values. The assumption of this analysis is that while a network’s conditions might be better or worse compared to other networks, the conditions tend to be equal across all carriers including the control group. While the carrier’s network is not the culprit causing additional loss, this analysis might be able to suggest whether those carriers peering with remote regions is deficient. Non-systemic issues with carriers will tend to cancel out with values being scattered equally above or below the zero line while systemic issues or gains will have a tendency to place a carrier consistently above or below it. The scatter plot highlights this assumption. More so, if we average gains or losses compared with the control group we expect the noise to cancel out.

Fig. 7. Average packet loss gains/losses by carrier February 2018
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied

The values for January 2018 are included for cross comparison.

Fig. 8. Average packet loss gains/losses by carrier January 2018
Averages determined for ALL datapoints or a cutoff at 4.5% control group applied

When comparing February and January data in terms of average Loss, all carriers except for Centurylink, have registered significant changes in their results. Based on all data points: XO, Zayo, and GTT have registered ~1.3-1.5% better loss values than the control group level. Level 3, NTT, and Centurylink follow the “winner group” with ~0.3-0.5% better loss values for each packet.

After the cut off at the 4.5% control group level has been applied Cogent occupied the same position as in the ALL data-points representation. These results suggest that Cogent has registered preponderantly lower value of loss for packets which transit it, below the 4.5%. Hurricane Electric disappeared from the second representation. This means that in February, this carrier registered higher than 4,5% packet loss.