Unwanted traffic over backup links

However, another common configuration is using one link as the primary link and using another link as a backup. The idea is that under normal circumstances, the primary link handles all traffic and the backup link isn’t used. Only when the primary link goes down or can’t be used towards certain destinations, the backup link starts carrying traffic. Often, it’s relatively cheap to have the backup link available, but the price for traffic over the backup link is relatively high.

The typical way to configure such a backup link is to assign a lower local preference to prefixes received over the backup link. As the local preference is the first attribute considered in the BGP path selection algorithm, this makes sure that traffic flows over the primary link (or links) when those are available. The routes learned over the backup link are only used as a last resort. Unfortunately, the local preference is called local preference for a reason: local preference values aren’t communicated to external BGP neighbors. But the next attribute in the path selection algorithm is: the AS path. So in order to dissuade incoming traffic from arriving over the backup link, the AS path gets prepended. The following configuration shows the relevant configuration for both the primary and the backup links:

This configuration works exactly as intended when both links go to the same ISP / autonomous system. In that case, they’ll see this in their BGP table:

So the route over the primary link is the preferred one because of the shorter AS path and the route over the backup link isn’t used at all as long as the route over the primary link is available.

However, when we make the same configuration where the primary and backup links go towards different ISPs, let’s say AS 65066 (primary) and AS 65077 (backup), it’s almost certain that we’ll see some incoming traffic over the backup link. To some degree, this can be explained by the fact that for some ASes, the path over AS 65077 is still shorter than the path over AS 65066, even with the prepends in place. The obvious solution to that is to apply even more prepends, and the incoming traffic will go down somewhat. However, at some point that simply stops working: even ridiculously long AS paths won’t completely stop the arrival of incoming traffic over the backup link.

To understand why this is we have to look at some scientific research into BGP. In their seminal paper Stable Internet Routing Without Global Coordination Lixin Gao and Jennifer Rexford show that a certain way of using the local preference attribute is necessary to make sure that BGP converges to a stable state. Using a different local preference configuration runs a significant risk that BGP will keep flipping from one state to another and back without ever reaching stability.

The guidelines for using the local preference are pretty simple: customers have the highest local preference, peers have a lower local preference and transit providers have the lowest local preference. This nicely conforms to the economics of internet connectivity: sending traffic through customers brings in money, sending traffic through peers is cost neutral and sending traffic through a transit provider costs money.

Later they slightly relax the guidelines so peers can have the same local preference as customers, but this is unusual in practice. Note that the actual values used for the local preference aren’t important, as long as customers ≥ peering and peering > transit.

So with the Gao-Rexford guidelines in place, we would expect the BGP table for AS 65077 (the backup ISP) to look something like this:

The first path has the lowest local preference (80), as it’s received through AS 4096, a transit ISP for AS 65077. The next path with local preference 90 is received over a peering connection with AS 65066. And finally, there’s the direct path to AS 65055, which is preferred because as a customer connection, it has the highest local preference of 110. Because the direct path wins based on the local preference alone, the fact that it has a much longer AS path than the other two options is irrelevant.

The result of the higher local preference for the direct path is that once traffic ends up at AS 65077, it will flow over the backup link. Because of the AS path prepending, most other ASes that have a choice between sending their traffic through AS 65066 or sending it through AS 65077 will probably send their traffic through the primary ISP, AS  65066. However, networks that are customers of AS 65077 and don’t have another ISP only have once choice: sending their traffic to AS 65077. So the incoming traffic over the backup link will mostly consist of traffic from the backup ISP and its customers.

But there’s a solution. The Gao-Rexford paper continues with a discussion on backup paths. Those should get a local preference value even lower than transit paths. So a valid backup configuration would look as follows in the BGP tables of AS 65077:

So now AS 65077 will send all traffic for 192.0.2.0/24 over the peering link to AS 65066.

An important caveat here is that the 65066 65055 path is now the best one, so that’s the path that AS 65077 will announce to its BGP neighbors. But that’s a peering path, so it won’t announce that path to transit providers or other peers. However, when AS 65055 loses its primary link to AS 65066, the 65066 65055 path as well as the 4096 65066 65055 path will disappear so the direct path to AS 65055 resurfaces and is announced to both transit providers and peers. However, there will be a slight delay between the paths over AS 65066 disappearing and the path over AS 65077 being propagated to the rest of the world.

Another downside of this setup is that if certain destinations can’t access the path through AS 65066, for instance, because of a peering dispute, they won’t be able to use the backup path over AS 65077. This backup configuration is an all or nothing deal.

This leaves one small detail: AS 65077 needs to apply a configuration that lowers the local preference from the default value of 110 for customers to the backup value of 70. Of course this can be configured manually by AS 65077, but a better solution is through a BGP community that makes the backup ISP apply the local preference value used for backup links. Please see our blog post Understanding BGP Communities for an explanation on how to do this. The relevant example is at the end of the post.