Inbound prefix lists for everything else

As mentioned, between ISPs there’s usually no strict filtering. Also, as customers expect to receive all prefixes reachable over the internet from their ISP(s), why have incoming filters on those BGP sessions?

However, even if you don’t have a list of prefixes that you do want to receive from a peer or an ISP, there’s still a list of prefixes that you don’t want to receive from a peer or ISP: your own prefixes and internet exchange prefixes.

Suppose your prefix is 198.51.100.0/24 and you run your mail server on 198.51.100.13. So you announce 198.51.100.0/24. But what if a peer or ISP announces 198.51.100.13/32 to you? In that case, your routers will happily send all your mail towards your peer or ISP. So what you’ll want to do is make a filter that doesn’t allow your own prefixes or any more specific / longer prefixes (smaller address blocks) that fall within those prefixes:

This filter rejects 198.51.100.0/24 and all the prefixes that fall within 198.51.100.0/24 all the way to individual IP addresses. The second line then allows everything, as long as the prefix length is no more than /24. This is a filter that everyone should apply as an inbound filter on BGP sessions that don’t have a stricter filter.

If your network is connected to one or more internet exchanges, you’ll also want to reject the prefixes used for those internet exchange peering LANs in your inbound filters. The reason for this is that if someone announces a more specific prefix that falls within the internet exchange prefix, two bad things can happen. Your routers may send the traffic that should go towards the peers with addresses within that more specific prefix towards the network that announces the more specific rather than to the peers in question, because the next hop address learned from those peers is now routed to the source of the more specific prefix. Your router or routers that connect to the internet exchange in question will also try to send their BGP packets to the source of the more specific prefix rather than to the actual BGP neighbors with addresses in the more specific prefix. So after some time, those BGP sessions will go down. So reject all the peering LAN prefixes for internet exchanges that your network connects to in your inbound prefix list.

Resetting sessions

After building the right filters and applying them to the appropriate BGP sessions, there’s one final step: resetting the BGP sessions. Changing filters on an active BGP session is like putting a bouncer in front of the door of a club when the club has been open for a while. Sure, no new people with the wrong shoes can get in, but the incorrectly dressed people who arrived before the bouncer was in place are still inside. So what’s needed is the equivalent of getting everyone out and come back in again so the bouncer has the opportunity to see if they’re dressed well enough. With BGP filters, we do this by clearing/resetting the BGP session.

The most straightforward way to reset a BGP session is with the clear ip bgp command.

All the clear commands take an IP address, an AS number or an asterisk as their first argument. Unsurprisingly, clear ip bgp 192.0.2.2 clears (resets) the BGP session towards neighbor 192.0.2.2. clear ip bgp 65001 clears all BGP sessions from this router towards routers in AS 65001. And clear ip bgp * clears all BGP sessions. Without further instructions, clearing a BGP session means that the session is torn down, all prefixes learned from the neighbor in question are removed from the BGP table and the routing table. The router then initiates a new BGP session. When that session is established, all prefixes are transferred and the new filter is applied.

Soft resets

Fortunately, there’s a better way. When you’ve changed an outgoing filter or route map, you can use the clear ip bgp … out command. This doesn’t reset the session, but rather, tells the router to push all the prefixes in the BGP table out to the neighbor or neighbors in question, applying the currently configured filters in the process. So prefixes that were previously filtered but are now allowed are then propagated to the neighbor, and prefixes that were allowed before but are filtered now are withdrawn. The result is exactly the same as with an actual reset of the BGP session, but without any disruption. The neighbor simply sees a sequence of updates. The clear … out command doesn’t require any cooperation from the BGP neighbor, so it always works, regardless of the capabilities of the neighbor.

It’s also possible to push prefixes through the incoming filters and route maps using the clear ip bgp … in command. The effect is the same as with the clear … out command. However, the implementation is different: in order for the router to apply its current inbound filters and route maps, it has to receive new copies of all the neighbor’s prefixes. It does this by asking the neighbor to send over its prefixes. This “route refresh” capability (RFC 2918) is an extension to the BGP protocol, so some routers don’t implement it. You can determine this with the show ip bgp neighbors <address> command. The output may include the following:

The 4-byte AS capability refers to the ability to handle 32-bit AS numbers. As 32-bit AS numbers are in wide use today, all current routers support this capability. In this example, the local router advertises the route refresh capability and it also received an advertisement of the route refresh capability from the neighbor. “Old” refers to the old Cisco implementation of the capability before RFC 2918 was published, “new” means the RFC 2918 compliant implementation of route refresh.

Should your neighbor not support route refresh, all is not lost. It’s possible to simulate this capability by configuring soft-reconfiguration inbound:

With this in place, the router will store a copy of all prefixes received from neighbor 192.0.2.2—including the ones that are filtered out. You can now use the clear ip bgp … in or clear ip bgp … soft in command to apply modified filters and route maps without a hard reset. Be warned that Cisco notes that “this method is memory-intensive and not recommended unless absolutely necessary.”

Inspecting the BGP table

However, soft-reconfiguration inbound does have an added benefit: you can inspect the full set of prefixes advertised by a neighbor, including all the prefixes filtered out. This is done with the show ip bgp neighbors … received-routes command. This lists all the prefixes learned from the neighbor in question in the same format as the show ip bgp output. This includes prefixes that are filtered out which aren’t added to the BGP table. The show ip bgp … routes command, on the other hand, only shows the prefixes learned from the neighbor that are actually added to the BGP table. Another difference is that show … received-routes shows the prefixes with their original attribute values, while show … routes shows them after a route map may have altered certain attributes. For instance, this route map changes most of a prefix’s BGP attributes:

Which results in:

The show … received-routes command, on the other hand, shows the prefix in its original form:

More detailed information on prefixes in the BGP table (but not ones filtered out and kept by soft-reconfiguration inboud) is shown using the show ip bgp <prefix> command:

In this example, no prefix length was specified for network 2.0.0.0, but the router found prefix 2.0.0.0/8—it didn’t automatically assume we meant /8 because 2.0.0.0 is a class A network. There are two paths/routes available towards this prefix. The first route is the one mangled by the route map above, as we can see from the presence of AS 4,200,000,000 in the AS path. On the next line, the first address is the next hop address. This is usually the same as the next address, the neighbor address. But in this case the route map changed the next hop address, but the neighbor address isn’t affected. The third address is the router ID of the neighbor, which we can use to determine whether two BGP sessions go towards the same router or to different routers.

The second path towards 2.0.0.0/8 is considered best by BGP; it also has the very high local preference value of 10,000, but the AS path is only one hop rather than two. This path also has two community values attached.

The router indicates that the prefix is not announced to peers (neighbors). If it is, you’ll usually see their IP addresses here. So the show ip bgp <prefix> command is useful to see whether a specific prefix is announced to neighbors. Alternatively, you can use the show ip bgp neighbors … advertised-routes command to list all the prefixes announced to a specific neighbor.

Propagation of updates

BGP is a real-time system, so when a BGP session is established or a new prefix is injected into BGP, the new information typically propagates very quickly. You can monitor this using one of the many “looking glasses” that are available around the globe. (Look for “BGP looking glass” in a search engine.) However, sometimes it takes a little longer of prefixes to propagate to all corners of the internet. The main reason for this is the minimum route advertisement interval (MRAI). RFC 4271 specifies that a certain minimum amount of time should elapse between eBGP updates for “routes to a particular destination”. The suggested value for the MRAI is 30 seconds.

The idea is that if there’s an update for the same prefix every five seconds, the first update is propagated, and then nothing happens for the next 30 seconds. After the 30 seconds, the most recent information is sent to the neighbor or neighbors in question. So the updates at 5, 10, 15, 20 and 25 seconds aren’t propagated. This reduces the instability that may be injected into the BGP system.

When a new prefix is announced because it’s newly injected into BGP or a session that has been down comes back up, usually the MRAI doesn’t come into play, because the update typically travels fastest over the best path. So that best path is installed in the BGP table and used. The update then comes in over longer paths, but those updates aren’t propagated because they contain paths that are worse than the already known path. (Exceptions are possible when for traffic engineering or policy reasons, a path that’s longer than the shortest path is preferred).

However, this is different when routes are removed from BGP, and to a lesser degree when routes are made less preferred, for instance, by path prepending. In that case, BGP will start “path hunting”. Consider the topology in figure 1.

Figure 1. Path hunting example topology

ASes 20, 30, 40 and 50 all receive the prefix that AS 10 advertises. AS 20 propagates the prefix to AS 30, AS 30 propagates it to AS 40, and AS 40 propagates it to AS 50. This results in the BGP tables shown in table 2. The greater-than sign indicates the best path.

Table 2. Initial AS paths

When at this point AS 10 goes down or otherwise stops advertising prefix 198.51.100.0/24, each AS will remove the direct path towards AS 10, as shown in table 3.

Table 3. AS paths immediately after AS 10 becomes unreachable

Now, the following updates will happen:

• AS 40 tells AS 50 that the path is now 30 10
• AS 30 tells AS 40 that the path is now 20 10
• AS 20 tells AS 30 that 198.51.100.0/24 is now unreachable

After this round of updates, the BGP tables look as shown in table 4.

Table 4. AS paths after the second round of updates, shortly after AS 10 becomes unreachable

Immediately after receiving its update from AS 30, AS 40 needs to send another update to AS 50, informing it that the path is now even longer. And AS 30 no longer has a route towards 198.51.100.0/24, so it needs to send a withdrawal towards AS 40.

However… This is where the minimum route advertisement interval kicks in, as AS 30 already just sent an update to AS 40 and AS 40 one to AS 50. So for 30 seconds, nothing happens. Then the updates are sent, resulting in the BGP tables shown in table 5.

Table 5. AS paths after the third round of updates, 30 seconds after AS 10 becomes unreachable

AS 40 now sees the withdrawal from AS 30 and needs to propagate that withdrawal to AS 50. But the MRAI delays the update once again, so this withdrawal is delayed by another 30 seconds. After that delay, we reach a new stable situation where 198.51.100.0/24 is considered unreachable by every AS, as shown in table 6.

Table 6. AS paths after the fourth round of updates, 60 seconds after AS 10 becomes unreachable

So the path hunting behavior means that when a prefix becomes unreachable, BGP will explore longer and longer paths before the prefix disappears from routing tables everywhere. When a prefix really becomes unreachable, it’s not a problem that for about two minutes, packets try to follow longer and longer paths before they’re ultimately lost anyway. However, path hunting can be very problematic when the prefix in question is a more specific prefix, and there’s also a less specific prefix covering the same address range.

For instance, suppose that a network has prefix 198.51.100.0/23 and is connected to two ISPs. In order to get their traffic engineering to work the way they want, they decide to split the /23 into two /24s, and announce 198.51.100.0/24 to one ISP and 198.51.101.0/24 to the other ISP. The /23 is announced to both ISPs as a backup. If now the link to the first ISP goes down, path hunting will happen for 198.51.00.0/24. Once the path hunting is over, traffic will flow towards the /23, but during path hunting, there will be instability, as paths keep changing every 30 seconds. Some of these paths may work, others may not. Typically, this takes two minutes, with pings coming through during some of the 30-second periods and going unanswered during others.