BGP errors, BGP error codes, and BGP error handling

In this simplified version the Connect, OpenSent and OpenConfirm states have been merged together.

BGP sessions start in the Idle state. At that point, the router doesn’t try to connect to the BGP neighbor in question, and it will ignore incoming connection attempts. So seeing a BGP neighbor in the Idle state is normal, but if the neighbor stays in the Idle state for a long time, this is usually the result of some kind of error condition. Under some circumstances, the neighbor may progress immediately from Idle to OpenSent or OpenConfirm, but usually, the progression is from Idle to Active.

The Active state is a bit of a misnomer, because there is no active BGP session at this point—active means that the router is actively trying to connect to the BGP neighbor in question. When error conditions arise, the state may go back to Idle. But the normal progression is from Active to Connect / OpenSent / OpenConfirm.

Connect, OpenSent and OpenConfirm are three stages in the establishment of a BGP session. There are many opportunities for errors in these states, as we’ll discuss below. Most of these errors result in the state returning to Idle and a few mean the session goes back to Active. But if everything goes according to plan, the session enters the Established state.

As soon as a session is in the Established state, the router starts to scan its BGP table and see if it needs to send BGP update messages to its neighbor. There is no further indicator that shows if the router is still in that initial scanning/update sending state, or that a full copy of the BGP table (in so far as permitted by filters) has been transmitted to the neighbor and now only keepalive messages are sent until something changes.

You can see which state a BGP neighbor is in with the command that shows an overview of the BGP neighbors. On a Cisco router, that would be the show ip bgp summary or show bgp ipv4 unicast summary command to display information about neighbors for which IPv4 is enabled, and show bgp ipv6 unicast summary to display information about neighbors for which IPv6 is enabled. For instance:

Router# show ip bgp summary
Neighbor      V     AS  MsgRcvd MsgSent  TblVer  InQ OutQ   Up/Down  State/PfxRcd
198.51.100.1  4  65001        0       0       0    0    0     never          Idle
198.51.100.2  4  65002        0       0       0    0    0     never        Active        
198.51.100.3  4  65003        3       8       0    0    0  00:01:18             1
198.51.100.4  4  65004        6       5       0    0    0  00:03:43        Idle (Admin)
198.51.100.5  4  65005        6       5       0    0    0  00:00:53        Idle (PfxCt)
198.51.100.6  4  65006        1       2       0    0    0     never          Idle   

The first neighbor is in the Idle state and the second in the Active state. In both cases, there hasn’t been any contact with the neighbor yet, as indicated by no messages received and sent (the MsgRcvd and MsgSent columns).

The third neighbor is in the Established state. However, in that state the router doesn’t say “Established”, but rather, it shows the number of prefixes received and accepted from the neighbor. If this number is zero, either the neighbor hasn’t sent any prefixes (yet), or the ones sent by the neighbor were filtered out.

The fourth neighbor is in the Idle state because we’ve configured neighbor 198.51.100.4 shutdown, as indicated by (Admin). It will remain in this state until we change the configuration for this neighbor with no neighbor 198.51.100.4 shutdown. Note that it’s also possible to configure shutdown for a peer group, and then all neighbors that are part of the peer group are shut down.

The fifth neighbor is in the Idle state, but here we also know why: the neighbor has exceeded the configured maximum prefix limit. (With the neighbor 198.51.100.5 maximum-prefix … command.) The neighbor will remain in the Idle state until the session is manually restarted with the command clear ip bgp 198.51.100.5. Of course if the neighbor is still sending the same number of prefixes and the maximum prefixes limit hasn’t been changed, it will quickly return to the Idle (PfxCt) state.

The sixth neighbor is again in the Idle state, but there is a difference here with the first neighbor: the router has exchanged some packets with the sixth neighbor, as we can see under the MsgRcvd and MsgSent columns. That means it’s very likely that the session won’t reach the Established state and remains in Idle because an error has occurred.

When something goes wrong with a BGP session, the neighbor that detects the error condition will send a BGP “notification” message and then terminate the BGP TCP session. With the command show ip bgp neighbors [neighbor address] we can see detailed information about BGP sessions. (Again, or show bgp ipv4 unicast neighbors or show bgp ipv6 unicast neighbors as appropriate.)

Towards the end, there’s a line “Last reset”. This line can be very informative:

Router# show ip bgp neighbors | incl (BGP neighbor is|Last reset)
BGP neighbor is 192.0.2.1,  remote AS 65501, external link
  Last reset never
BGP neighbor is 192.0.2.2,  remote AS 65502, external link
  Last reset 1w6d, due to BGP Notification received, hold time expired
BGP neighbor is 192.0.2.3,  remote AS 65503, external link
  Last reset 7w0d, due to Active open failed
BGP neighbor is192.0.2.4,  remote AS 65504, external link
  Last reset 1w6d, due to BGP Notification received, Connection Rejected
BGP neighbor is 192.0.2.5,  remote AS 65505, external link
Last reset 2d18h, due to BGP Notification received, Connection Collision Resolution
BGP neighbor is 192.0.2.6,  remote AS 65506, external link
  Last reset 28w4d, due to Peer closed the session

The IANA registry for BGP parameters lists a large number of BGP error codes and subcodes for the BGP message types that may appear in a notification message. Most of these should be rare, as they indicate a problem with the implementation of the BGP protocol. The most relevant ones to BGP operation are probably the Cease subcodes:

• Maximum Number of Prefixes Reached
• Administrative Shutdown
• Peer De-configured
• Administrative Reset
• Connection Rejected
• Other Configuration Change
• Connection Collision Resolution
• Out of Resources
• Hard Reset

These are described in RFC 4486, which is only a few pages long. Note that Connection Collision Resolution is not an error, this happens when two routers connect to each other at the same time so there are initially two BGP sessions, so one needs to be terminated.

Most of these errors are rather obvious. The main cause of many errors is misconfiguration on one side. This is especially likely if the connection hasn’t yet been established successfully after configuring a neighbor. So check the settings and parameters such as IP addresses and AS numbers on both sides—nobody is immune from typos.

If the session worked before, and neither side has made changes, then perhaps there is an issue with the router that sent the notification. For instance, the router may have run out of memory, or there may be a bug in the BGP implementation. It’s also possible that an AS further upstream is sending BGP updates that trigger an issue in the local router or the neighboring router. An example of this is the issue triggered by AS 47868 back in 2009. Due to unfortunate circumstances, AS 47868 sent out an AS path with 252 prepends. This triggered a bug in Cisco routers with the handling of paths with 255 ASes or more a few hops further downstream. So when the errors really don’t make any sense, check what’s happening in ASes multiple hops away.

Last but not least, it can be useful to check the router’s log. The command that does this is show logging. This will display recent log messages in the router’s log buffer in RAM. If this feature isn’t enabled, or you want to change the log level, use the configuration command logging buffered <level>. You can also log to other destinations, such as a syslog server.

It’s also possible to turn on displaying log messages and debugging information live on a command line session to the router with the command terminal monitor. Turn this back off again with terminal no monitor. Two messages to look out for in the log are these ones:

%TCP-6-BADAUTH: No MD5 digest from 192.0.2.1(179) to 192.0.2.2(64444)
%TCP-6-BADAUTH: Invalid MD5 digest from 192.0.2.1(11004) to 192.0.2.1(179)

(The 64444 and 11004 numbers are TCP port numbers that will be different and the 179 port number may follow the first or the second IP address based on which router happened to initiate the BGP session.)

As these issues are at the TCP level and not the BGP level, and the BGP process doesn’t get to see the messages created with a missing or incorrect password, this issue is easy to overlook without checking the log.

When debugging is enabled along with terminal monitor, debugging messages are displayed through the command line session. Be careful with enabling different types of debugging, as debugging too much can easily lock up a command line session, and sometimes even bring down the router. The most useful type of debugging is debug bgp events. This provides some useful information on BGP events, without being too overwhelming. Turn it back off again with undebug bgp events or undebug all

With the above in mind, you should be able to find virtually all issues with non-working BGP sessions.