The 768k or Another Internet Doomsday? Here’s how to deal with the TCAM overflow at the 768k boundary.

 
Now, after almost 5 years, we might encounter a similar problem. The size of the global Internet routing table is reaching 768k routes so the possibility of the TCAM resource exhaustion at 768k routes is coming. Although many legacy routers with limited TCAM size have been replaced already, it is possible that there are still many devices remaining that may encounter the TCAM issues. The good news is that the TCAM size for IPV4 routes can be modified for some network equipment. For example, to increase TCAM on 3BXL modules for Cisco Catalyst 6500 and 7600 series routers and switches you can enter the command below. Afterward, save the configuration and reboot the device [2].

7600(config)# mls cef maximum-routes ip 1000

The command does not increase the overall size of the FIB TCAM, but it reduces the number of routing entries that are allocated to the IPv6 in order to increase the amount of TCAM space for IPv4. Maximum routes are set to 1024000 (1000k or 1000 * 1024 ) and the total number of available MPLS labels, IPv6 routes, and IPv4 multicast routes is reduced to only 8,000. See Picture 1.

Picture 1: Overall Size of the TCAM FIB

Picture 2 depicts a new TCAM allocation after the reboot. These new settings will help your network avoid any performance degradation, routing instability, or impact on availability. In other words, the proposed solution will help you deal with TCAM overflow at the 768k boundary.

Picture 2: New TCAM Allocation After Reboot
Source: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

Picture 3: Checking the Total Number of Routes per Protocol
Source: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

So far, we have explained the effects of TCAM exhaustion and discussed a workaround that reduces the size of  IPv6 TCAM in favor of IPv4 TCAM. The most common question, however, is when the global BGP table size will reach the 768,000 routes. The answer can be provided by a Twitter bot that tweets the size of IPv4 BGP table 4 times a day.  The bot reported that the global Internet routing table had passed 768,000 routes on April 17th, 2019 (Picture 4). The prefixes smaller than /24 were filtered.

Picture 4: The Global BGP table Exceeds 768k Routes on Apr 17th, 2019
Source: https://twitter.com/bgp4_table

However, the Classless Inter-Domain Routing (CIDR) that also provides statistics on the global Internet routing table, reported that the global Internet routing table had passed 768,000 routes on March 5th, 2019. This illustrates an important principle in BGP, that there is no single authoritative view of the Internet’s inter-domain routing table – all views are in fact relative to the perspective of each BGP speaker [3].

Picture 5: The BGP table Exceeded 768k Routes on Match 5th, 2019

Another interesting factor that raises ambiguity in when and if the 768k boundary has been reached is the interpretation of the “k” value. Some sources suggest that “k” should be interpreted as 1024 for TCAM size. This gives us the value of 786432 global Internet routes that have not been reached yet.

You can simply check an actual size of the global Internet routing table connecting to any of the public route servers from the list. Open a terminal and run the command below. The username is ‘rviews’ with the password set to ‘rviews’.

$ telnet route-server.ip.att.net 
rviews@route-server.ip.att.net> show route summary

Picture 6: Checking the Size of the Global Internet Routing Table from AT&T Looking Glass Service

There are 735,386 active IPv4 BGP routes and 64,665 active IPv6 BGP routes as per AT&T Looking glass service (Picture 6).

Conclusion:

Your networking equipment should be able to manage the ongoing growth of the Internet routing table. If your router might be affected by the 768k limit, you have a number of options here. For instance, you can reconfigure your router to decrease the number of TCAM entries allocated for IPv6 routes in favor of IPv4 routes before your table hits 768k or to filter some prefixes. An upgrade or a replacement of your legacy network equipment can also be an effective solution in order to cope with the outgoing growth of the table.