BGP and asymmetric routing

What is asymmetric routing?

Asymmetric routing is a network communication scenario where the forward and reverse paths of network traffic take different routes. In other words, asymmetric routing is the situation where packets from A to B follow a different path than packets from B to A. It is very common with BGP and can happen for various reasons, such as load balancing, network topology, or network security measures. Completely avoiding asymmetric routing is nearly impossible. Fortunately, under normal circumstances, asymmetric routing doesn’t cause any problems, as routers don’t care about this and obviously, the sending and receiving hosts see packets in both directions. Firewalls, on the other hand, often need to see the packets flowing in both directions to function properly. So it’s important to place firewalls somewhere before the packet flows in the incoming and outgoing directions diverge.

Asymmetric Routing and BGP

Asymmetric routing is a fundamental aspect of modern network architecture, particularly in networks utilizing the Border Gateway Protocol (BGP). This phenomenon occurs when the forward and reverse paths of network traffic take different routes. While asymmetric routing can offer benefits such as increased network capacity and improved load balancing, it also poses significant challenges, especially in terms of network security and troubleshooting.

This article provides a comprehensive exploration of asymmetric routing, its advantages and drawbacks, the role of BGP, and strategies for managing and mitigating its impacts. By understanding the complexities of asymmetric routing and the associated trade-offs, network administrators can make informed decisions to optimize network performance and reliability.

Understanding Asymmetric Routing

The Basics of Asymmetric Routing

Asymmetric routing, as defined, is when packets travel from point A to point B via one route and return via a different route. This phenomenon is common in BGP scenarios due to various factors such as load balancing, network topology, and security policies. 

Advantages of Asymmetric Routing

Increased Network Capacity: By leveraging multiple paths for traffic, asymmetric routing can enhance overall network capacity. This distribution of traffic helps prevent any single path from becoming a bottleneck, thereby improving network performance and reliability.

Improved Load Balancing: Asymmetric routing allows for better load distribution across the network. This balanced approach can prevent congestion and ensure more efficient use of available bandwidth, leading to optimized network performance.

Example Scenario: When Asymmetric Routing is Beneficial

In a scenario where network resilience and load balancing are paramount, asymmetric routing can be advantageous. For instance, in a large-scale content delivery network (CDN), distributing traffic across multiple paths can prevent overloading any single path, ensuring higher availability and performance. Here, the benefits of improved load balancing and optimized network capacity outweigh the potential drawbacks of asymmetric routing.

Drawbacks of Asymmetric Routing

Network Security Issues: While routers typically handle asymmetric routing without issues, firewalls and other security devices often require symmetric traffic flows to function correctly. Asymmetric routing can disrupt these devices’ ability to track sessions, leading to dropped packets.

When a (stateful) firewall detects the first packets of a new session (like when you start browsing a website), it records information about this session in an internal table. Firewalls typically track information like source and destination IP addresses, port numbers, and the protocol being used (TCP, UDP, etc.). 

The firewall keeps track of the session as long as it’s active. Once the session ends, the firewall removes the session entry from its table. By keeping track of sessions, the firewall ensures that only packets related to a legitimate session are allowed through. 

However, in an asymmetric routing scenario, request and response packets follow different paths and do not cross the same firewall. Therefore, the firewall will receive the response packets but not the requests and block the packets that are invalid.

Because firewalls need to see both the incoming and outgoing packets of a connection to track its state correctly, their placement in the network is critical. If a firewall only sees one direction of the traffic due to asymmetric routing, it won’t be able to maintain the connection state accurately. This can lead to issues with traffic being blocked or misidentified.

Complex Troubleshooting: Diagnosing issues in a network with asymmetric routing can be challenging. The differing paths for incoming and outgoing traffic can complicate the identification and resolution of network problems, requiring more sophisticated diagnostic tools and techniques.

Figure 1: symmetric routing.

Figure 1 shows a situation where asymmetric routing is unlikely. There are two paths between AS100 and AS500: through AS200 and AS300 for a total of three hops or through AS400 for a total of two hops. So if the local preference attribute is the same, the shorter AS path will make packets flow through AS400 in both directions, as shown by the blue arrows for packets from AS100 to AS500, and the red arrows showing the direction from AS500 to AS100.

Figure 2: asymmetric routing.

In figure 2, however, the two available paths are both two AS hops, so the AS path length step in the BGP path selection algorithm won’t force a decision. This generally means that the path selection decision will come down to the tie breaker rules, so there’s about a 50% chance that traffic from AS100 to AS500 will flow through AS200 and a 50% chance that it will flow through AS400. The same is true in the opposite direction, making for the following possibilities:

So there’s a 50% probability that traffic is routed symmetrically and a 50% probability that traffic is routed asymmetrically.

Figure 3: (asymmetric) hot potato routing between two ASes

However, there’s another feature of BGP that all but guarantees asymmetric routing. Consider figure 3, which shows ASes 100 and 500 that interconnect in two locations: west and east. A customer of AS100 in location west communicates with a customer of AS500 in location east. The default behavior of BGP in this situation is to hand off the traffic to the neighboring AS at the first possible opportunity. (“Hot potato” or “early exit” routing.) So the traffic from AS100 to AS500 flows to AS500 in location west, and the return traffic from AS500 to AS100 is handed off in location east. The reasons for this behavior are the “prefer eBGP paths over iBGP paths” rule and the “prefer the path with the lowest IGP cost” rule in the BGP path selection algorithm.

In the situation in figure 2, the fact that routing is asymmetric is easily determined by looking at the AS paths: if the AS paths in the two directions show the same ASes, packets are routed symmetrically; if not, asymmetrically. But in the situation shown in figure 3, the AS paths don’t indicate asymmetric routing. But traceroutes in the two directions will show differences, assuming that the routers that show up in the traceroutes have a DNS name that indicates their location. (The addresses in the traceroutes will be different in both directions even with symmetric routing because the traceroutes hit different interfaces with different addresses in the two directions.)

The Role of BGP in Asymmetric Routing

BGP and Path Selection

BGP’s path selection process often contributes to asymmetric routing. BGP prefers eBGP paths over iBGP paths and selects routes based on the lowest IGP cost. It leads to scenarios where the forward and reverse paths differ. This is especially evident in “hot potato” routing, where traffic is handed off to the neighboring AS at the earliest possible point, resulting in asymmetric paths.

Hot Potato Routing is a strategy used by internet service providers (ISPs) and network operators to efficiently manage and forward internet traffic. The goal is to quickly pass data traffic to its next destination, much like swiftly passing a hot potato to avoid getting burnt. By doing this, the data spends minimal time within the network, which helps to reduce latency and improve overall network performance. The primary objective is to offload the data as quickly as possible to optimize the flow and speed of Internet traffic.

Detection and Monitoring Asymmetric Routing

Consider two autonomous systems (AS), AS10 and AS50, interconnected at multiple locations.

Figure 4. Network Topology with Asymmetric Routing

Traffic from a server (Server50) in the eastern location of AS50, destined for a server (Server10) in the western location of AS10, will be handed off at the nearest interconnection point (East50). The next-hop BGP router IP address in the BGP table of the router East50 is therefore 192.168.30.10, which is the BGP speaker East10 (Figure 5).

Figure 5. BGP Table of East50 with East10 as Exit Point for ICMP Echo Request to 10.10.10.10

The return traffic from AS10 to AS50 follows the same logic but in reverse, using a different path with West10 as the exit point. Figure 6 depicts the BGP table of West10, with the next-hop IP address 192.168.20.50, which is the BGP speaker West50.

Figure 6. BGP Table of West10 with West50 as Exit Point for ICMP Echo Reply to 50.50.50.50

As a result, forward traffic and return traffic take different path, effectively creating asymmetric routing. This approach helps prevent congestion and overuse within the ISP’s own network. By doing so, it reduces the chances of delays and packet loss, ensuring smoother and more efficient data transmission.

Inspecting the routing tables on network devices along the path between the source and destination is a valuable technique for gaining deeper insights into the routes that packets take. By accessing the routing tables on each device, we can compare the routes for traffic in both directions. It allows you to identify points where the paths diverge, which can contribute to asymmetric routing.

Asymmetric routing can also be detected using the traceroute utility, which shows the path packets take to reach their destination. The path packets take from Server10 to Server50 is depicted in Figure 7.

Figure 7. Checking Path from Server50 to Server10 with Traceroute

To get a complete picture, we need to run traceroute from the destination back to the source. This may require access to the destination machine. In order to achieve this, we will use the appropriate traceroute command targeting the original source IP 50.50.50.50 (Figure 8).

Figure 8. Checking Path from Server10 to Server50 with Traceroute

Comparing the paths shown in both traceroute outputs with a network topology, we will get information if asymmetric routing is in place. In our case, the packets are taking different paths in the forward and return directions, which is a clear indicator of asymmetric routing.

Is asymmetric routing bad?

Asymmetric routing can have both advantages and disadvantages, depending on the specific scenario and network configuration. Advantages of asymmetric routing include increased network capacity and improved load balancing. By distributing traffic across multiple paths, network administrators can optimize network performance and ensure that no single path becomes overloaded or congested.

However, asymmetric routing can also have some drawbacks. One of the main issues is that it can cause problems with network security and traffic filtering. Some network devices may not be able to properly handle asymmetric traffic, leading to dropped packets or other issues. Additionally, asymmetric routing can complicate network troubleshooting and diagnostics, making it more difficult to identify and resolve issues.

Asymmetric routing is not inherently “bad”, but it is important for network administrators to be aware of its potential advantages and disadvantages and carefully consider how it will impact their network architecture and performance.

When does asymmetric routing become problematic?

We now know how asymmetric routing happens, but not why it can be problematic. For that, consider figure 9.

Figure 9. Asymmetric routing and firewalls

In this network, there are two routers that connect to different external networks. Each router then connects to a firewall in order to reach the internal network. And what happens in this network setup is that traffic that enters the network through router 2 is then handled by firewall 2. However, outgoing traffic is handled by firewall 1. Most firewalls are configured to observe traffic in the outgoing direction and then automatically allow through the incoming packets belonging to the same TCP session. So firewall 1 is looking out for the return traffic matching the earlier outgoing traffic, but that traffic never shows up at firewall 1. Firewall 2, on the other hand, doesn’t know that incoming packets match earlier outgoing packets, as it never saw those outgoing packets, and thus blocks those incoming packets. As a result, all destinations that are routed asymmetrically are unreachable.

The solution to this problem is to adjust the placement of the firewalls or internal routing such that traffic in both directions flows through the same firewall, even if incoming traffic enters the network through a different router than the router that handled the matching outgoing traffic. This is shown in figure 10.

Figure 10. Asymmetric routing is hidden from the firewalls

It’s important to realize that if multiple paths are used at the same time, it is not possible to make BGP always select symmetric paths. The only BGP configuration that avoids asymmetric paths is a primary/backup configuration, where under normal circumstances only the primary path is used and the backup path doesn’t handle any traffic at all. See our blog post on BGP communities for an example of how to set up a BGP primary/backup configuration. But simply placing firewalls closer to the systems they protect rather than at the edge of the network will usually avoid problems related to asymmetric routing.

Addressing Asymmetric Routing

While completely avoiding asymmetric routing in BGP is nearly impossible, certain configurations can mitigate its impact. For instance, a primary/backup configuration can ensure traffic predominantly uses a single path. The secondary path is then reserved for failover scenarios. 

Placing firewalls closer to the systems they protect rather than at the edge of the network will usually avoid problems related to asymmetric routing. 

Additionally, some firewalls allow to permit asymmetric routing in order to avoid blocking response packets when corresponding request packets are not found. Let’s say that asymmetric routing is enabled on the FortiGate firewall. When non-SYN packets are received and the session does not exist. In such cases, all packets are passed to the CPU and the FortiGate does not lookup for matching firewall policies. Since no policy is matched, the packet is simply forwarded based on the routing table and the firewall acts as a router that only makes routing decisions. No security inspection will be performed [1]. 

Transitioning from Asymmetric to Symmetric Routing

Network Design Adjustments: Altering the network topology to minimize the divergence of traffic paths can help achieve more symmetric routing. This might involve restructuring the network to create more direct routes between endpoints.

Routing Policies: Implementing specific routing policies that favor symmetric paths can also be effective. For example, adjusting BGP local preference and MED (Multi-Exit Discriminator) values can influence the path selection process to favor symmetric routing.

Conclusion

Asymmetric routing is an integral part of modern network architecture, particularly in BGP-enabled environments. While it can bring benefits like enhanced network capacity and better load balancing, it also introduces challenges related to network security and troubleshooting.

By understanding the role of BGP in contributing to asymmetric routing and employing strategies to manage its impact, network administrators can strike a balance between the advantages and drawbacks of this phenomenon. Whether transitioning to more symmetric routing or leveraging the benefits of asymmetric paths, a thorough understanding of network design, routing policies, and security considerations is essential for optimizing network performance and reliability.