BGP Anycast Best Practices & Configurations

What is Anycast?

Anycast is not a new version of IP or the latest networking technology. The RFC 1546 describing Internet anycasting for IP has been with us since 1993. Anycast is a configuration method where multiple instances of a service share a single IP address. An anycast address is no different from any other IP address; it is not assigned from a specific range, such as a multicast address (224.0. 0.0 – 239.255.255.255).

Anycast and BGP

Anycast is tightly coupled with the BGP protocol, which ensures that the anycast route is advertised among autonomous systems (ASes) on the Internet. Packets to the anycast IP are routed through the shortest path to the server, reducing latency and improving service resiliency. If the server fails, the anycast route is withdrawn from the routing table, and a router forwards the client’s request through an alternate path to another server that is listening on the same anycast IP address.

Domain Name System (DNS) Anycast

Services based on connection-less UDP network transport protocol are ideal for use with Anycast. For instance, all root DNS servers are implemented as a cluster of nodes using Anycast addressing [1]. Each nameserver advertises the same anycast address to the router, and BGP distributes the route accordingly. All nodes can handle DNS requests, but the query is forwarded to exactly one node, the best node, which is the closest node determined by BGP.

Let’s illustrate how anycast routing works on the network shown in Figure 1. DNS servers may be located in different geographical areas (Figure 1). The BGP speaker R1 installs a route to anycast IP 35.177.2.19 via R2, since the number of BGP hops over R2 is less than via R3.

Figure 1 – DNS Resolvers Sharing Same Anycast IP

R1 considers a path via R3 as an alternative route to the same destination, although there are two routes to different destinations with the same address. Figure 2 shows the network topology from the perspective of R1.

Figure 2 – Network Topology from R1 Perspective

Anycast Routing

Routing to Anycast destination can be determined by one of two schemes. In a network layer anycast scheme, the router selects a destination optimal for the user based on the number of hops.

In an application layer anycast scheme, a router can also take into account additional calculations, such as server availability, time to response, etc. For example, the server may have implemented a mechanism that periodically checks the status of the service. If the service does not work, the server sends a withdraw message to the neighbor.

We will configure and simulate both routing schemes in the network shown in Figure 3.

Figure 3 – DNS Resolvers Sharing Anycast IP

Anycast Configuration

Our network consists of Debian Linux servers running bind9 DNS software (Figure 3). Both servers are configured with the same IP address 52.37.97.240/32 (Figure 4).

debian@nameserver-01:~$ cat /etc/network/interfaces

Figure 4 – Anycast IP Configured on Loopback

When the loopback interface is used for anycast addressing, routers can continue to advertise anycast addresses without being affected by hardware malfunctions (Figure 5).

debian@nameserver-01:~$ ifconfig lo:1

Figure 5 – Anycast IP Configured on Loopback

To check if DNS resolution is working locally, we will translate noction.com locally from both servers (Figure 6):

debian@nameserver-01:~$ dig noction.com @127.0.0.1

Figure 6 – Checking DNS Resolution from Nameserver-01

Both name servers have the FRR protocol suite (FRR) installed and the BGP daemon running. The statement “network 52.37.97.240/32” on both servers ensures that the anycast host route is advertised to the neighbor Cisco router (Figure 7).

Figure 7 – BGP Configured on Nameserver-01

Both Cisco routers are in AS 64500 with an eBGP connection to their neighbors. The network statement ensures that nameservers are aware of the client’s network (192.168.x./24) (Figure 8).

Figure 8 – BGP Configuration on Cisco-01 Router

There are currently two paths to the anycast IP 52.37.97.240/32 installed in the BGP table of the cisco-01. The best path is through the next-hop 10.0.0.6, and this path is also installed in the routing table. The alternate path is through the next-hop 10.0.0.10 (Figure 9). As a result, DNS requests to 52.37.97.240 received by cisco-01 are forwarded to nameserver-01.

Figure 9 – BGP Table of Cisco-01

The router cisco-02 forwards DNS queries to nameserver-02 and has installed an alternate path through 10.0.0.6 (nameserver-01) into the BGP table (Figure 10).

Figure 10 – BGP Table of Cisco-02

DNS Service Testing When Network Layer Anycast Scheme Is In Use

To check if the nameserver-01 responds to DNS queries sent from client-01, we run the dig command on the client (Figure 11):

tc@client-01:~$ dig noction.com @52.37.97.240

Figure 11 – DNS Query Sent from Client-01 to Nameserver-01

The output of the trace command confirms that it is the nameserver-01 that responds to the query from client-01, and cisco-01 has selected anycast destination based on the number of BGP hops (Figure 12).

Figure 12 – Checking Path Between Client-01 and Nameserver-01

Now we turn off nameserver-01 and check if the DNS resolution still works. Once the network is converged, we can still translate noction.com to an IP address (Figure 13).

tc@client-01:~$ dig noction.com @52.37.97.240

Figure 13 – DNS Query from Client-01 to Nameserver-02

However, nameserver-01 is down, so nameserver-02 is now answering client-01 queries. The path changed to client-01 -> cisco-01-> cisco-02 and finally nameserver-02 with hops in the path (Figure 14).

Figure 14 – Checking Path Between Client-01 and Nameserver-02

DNS queries are sent to nameserver-02 because cisco-01 removed the path to nameserver-01 from the routing table.
However, if we stop the bind service on the nameserver-01, FRRouting on that server will continue to announce the anycast route to cisco-01. In this case, DNS queries from client-01 still reach nameserver-01, but no DNS response will be sent to client-01, and domain translation will fail (Figure 15).

Figure 15 – DNS translation Fails When Bind Service is Stopped on Nameserver-01

To overcome the disadvantage of a network layer anycast scheme, we must use an application layer anycast scheme that takes into account the availability of DNS service.

ExaBGP Configuration to Support Layer Anycast Scheme

We will deploy ExaBGP to detect the anycasted DNS service failures and to handle them gracefully.

Let’s assume that ExaBGP is installed correctly. Our next step is to remove the BGP configuration from frr for both nameservers. ExaBGP can manipulate BGP updates, but it does not install the received routes into the Linux kernel forwarding table. Therefore, we will create static routes on both servers so they can route packets to their clients.

nameserver-01(config)# no router bgp 64501
nameserver-01(config)# ip route 192.168.1.0/24 10.0.0.5
nameserver-01(config)# ip route 192.168.2.0/24 10.0.0.5
nameserver-01(config)# ip route 10.0.0.0/30 10.0.0.5

nameserver-02(config)# no router bgp 64502
nameserver-02(config)# ip route 192.168.2.0/24 10.0.0.9
nameserver-02(config)# ip route 192.168.1.0/24 10.0.0.9
nameserver-01(config)# ip route 10.0.0.0/30 10.0.0.9

We need to create a script /etc/exabgp/noction-dns-check.sh to check DNS response from the local server. If noction.com can be translated, the first condition is matched, and the script writes the command “announce route 52.37.97.240 next-hop 10.0.0.5” to STDOUT. ExaBGP will read the output and turn it into BGP messages. If the condition is not matched, the second statement is echoed to STDOUT, and ExaBPG withdraws the anycast route.

Below is the BASH script for nameserver-01. Create the same script for nameserver-02, but be sure to change the IP address of the neighbor to 10.0.0.10. Copy the script to the directory /etc/exabgp/ and give it execute privileges.

#!/usr/bin/bash

while true; do
    /usr/bin/dig noction.com @52.37.97.240 > /dev/null;
    if [ "$?" == 0 ]; then
          echo "announce route 52.37.97.240 next-hop 10.0.0.6"
    else       
          echo "withdraw route 52.37.97.240 next-hop 10.0.0.6"
    fi
    sleep 1
  done

Now we need to create the ExaBGP configuration file /etc/exabgp/exabgp.conf for namserver-01. Change the neighbor IP address to 10.0.0.9, local-address to 10.0.0.10 and local-as to 64502 for nameserver-02.

process announce-routes {
  run /etc/exabgp/noction-dns-check.sh;
  encoder text;
}

neighbor 10.0.0.5 {
    local-address 10.0.0.6;
    local-as 64501;
    peer-as 64500; 

    api {
        processes [ announce-routes ];
   }

}

Finally, we create a unit file /etc/systemd/system/exabgp.service on both nameservers to ensure that exabgp runs as a service.

[Unit]
Description=ExaBGP
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf

[Service]
Environment=exabgp_daemon_daemonize=false
Environment=ETC=/etc
ExecStart=/usr/local/bin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID

[Install]
WantedBy=multi-user.target

After creating or modifying any unit files, we must tell systemd that we want it to look for new things.

debian@nameserver-01:~$ sudo systemctl daemon-reload

Our new service should be recognized at this point, but it will not run yet. We need to do two more things. First, tell systemd to enable it so that it will start every time we boot.

debian@nameserver-01:~$ sudo systemctl enable exabgp

Second, start it now.

debian@nameserver-01:~$ sudo systemctl start exabgp

DNS Service Testing When Application Layer Anycast Scheme Is In Use

Firstly, check the routing tables on the Cisco-01 router (Figure 16).

Figure 16 – BGP Table Cisco-01

Connect to client-01 and verify if we can translate noction.com (Figure 17).

Figure 17 – DNS Response from Nameserver-01 Received on Client-01

Now we will stop the bind9 service on the nameserver-01 server.

debian@nameserver-01:~$ sudo systemctl stop bind9

After a few seconds, nameserver-01 sends a BGP UPDATE message with withdrawn route 52.37.97.240/32 to cisco-01, indicating that the previously announced anycast prefix has become unreachable (Figure 18).

Figure 18 – BGP UPDATE Message

DNS Anycast IP 52.37.97.240/32 route with the next-hop 10.0.0.10 is installed in the BGP table of the device cisco-01 (Figure 19).

Figure 19 – BGP Table of Cisco-01

Figure 21 confirms that we can translate noction.com to the IP address and a DNS resolver nameserver-02 is three hops away from the client-01 (Figure 20).

tc@client-01:~$ dig noction.com @52.37.97.240

Figure 20 – Sending DNS Query from Client-01 to Anycast IP

Once we start the bind9 service on nameserver-01, nameserver-01 sends an UPDATE message to cisco-01, indicating that anycast routes have become reachable via 10.0.0.6 (Figure 21).

Figure 21 – Anycast Route Installation to Routing Table of Cisco-01 via Nameserver-01

DNS queries sent to anycast IP from client-01 will again be answered by nameserver-01 (Figure 22).

Figure 22 – Sending DNS Query from Client-01 to Anycast IP

Conclusion

Anycast is an IP address configured on multiple servers that provide the same service to clients.

The network layer failover service for anycast is simple and requires no special configuration. However, this concept fails when the server is reachable but not functional at the application layer. Therefore, we used ExaBGP to monitor an application state of the service and generate BGP messages based on this state. This allows the network to converge quickly, and queries are forwarded to the nearest fully functional server.