BGP and Traffic Engineering Mechanisms for WISPs

Network traffic is forwarded based on the installed routes and their appropriate next-hops inside of a routing table. Although any IGP can be used, OSPF is often the first choice for WISP. Using OSPF, the equal-cost multi-path routing (ECMP) can be employed. ECMP is a mechanism that allows to select more than one path, so traffic is load balanced over multiple paths. However in a WISP world, the speed of the physical port does not equal the speed of the path between the two points (towers). It leaves some bandwidth of the higher speed link capacity unused even though ECMP is employed. WISP can tune OSPF and prefer some paths by changing the cost of the link so that the ECMP is not employed. However, the preferred path (with a lower cost) might be overloaded, when the additional towers are added to a WISP network while the not preferred path (with higher) cost is not used at all. This can make the situation even worse.

One might ask whether EIGRP can be used to solve the problem. As we know, EIGRP is a Cisco proprietary routing protocol that supports unequal cost path load balancing. The command variance n under the eigrp configuration multiplies the minimum metric n times. It makes EIGRP to include all routes that have a metric of less than or equal to the minimum metric. EIGRP will send more traffic over the better path and less traffic over the worse path proportionally. However, while EIGRP has been an open standard (RFC 7868) since 2016, it is not widely adopted by WISP vendors. For instance MikroTik, preferred by many WISPs for the good price/performance ratio has not implemented the EIGRP protocol yet. Since WISPs typically use equipments from multiple vendors, EIGRP is out of the game for a long time.

Kevin Myers comes up with a new concept that he calls OSPF transit fabric (TF). It represents a new approach to load balancing over unequal paths. He suggests to divide a physical link to multiple logical links by VLANs and let ECMP to balance traffic across logical links with the same cost. For instance, let’s have two PtP RF links between towers, one with the capacity of 900Mbps and the second link with 150Mbps capacity. If we create 8 VLANs on a link with a higher capacity and two VLANs on the link with a lower capacity, we will achieve 6 to 2 ratio in favor of the link with the higher capacity. OSPF then sees 8 equal cost next-hops for both links – six hops for the 900Mbps link and two next-hops for the 150Mbps link. Every new connection with a different source and destination address will be sent over the other logical link. The path across the link with the higher capacity will be preferred to the path with a lower capacity as the 6 from eight next-hops are on the link with the higher capacity. This allows both links to be fully utilized.

If there are more than 50 towers in a WISP, Kevin recommends to run eBGP on top of OSPF TF between the towers. In this design, BGP loopback addresses and VLAN subnets are advertised using OSPF, no VLANs subnets are advertised by eBGP. BGP is used to advertise customers’ prefixes. This design combines the benefits of OSPF TF and allows the eBGP traffic engineering. Thanks to BGP, the shortest path between towers can be chosen based on 11 matching attributes. The best path selection attributes can be overridden by BGP communities. For instance, we can tag a certain customer prefix with a BGP community number and let BGP to advertise the prefix to other towers along the path. Then we can match the community number inside a BGP advertisement and set a higher LOCAL_PREF to prefer particular next-hops along the path. Similarly, we can attach a different community number to other customer’s prefix and once matched, a local preference can be set to favor another next-hop. Traffic sent to those prefixes takes different path in a WISP network so the unequal P2P RF links can be fully utilized. Moreover, besides the traffic engineering communities, we can also use BGP communities for tower identification.

Conclusion:

eBGP has already found its way into the Data Centers several years ago. As Kevin demonstrates in his presentation, eBGP along with OSPF can be successfully combined to manage and identify traffic in a WISP network, while utilizing all available bandwidth of the links between towers.