Oversubscription in Networking

In general, oversubscription is a subscription for more than what is available. Oversubscription represents an intentional business model and is a widespread practice in all areas of life. For instance, airlines rely on the fact that not all passengers typically arrive to take the actual flight, and some cancel their flights. Hence, companies usually sell more tickets than the available aircraft seats.

There are different types of oversubscriptions in networking. Dynamic IP and Port (DIPP) NAT oversubscription allows the reusability of a translated IP address and port. A device uses the same NAT IP address and port pair multiple times (8, 4, or 2 times) for connecting to different destinations when an oversubscription is enabled. By default, 64k sessions are allowed for a single public IP address. If oversubscription is enabled on the device, the maximum number of sessions is multiplied by the oversubscription rate. For example, the default limit of 64K concurrent sessions allowed, when multiplied by an oversubscription rate of 8, results in 512K concurrent sessions allowed. This permits customers to have fewer public IP addresses.

Another type of subscription is port oversubscription when switching bandwidth allocated to the switch port is less than the connection speed of the devices connected to the port. This may happen if the switch port has a particular connection speed, but it cannot achieve wire-rate performance.

Oversubscription in Three-tier Networks

Data centers and campus networks are engineered with oversubscription. For instance, oversubscription recommendation for the traditional three-tier model (access, distribution, and core layer) in a campus network is 20:1 for access ports on the access to distribution uplink (Figure 1). The oversubscription ratio for distribution to core links is 4:1. This three-tier design is highly oversubscribed with uplink bottlenecks and added latency for east-west traffic (traffic between devices in the data centers). Therefore, a spine-leaf model is commonly used in modern data centers, so the latency is on predictable levels, and the number of hops is minimized.

Figure 1 – Three-Tier Campus Network with Oversubscription

Oversubscription in Two-tiers Leaf-spine Networks

The two-tier Leaf-spine model, which is mainstream in modern data centers, overcomes the traditional three-tier network model limitation. The majority of network traffic in data centers is east to west, e.g., from compute server to storage located anywhere in the data center. In a three-tier model, traffic traverses through two aggregation switches and one core, while in Leaf-spine topology, it must only hop to a spine switch and another leaf switch. Therefore, latency is improved and bottleneck minimized in two-tier Leaf-spine architecture. The spine switches are on the top of the tier, and the leaf switches on the bottom tier with servers connected to leaf switches at the top of every rack.

Servers are connected to leaf switches only. There is no connection between leaf switches. The number of leaf switches depends on the number of required network interfaces for the server’s connection. Another leaf switch is added to the fabric with uplinks connected to all spine switches if more servers are needed. The number of leaf switch uplinks determines the number of spine switches, and port density on the spine switch limits the maximum number of leaf switches. However, the count of leaf switches cannot be random or unlimited. The acceptable oversubscription ratio should be 3:1 or even less e,.g., 2.5:1, to ensure there is no excessive bandwidth contention when all servers send traffic simultaneously. The oversubscription ratio increases with the number of servers in the fabric, and it is reduced by adding more spine switches to the fabric.

Figure 2 depicts 100G Spine-leaf network architecture. Let’ say we want to build a data center fabric with the goal of having 960 10G servers in one fabric with oversubscription 2.4:1. We have the leaf switches in the top of rack supporting 48 x 10GB ports for servers and 8 x 100G uplink ports. The spine switch supports 64 x 100G ports. To cover all 960 servers, we need 20 leaf switches (960 servers/ 48ports) and two leaf switches. Every leaf switch is attached to the spine with two 100G uplinks. The maximum number of servers is 960 at 2.4:1 oversubscription (48 x 10Gbps downlink to servers / 2 x 100Gbps uplink to the spine = 2.4).

Figure 2 – Two-tier Leaf-spine Network Topology with 960 10G Servers in Fabric with 2.4:1 Oversubscription

If we add another two leaf switches, the oversubscription ratio will be 1.2:1 (480G / 400G). This is close to 1:1, so there are no network bottlenecks; thus, leaf switches forward traffic with no packet loss. However, 1:1 oversubscription may result in excess capacity during non-peak times. We will likely never encounter a situation where all ports receive traffic at their maximum line rate at the same time.

A 1:1 oversubscription

The ideal network design tries to approach the 1:1 oversubscription but entirely depends on the applications, traffic patterns, and capacity needed by the administrators. When we estimate a network traffic oversubscription ratio for a new network, we need to assess the expected traffic. This includes understanding the service applications and features deployed on the network and determining the network services. For existing networks, close bandwidth usage monitoring with a NetFlow/sFlow analyzer is a must. Noction Flow Analyzer can be of help here. It is a great tool that provides insights into the volume and ratio of the east-west and north-south traffic and applications using the bandwidth. It enables engineers to optimize their networks and applications’ performance, control bandwidth utilization, and perform better network capacity planning. NFA supports NetFlow, J-Flow, sFlow, IPFIX, and NetStream. Priced at $299/month per license with no limit on the number of network devices, interfaces, or sites, NFA represents an affordable and cost-effective solution for your business.