Recent disruptions to two undersea internet cables in the Baltic Sea have yet again...
2.8.1 Specific PBR configuration scenarios
PBRs can be setup in multiple ways, depending on the existing network infrastructure.
We will assume the following IP addresses/networks are used:
- 10.0.0.0/24 – used on the IRP server as well as the probing VLANs
-
- 10.0.0.2/32 – main IRP server IP address
10.0.0.3-10.0.0.5 – probing IP addresses
10.0.0.250-10.0.0.254 – router-side IP addresses for the probing VLANs
10.0.1.0/24 – used for GRE tunnel interfaces, if needed
10.10.0.0/24 – real edge routers IP addresses
- 10.0.0.2/32 – main IRP server IP address
- 10.11.0.0/30 – BGP session with the 1st provider, 10.11.0.1 being the ISP BGP neighbor IP
10.12.0.0/30 – BGP session with the 2nd provider, 10.12.0.1 being the ISP BGP neighbor IP
10.13.0.0/30 – BGP session with the 3rd provider, 10.13.0.1 being the ISP BGP neighbor IP
- Vlan 3 – the probing Vlan
eth0 – the probing network interface on the IRP server
In a production network, please change the IP addresses used in these examples to the actual addresses assigned and used in the network. Same goes for the Vlan interface.
Brocade routers use Cisco compliant configuration commands. Unless otherwise noted, the Cisco configuration examples will also work on Brocade devices.
Case 1: Single router, two providers, and separate probing Vlan. #
10.0.0.1/32 is configured on the edge router, on the probing vlan interface (ve3).
In this case, a single PBR rule should be enforced on the Edge router for each of the probing IP addresses.
access-list 1 permit ip host 10.0.0.3 access-list 2 permit ip host 10.0.0.4 ! route-map irp-peer permit 10 match ip address 1 set ip next-hop 10.11.0.1 set interface Null0 ! route-map irp-peer permit 20 match ip address 2 set ip next-hop 10.12.0.1 set interface Null0 ! interface ve 3 ip policy route-map irp-peer
Route-map entries with the destination set to Null0 interface are used for preventing packets to flow through the default route in the case that the corresponding next-hop does not exist in the routing table (typically when a physical interface administrative/operational status is down).
Cisco ASR9000 routers running IOS XR use a different PBR syntax.
configure
ipv4 access-list irp-peer
10 permit ipv4 host 10.0.0.3 any nexthop1 ipv4 10.11.0.1 nexthop2 ipv4 169.254.0.254
11 permit ipv4 host 10.0.0.4 any nexthop1 ipv4 10.12.0.1 nexthop2 ipv4 169.254.0.254
end
router static
address-family ipv4 unicast
169.254.0.254 Null0
end
interface FastEthernet1/1
ipv4 access-group irp-peer ingress
end
For Juniper routers, a more complex configuration is required:
[edit interfaces]
xe-0/0/0 {
unit 3 {
family inet {
filter {
input IRP-policy;
}
}
}
}
[edit firewall]
family inet {
filter IRP-policy {
term irp-peer1 {
from {
source-address 10.0.0.3/32;
}
then {
routing-instance irp-isp1-route;
}
}
term irp-peer2 {
from {
source-address 10.0.0.4/32;
}
then {
routing-instance irp-isp2-route;
}
}
term default {
then {
accept;
}
}
}
}
[edit]
routing-instances {
irp-isp1-route {
instance-type forwarding;
routing-options {
static {
route 0.0.0.0/0 next-hop 10.11.0.1;
}
}
}
irp-isp2-route {
instance-type forwarding;
routing-options {
static {
route 0.0.0.0/0 next-hop 10.12.0.1;
}
}
}
}
routing-options {
interface-routes {
rib-group inet irp-policies;
}
rib-groups {
irp-policies {
import-rib [ inet.0 irp-isp1-route.inet.0 irp-isp2-route.inet.0 ];
}
}
}
PBR configuration on Vyatta routers.
Unfortunately, prior to and including VC6.4, Vyatta does not natively support policy-based routing. Thus, the PBR rules should be configured using the standard Linux
ip toolset. To make these rules persistent, they should be also added to /etc/rc.local on the Vyatta system.Listing 2.18: Vyatta IPv4 PBR configuration example
ip route add default via 10.11.0.1 table 101 ip route add default via 10.12.0.1 table 102 ip rule add from 10.0.0.3 table 101 pref 32001 ip rule add from 10.0.0.4 table 102 pref 32002
Vyatta versions VC6.5 and up, natively support source-based routing. The following example can be used:
Listing 2.19: Vyatta (VC6.5 and up) IPv4 PBR configuration example
# Setup the routing policy: set policy route IRP-ROUTE set policy route IRP-ROUTE rule 10 destination address 0.0.0.0/0 set policy route IRP-ROUTE rule 10 source address 10.0.0.3/32 set policy route IRP-ROUTE rule 10 set table 103 set policy route IRP-ROUTE rule 20 destination address 0.0.0.0/0 set policy route IRP-ROUTE rule 20 source address 10.0.0.4/32 set policy route IRP-ROUTE rule 20 set table 104 set policy route IRP-ROUTE rule 30 destination address 0.0.0.0/0 set policy route IRP-ROUTE rule 30 source address 0.0.0.0/0 set policy route IRP-ROUTE rule 30 set table main commit # Create static route tables: set protocols static table 103 route 0.0.0.0/0 nexthop 10.11.0.1 set protocols static table 104 route 0.0.0.0/0 nexthop 10.12.0.1 commit # Assign policies to specific interfaces, Vlan 3 on eth1 in this example: set interfaces ethernet eth1.3 policy route IRP-ROUTE # Verify the configuration: show policy route IRP-ROUTE show protocols static show interfaces ethernet eth1.3
Case 2: Two edge routers, two providers, and a separate probing Vlan. #
Following IP addresses are configured on the routers:
–
10.0.0.251 is configured on R1, VE3–
10.0.0.252 is configured on R2, VE3
To reduce the number of PBR rules to one per each router, additional source based routing rules must be configured on the IRP server.
ip route add default via 10.0.0.251 table 201 ip route add default via 10.0.0.252 table 202 ip rule add from 10.0.0.3 table 201 pref 32101 ip rule add from 10.0.0.4 table 202 pref 32102
Refer to OS configuration manual for configuration guidelines.
Router configuration looks similar to the previous case. A Cisco/Brocade example will be provided.
Some Brocade routers/switches have PBR configuration limitations. Please refer to the “Policy-Based Routing” → “Configuration considerations” section in the Brocade documentation for your router/switch model.
For example, BigIron RX Series of switches do not support more than 6 instances of a route map, more than 6 ACLs in a matching policy of each route map instance, and more than 6 next hops in a set policy of each route map instance.
On the other hand, some Brocade CER/CES routers/switches have these limits raised up to 200 instances (depending on package version).
#Router R1 access-list 1 permit ip host 10.0.0.3 ! route-map irp-peer permit 10 match ip address 1 set ip next-hop 10.11.0.1 set interface Null0 ! interface ve 3 ip policy route-map irp-peer #Router R2 access-list 1 permit ip host 10.0.0.4 ! route-map irp-peer permit 10 match ip address 1 set ip next-hop 10.12.0.1 set interface Null0 ! interface ve 3 ip policy route-map irp-peer
Case 3: Complex network infrastructure, multiple routers, no probing VLAN #
In specific complex scenarios, traffic from the IRP server should pass multiple routers before getting to the provider. If a separate probing Vlan cannot be configured across all routers, GRE tunnels from IRP to the Edge routers should be configured.
It is also possible to configure the PBRs without GRE tunnels, by setting PBR rules on each transit router on the IRP ↔ provider path.Brocade routers do not support PBR set up on GRE tunnel interfaces. In this case the workaround is to configure PBR on each transit interface towards the exit edge router(s) interface(s).
GRE Tunnels configuration
modprobe ip_gre ip tunnel add tun0 mode gre remote 10.10.0.1 local 10.0.0.2 ttl 64 dev eth0 ip addr add dev tun0 10.0.1.2/32 peer 10.0.1.1/32 ip link set dev tun0 up
#/etc/sysconfig/network-scripts/ifcfg-tun0 DEVICE=tun0 TYPE=GRE ONBOOT=yes MY_INNER_IPADDR=10.0.1.2 MY_OUTER_IPADDR=10.0.0.2 PEER_INNER_IPADDR=10.0.1.1 PEER_OUTER_IPADDR=10.10.0.1 TTL=64
Refer to OS configuration manual for configuration guidelines.
set interfaces tunnel tun0 set interfaces tunnel tun0 address 10.0.1.1/30 set interfaces tunnel tun0 description "IRP Tunnel 1" set interfaces tunnel tun0 encapsulation gre set interfaces tunnel tun0 local-ip 10.10.0.1 set interfaces tunnel tun0 remote-ip 10.0.0.2
interface Tunnel0 routers ip address 10.0.1.1 255.255.255.252 tunnel mode gre ip tunnel source Loopback1 tunnel destination 10.0.0.2
interfaces {
gr-0/0/0 {
unit 0 {
tunnel {
source 10.0.0.2;
destination 10.10.0.1;
}
family inet {
address 10.0.1.1/32;
}
}
}
}
The above configuration is for the first edge router (R1). One more GRE tunnel should be configured on the 2nd router (R2).
As soon as the GRE tunnels are configured, the source-based routing and the PBR rules should be configured, similar to the previous section.
ip route add default dev tun0 table 201 ip route add default dev tun1 table 202 ip route add default dev tun2 table 203 ip rule add from 10.0.1.2 table 201 pref 32101 ip rule add from 10.0.1.6 table 202 pref 32102 ip rule add from 10.0.1.10 table 202 pref 32103
Listing 2.28: IRP server IPv4 source-based routing via GRE using standard CentOS configuration files
#/etc/sysconfig/network-scripts/route-tun0: default dev tun0 table 201 default dev tun1 table 202 default dev tun2 table 203 #/etc/sysconfig/network-scripts/rule-tun0: from 10.0.1.2 table 201 pref 32101 from 10.0.1.6 table 202 pref 32102 from 10.0.1.10 table 203 pref 32103
Refer to OS configuration manual for configuration guidelines.Router configuration looks similar to the previous cases. A Cisco/Brocade example will be provided.
#Router R1 access-list 1 permit ip host 10.0.1.2 access-list 2 permit ip host 10.0.1.6 ! route-map irp-peer permit 10 match ip address 1 set ip next-hop 10.11.0.1 set interface Null0 ! route-map irp-peer permit 20 match ip address 2 set ip next-hop 10.12.0.1 set interface Null0 ! interface Tunnel0 ip policy route-map irp-peer interface Tunnel1 ip policy route-map irp-peer #Router R2 access-list 1 permit ip host 10.0.1.10 ! route-map irp-peer permit 10 match ip address 1 set ip next-hop 10.13.0.1 set interface Null0 ! interface Tunnel0 ip policy route-map irp-peer
Case 4: Internet Exchanges configuration examples #
The PBR rules for Cisco routers/switches are generated by IRP, following the template below.
Listing 2.30: Cisco IPv4 PBR configuration template
!--- repeated block for each peering partner no route-map <ROUTEMAP> permit <ACL> no ip access-list extended <ROUTEMAP>-<ACL> ip access-list extended <ROUTEMAP>-<ACL> permit ip host <PROBING_IP> any dscp <PROBING_DSCP> route-map <ROUTEMAP> permit <ACL> match ip address <ROUTEMAP>-<ACL> set ip next-hop <NEXT_HOP> set interface Null0 !--- block at the end of PBR file interface <INTERFACE> ip policy route-map <ROUTEMAP>
The “<>” elements represent variables with the following meaning:
- <ROUTEMAP> represents the name assigned by IRP and equals the value of the Route Map parameter in PBR Generator (“irp-ix” in Figure 4)
- <ACL> represents a counter that identifies individual ACL rules. This variable’s initial value is taken from ACL name start field of PBR Generator and is subsequently incremented for each ACL
- <PROBING_IP> one of the configured probing IPs that IRP uses to probe link characteristics via different peering partners. One probing IP is sufficient to cover up to 64 peering partners
- <PROBING_DSCP> an incremented DSCP value assigned by IRP for probing a specific peering partner. This is used in combination with the probing IP
- <NEXT_HOP> represents the IP address identifying the peering partner on the exchange. This parameter is retrieved during autoconfiguration and preserved in Exchange configuration
- <INTERFACE> represents the interface where traffic conforming to the rule will exist the Exchange router. This is populated with the Interface value of PBR Generator
The PBR rules for Brocade non-XMR router/switches are generated by IRP, following the template below.
Listing 2.31: Cisco IPv4 PBR configuration template
!--- repeated block for each peering partner no route-map <ROUTEMAP> permit <ACL> no ip access-list extended <ROUTEMAP>-<ACL> ip access-list extended <ROUTEMAP>-<ACL> permit ip host <PROBING_IP> any dscp-matching <PROBING_DSCP> route-map <ROUTEMAP> permit <ACL> match ip address <ROUTEMAP>-<ACL> set ip next-hop <NEXT_HOP> set interface Null0 !--- block at the end of PBR file interface <INTERFACE> ip policy route-map <ROUTEMAP>
The “<>” elements represent variables with the same meaning as per Cisco example.
Brocade XMR routers use keyword “dscp-mapping” instead of “dscp-matching”.The PBR rules for Juniper routers/switches are generated by IRP, following the template below.
It is important to note that the different sections of the PBR rules (load replace/merge relative terminal) should be entered independently and not as a single file that is output by IRP. Also take note that the last group includes a ‘load merge’ combo and not a ‘load replace’ as the first three groups.Listing 2.32: Juniper IPv4 PBR configuration template
load replace relative terminal
[Type ^D at a new line to end input]
interfaces {
<INTERFACE> {
unit <INTERFACE_UNIT> {
family inet {
filter {
replace:
input <ROUTEMAP>;
}
}
}
}
}
load replace relative terminal
[Type ^D at a new line to end input]
firewall {
family inet {
filter <ROUTEMAP> {
replace:
term <ROUTEMAP><ACL> {
from {
source-address <PROBING_IP>;
dscp <PROBING_DSCP>;
}
then {
routing-instance <ROUTEMAP><ACL>-route;
}
}
...
replace:
term default {
then {
accept;
}
}
}
}
}
load replace relative terminal
[Type ^D at a new line to end input]
routing-instances {
replace:
<ROUTEMAP><ACL>-route {
instance-type forwarding;
routing-options {
static {
route 0.0.0.0/0 next-hop <NEXT_HOP>;
}
}
}
...
}
load merge relative terminal
[Type ^D at a new line to end input]
routing-options {
interface-routes {
replace:
rib-group inet <ROUTEMAP>rib;
}
rib-groups {
replace:
<ROUTEMAP>rib {
import-rib [ inet.0 <ROUTEMAP><ACL>-route.inet.0 ... ];
}
}
}
The “<>” elements represent variables with the following meaning:
- <INTERFACE> represents the interface where traffic conforming to the rule will exist the Exchange router. This is populated with the Interface value of PBR Generator
- <INTERFACE_UNIT> is the value of the Interface Unit parameter in PBR Generator
- <ROUTEMAP>represents the name assigned by IRP and equals the value of the Route Map parameter in PBR Generator
- <ACL> represents a combined counter like “00009” that identifies individual ACL rules. This variable’s initial value is taken from ACL name start field of PBR Generator and is subsequently incremented for each ACL
- <PROBING_IP> one of the configured probing IPs that IRP uses to probe link characteristics via different peering partners. One probing IP is sufficient to cover up to 64 peering partners
- <PROBING_DSCP> an incremented DSCP value assigned by IRP for probing a specific peering partner. This is used in combination with the probing IP
- <NEXT_HOP> represents the IP address identifying the peering partner on the exchange. This parameter is retrieved during autoconfiguration and preserved in Exchange configuration
Note that Juniper routers/switches need an additional parameter in order to correctly configure PBRs – Interface unit.
Verifying PBR configuration #
To ensure that the PBRs are properly configured on the router(s), the standard *NIX traceroute command can be used, by specifying each probing IP as the source IP address for tracing the route. The route should pass through a different provider (note the ISP BGP neighbor IP).
Listing 2.33: PBR validation using `traceroute`
root@server ~ $ traceroute -m 5 8.8.8.8 -nns 10.0.0.3 traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 10.0.0.1 0.696 ms 0.716 ms 0.783 ms 2 10.11.0.1 0.689 ms 0.695 ms 0.714 ms 3 84.116.132.146 14.384 ms 13.882 ms 13.891 ms 4 72.14.219.9 13.926 ms 14.477 ms 14.473 ms 5 209.85.240.64 14.397 ms 13.989 ms 14.462 ms root@server ~ $ traceroute -m 5 8.8.8.8 -nns 10.0.0.4 traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 10.0.0.1 0.696 ms 0.516 ms 0.723 ms 2 10.12.0.1 0.619 ms 0.625 ms 0.864 ms 3 83.16.126.26 13.324 ms 13.812 ms 13.983 ms 4 72.14.219.9 15.262 ms 15.347 ms 15.431 ms 5 209.85.240.64 16.371 ms 16.991 ms 16.162 ms
Another useful method for checking that the PBRs are properly configured is to use Explorer self check option. Make sure that the providers are added to IRP configuration before executing Explorer self check.
Listing 2.34: PBR validation using Explorer self check
root@server ~ $ /usr/sbin/explorer -s Starting PBR check PBR check failed for provider A[2]. Diagnostic hop information: IP=10.11.0.12 TTL=3 PBR check succeeded for provider B[3]. Diagnostic hop information: IP=10.12.0.1 TTL=3
In order to ensure that the PBRs are properly configured for Internet Exchanges, the method below can be used.
Listing 2.35: PBR validation using `iptables` and `traceroute`
root@server ~ $ iptables -t mangle -I OUTPUT -d 8.8.8.8 -j DSCP --set-dscp <PROBING_DSCP> root@server ~ $ traceroute -m 5 -nns <PROBING_IP> 8.8.8.8 traceroute to 8.8.8.8, 30 hops max, 60 byte packets 1 ... 2 ... 3 <NEXT_HOP> 126.475 ms !X^C
where
- <NEXT_HOP> is a Peering Partner’s next-hop IP address in IRP configuration
- <PROBING_DSCP> is a Peering Partner’s DSCP value in IRP configuration
- <PROBING_IP> is a Peering Partner’s probing IP address in IRP configuration
The first IP of the trace leaving client infrastructure should be on the Exchange and the next-hop should belong to the correct Peering Partner.
iptables rules should be deleted after all tests are done.
2.8.1.1 Current route detection #
Current route detection algorithm requires the explorer.infra_ips parameter to be configured. All the IP addresses and subnets belonging to the infrastructure should be added here. These IP addresses can be easily determined by using the
traceroute command.
2.8.1.2 Providers configuration #
Before Explorer starts probing the prefixes detected by the Collector, all providers should be configured.
For the complete list of provider settings please see the Provider section.Before configuring providers in IRP, the BGP sessions need to be defined, see Bgpd Configuration
Please note that some Brocade router models do not support SNMP counters per Vlan, therefore physical interfaces should be used for gathering traffic statistics.For the sample configuration below, let’s assume the following:
ISP1– the provider’s name
10.0.0.1– Router IP configured on the probing Vlan
10.0.0.3– Probing IP for ISP1, configured on the IRP server
10.11.0.1, 10.11.0.2– IP addresses used for the EBGP session with the ISP, 10.11.0.2 being configured on the router
400Mbps– the agreed bandwidth
1Gbps– the physical interface throughput
'public'– read-only SNMP community configured on R1
GigabitEthernet2/1– the physical interface that connects R1 to ISP1
As presented in Listing 2.36, all the parameters are pretty self-explanatory. Please check the BGP Monitoring section as well.
peer.1.95th = 400 peer.1.95th.bill_day = 1 peer.1.bgp_peer = R1 peer.1.cost = 6 peer.1.description = ISP1 peer.1.ipv4.next_hop = 10.11.0.1 peer.1.ipv4.probing = 10.0.0.3 peer.1.ipv4.diag_hop = 10.11.0.1 peer.1.ipv4.mon = 10.11.0.1 10.11.0.2 peer.1.limit_load = 1000 peer.1.shortname = ISP1 peer.1.snmp.interfaces = 1:GigabitEthernet2/1 peer.1.mon.ipv4.bgp_peer = 10.11.0.1 snmp.1.name = Host1 snmp.1.ip = 10.0.0.1 snmp.1.community = public
SNMP parameters validation #
To make sure that the SNMP parameters are correct, the `snmpwalk` tool can be run on the IRP server:
Listing 2.37: SNMP parameters validation
root@server ~ $ snmpwalk -v2c -c irp-public 10.0.0.1 ifDescr IF-MIB::ifDescr.1 = STRING: GigabitEthernet1/1 IF-MIB::ifDescr.2 = STRING: GigabitEthernet1/2 IF-MIB::ifDescr.3 = STRING: GigabitEthernet2/1 IF-MIB::ifDescr.4 = STRING: GigabitEthernet2/2 IF-MIB::ifDescr.5 = STRING: GigabitEthernet2/3 IF-MIB::ifDescr.6 = STRING: GigabitEthernet2/4 root@server ~ $ snmpwalk -v2c -c irp-public 10.0.0.1 ifIndex IF-MIB::ifIndex.1 = INTEGER: 1 IF-MIB::ifIndex.2 = INTEGER: 2 IF-MIB::ifIndex.3 = INTEGER: 3 IF-MIB::ifIndex.4 = INTEGER: 4 IF-MIB::ifIndex.5 = INTEGER: 5 IF-MIB::ifIndex.6 = INTEGER: 6