Situation
I’ve noticed that Cisco NX-OS prioritizes a locally redistributed route (from OSPF to BGP) over a route learned from an external BGP (eBGP) peer. However, when the route is cleared, the system correctly favors the learned BGP route.
Introduction
In the intricate networking world, routing protocols are crucial in establishing and maintaining seamless connectivity between different network segments. A comprehensive understanding of the interactions between these protocols and their collective contribution to the Routing Information Base (RIB) is vital. In this blog post, we’re going to delve into the interplay between Open Shortest Path First (OSPF), Enhanced Interior Gateway Routing Protocol (EIGRP), Border Gateway Protocol (BGP), and the RIB. The focus will be on a complex phenomenon called a race condition.
Understanding Race Conditions in Route Selection
The term race condition in networking describes a situation where the outcome depends on the sequence or timing of uncontrollable events. In my case, the timing of BGP learning a route versus the redistribution of that route by OSPF or EIGRP can create different outcomes. This inconsistency arises due to the way BGP handles route preference.
By design, BGP favors routes originating from the local system (including those redistributed locally by OSPF or EIGRP) over those learned from external BGP peers. This preference is determined by the weight attribute of the route, with locally originated BGP routes assigned a default weight of 32768, while other routes are assigned a weight of 0.
| Scenario | BGP Weight |
|---|---|
| Route from eBGP peer | 0 |
| Route from iBGP peer | 0 |
| Route redistributed into BGP | 32768 |
| Route originated on local router | 32768 |
Note: Weight can be manually configured, with higher values being more preferred. The weight range is from 0 to 65,535.
Example Scenario 1
Consider a scenario where the RIB already contains a route learned by OSPF or EIGRP. Before BGP learns the same route from an external peer, it redistributes it from OSPF or EIGRP. Given the inherent preference for locally originated routes, the BGP best-path algorithm selects the locally redistributed route as the best, thus retaining the OSPF or EIGRP route in the RIB, even after learning the same route from an external BGP peer.
Example Scenario 2
In contrast, let’s consider a situation where the RIB does not contain the OSPF or EIGRP learned route before BGP learning it from an external peer. As there’s no competing route, the BGP best-path algorithm will naturally select the externally learned BGP route and install it into the RIB with an Administrative Distance (AD) of 20. Later, even if OSPF or EIGRP tries to introduce their route into the RIB, they will fail due to their higher AD (110 for OSPF and 90 for EIGRP).
This discrepancy in outcomes under identical configurations, determined by the initial state of the RIB, is what we identify as a race condition.
The Role of Protocols and the RIB
This interaction has three critical components: BGP, the IGPs (OSPF and EIGRP), and the RIB. Each operates based on its momentary state.
OSPF and EIGRP remain consistent. Their internal algorithms (Shortest Path First for OSPF and DUAL for EIGRP) always produce the same results, irrespective of the state of BGP or the RIB. Their routes might not always be selected as best by the RIB, but this does not affect their internal operations.
BGP, however, is more susceptible to change due to the influence of route redistribution. Its best-path algorithm’s results depend on the available routes, favoring a locally injected route over any other path. This preference also influences the AD value associated with the route in the RIB.
It’s important to clarify that neither OSPF, EIGRP, nor BGP compare or use ADs in their internal operations. In the case of redistribution, BGP determines that the OSPF or EIGRP route, redistributed into BGP, is superior due to its higher weight. The AD assigned to routes offered to the RIB by BGP is a result, not an input, of its best-path selection process.
The Perils of Redistribution
One scenario to avoid in routing is circular redistribution, which can lead to routing loops and instability. A protocol won’t attempt to install a route it has redistributed back into the RIB, as this could change the route’s origin protocol, potentially causing route flapping.
Potential Issues Caused by Race Conditions
- Traffic loss or blackholing – Packets forwarded based on inconsistent routing information may get dropped or blackholed. This leads to a loss of connectivity to the destination.
- Suboptimal routing – Traffic may take longer than necessary while the protocols converge, leading to increased latency.
- Congestion and packet delay – Incorrect traffic forwarding can congest links that would not normally see that traffic flow. This leads to queuing delays.
- Routing loops – Temporary forwarding loops can occur between routers with inconsistent routing tables during convergence. This results in severe packet delay and loss.
- Instability – The inconsistent routing state may trigger routing changes back and forth, leading to route flaps and further stability issues.
- Protocol oscillations – Oscillations between IGP and BGP path selections can prevent both protocols from fully converging.
- Prefix hijacks – If a prefix is accidentally leaked into the IGP before BGP convergence, it could hijack that prefix’s traffic.
- Security issues – Incorrect routing and protocol instability leave the network vulnerable to misdirecting traffic for malicious purposes.
How to Prevent Racing Conditions
- Increase BGP timers – Slowing down BGP convergence by increasing timers like hold time gives BGP more time to converge before the IGP routes take effect.
- Use IGP route filters – Filtering BGP learned routes from being installed into the IGP routing table prevents the IGP from using inconsistent information.
- Tune IGP metrics – Adjust IGP metrics to prefer internal IGP paths over BGP paths to avoid the IGP switching to BGP routes too early.
- Advertise BGP prefixes into IGP with delay – Introduce a delay when advertising BGP prefixes to the IGP to allow BGP to converge first.
- Implement BGP PIC – Use BGP Prefix Independent Convergence to avoid flaps of BGP paths when the IGP topology changes.
- Prefix deaggregation – Avoid advertising specific prefixes and advertise aggregates to minimize instability.
- Reduce IGP convergence times – Tune IGPs like OSPF and IS-IS to converge faster and reduce the race window.
- Route damping – Use route damping to suppress unstable routes and prevent route flaps.
- Optimize network design – Minimize asymmetric paths between IGP and BGP by optimal network design and routing policy.
The key is tuning the interaction between the two protocols – either temporarily delaying the IGP from using BGP routes or speeding up BGP convergence times. Proper network design and routing policies also help avoid race conditions.
Conclusion
Understanding the intricate interplay of OSPF, EIGRP, BGP, and the RIB is critical for network administrators and engineers. While these interactions can be complex, a deep understanding of each protocol’s behavior and the factors influencing the RIB’s selection process can facilitate network design and troubleshooting. The concept of race conditions underlines the importance of route learning and redistribution timing, adding another layer of complexity to the fascinating world of routing.