In many routed-network tests, removing the preferred path produces a longer traffic interruption than restoring it. The difference comes from how the failure is detected, how alternate paths are installed, and how the routing change propagates end to end. In our own testing, failover took roughly 30 seconds of measured traffic loss while failback took one to two. This post explains why that asymmetry is common, and just as importantly, why the exact failover number should be measured rather than assumed.
The observation
During failover testing, withdrawing the primary path took roughly 30 seconds before traffic settled onto the backup. Restoring the primary took one to two seconds. Same routers, same routes, same links, yet very different recovery times depending on the direction of the change.
One honest caveat before the explanation: those 30 seconds are an observed end-to-end traffic recovery time, not a proven BGP convergence duration. The interval covers everything between the triggering event and the moment our measurement tool saw traffic flowing again: route withdrawal, best-path selection, forwarding updates, propagation through the provider’s network, return-path convergence, and even how the measurement itself declares “recovered.” Which of those consumed the time is a question for instrumentation, not intuition. We’ll come back to that.
The conceptual asymmetry, though, is real and general. Failover and failback are not mirror images of each other. Failover is a subtractive event, where the network must invalidate an existing path and make the remaining path usable across the whole forwarding chain. Failback is an additive event, where a more preferred path appears and can win as soon as its advertisements are processed. Cleanup is expensive; accepting an upgrade is cheap.
Failover: a pipeline, not an event
It’s tempting to think of failover as one action: “switch to the backup.” In reality it involves a chain of dependent work, and traffic isn’t restored until the slowest relevant piece completes end to end. The stages below have logical dependencies, though real implementations may batch or pipeline portions of the work across prefixes and devices. End-to-end recovery is determined by the slowest relevant control-plane, forwarding-plane, or downstream propagation component.
1. Detect: establish that the path is unusable. How long this takes depends entirely on the failure type. A silent failure (a path that just stops passing traffic) must be discovered, either by BFD in milliseconds or, without BFD or any lower-layer signal, by the BGP hold timer, which can consume 30+ seconds on its own. But an administrative shutdown or a local interface-down event skips detection entirely: the router already knows, and teardown begins immediately. More on this below, because it matters for interpreting test results.
2. Withdraw: tear down the routing state. The session or next-hop is declared down, and every prefix learned over the failed path must be withdrawn or invalidated. On sessions carrying large tables, walking thousands of prefixes takes real time, though far less than most people assume on modern hardware.
3. Best-path: re-run the decision process. For each affected prefix, BGP re-evaluates the remaining candidates and selects the backup as the new best path.
4. RIB/FIB: program the forwarding plane. The new best path is installed in the RIB, then pushed into the FIB in hardware. Until the FIB is reprogrammed, packets are still being forwarded (or blackholed) along the dead path. Technologies like BGP PIC exist precisely to shortcut this: backup forwarding state is precomputed and preinstalled, so the forwarding plane can flip without repeating per-prefix convergence.
5. Propagate: tell everyone else. Withdrawals and new advertisements ripple to upstream and downstream neighbors, each of which does its own selection and programming. In cloud and provider environments this stage can be significant, because you don’t control (or see) how quickly the far side’s control plane propagates the change. End-to-end recovery is only done when the farthest relevant device has updated its forwarding, in both directions of the traffic flow.
The key insight: fast detection does not guarantee equally fast end-to-end traffic recovery. If the session drops quickly but traffic remains interrupted, the remaining delay must be isolated across route withdrawal, downstream propagation, forwarding updates, and application or measurement behavior. Detection opens the gate; the rest of the recovery still has to be accounted for, by measurement.
Match the test to the failure you’re evaluating
Here’s the part that changes how you should read failover numbers, including ours. Not all “failures” exercise the same machinery. Our test used an administrative neighbor shutdown, and that is not a silent physical failure:
- The local router already knows the neighbor is being disabled.
- It does not need BFD to discover anything.
- It does not wait for the BGP hold timer.
- Session teardown and local route removal begin immediately.
| Test method | Detection mechanism | What it actually measures |
|---|---|---|
neighbor shutdown |
Immediate administrative event | Controlled route withdrawal and downstream convergence |
| Physical interface shutdown | Immediate local interface event | Link-state loss plus routing convergence |
| Silent remote/path failure | BFD or BGP hold timer | Failure detection plus everything above |
This distinction is essential for interpreting results. An administrative shutdown intentionally bypasses failure detection, so if traffic still takes 30 seconds to recover, those 30 seconds were not spent detecting anything. In our case that points the investigation at the later stages: downstream propagation through the provider’s control plane, stale-route or graceful-restart handling, return-path convergence, or the measurement method itself. Those are hypotheses to test with timestamps, not conclusions. Document your failure method whenever you publish or compare convergence numbers, because it determines which mechanisms even participated.
When slow failover is configured, not inherent: graceful restart
There’s another place recovery time can hide, and it’s worth checking before blaming any protocol or provider: features that intentionally delay convergence. The biggest one for BGP is graceful restart.
Graceful restart and BFD serve different purposes, and their interaction must be understood before both are enabled on the same Cisco BGP session. BFD is designed to identify an unusable forwarding path quickly and notify BGP. Graceful restart is designed for a different failure model: a control-plane restart in which the forwarding plane may still be capable of carrying traffic. In that case, a GR helper may temporarily retain the restarting peer’s routes as stale instead of withdrawing them immediately.
The two features are therefore not universally incompatible, but they can work against the intended outcome if the failure is misclassified. If the forwarding path is genuinely dead, stale-route retention can prolong blackholing. If only the BGP control plane restarted while forwarding remained intact, immediate withdrawal can create an unnecessary outage. Cisco IOS-XE includes platform- and feature-specific mechanisms for distinguishing some control-plane events from forwarding failures, including BFD control-plane-independence signaling on supported implementations. The practical rule is not “never combine BFD and graceful restart”; it is to enable the combination only when the exact IOS-XE behavior has been validated for the platform, software release, topology, and failure modes being tested.
Two diagnostics before you conclude anything. First, inspect both the running configuration and the full neighbor capability output. Seeing a graceful-restart capability in show bgp ... neighbors tells you what was advertised or received, but it does not by itself prove that stale routes were retained during this specific reset. On Cisco IOS-XE, graceful restart can be enabled globally, per neighbor, or through a peer-session template, and helper/restarting behavior depends on the negotiated capabilities and the reason the session went down. Second, correlate timestamps: determine when the BGP session changed state, when the affected routes disappeared from the BGP table and RIB, when the backup entered the FIB, and when traffic recovered. If routes remain present as stale after the session drops, graceful-restart handling becomes a strong lead. If they are removed immediately, the delay is elsewhere.
Graceful restart also completes a symmetry this post has been circling. Dampening deliberately slows failback to buy stability; graceful restart deliberately slows failover to buy control-plane resilience. In both cases, “slow” isn’t a protocol inevitability. It’s a trade someone configured, and your job during testing is to know which trades are active on your sessions.
Why the backup usually can’t just take over instantly
A router forwards from its FIB, and the FIB reflects the last completed best-path decision. Until the primary is invalidated and the backup is programmed, the router keeps forwarding along the path it still believes is best. That dependency is why the work chain exists at all: by BGP’s own rules the primary remains the best path until it’s withdrawn or its next-hop becomes unreachable. The notable exception proves the point: BGP PIC and preinstalled backup paths work by doing the expensive parts ahead of time, so that when the trigger fires, the forwarding plane can switch without waiting on per-prefix reconvergence.
Failback: a preferred path that wins on arrival
Now run the film in reverse. The primary path recovers. What has to happen?
Comparatively little. The neighbor re-advertises the primary routes in BGP UPDATE messages (for large tables, that’s many messages, not one). As each advertisement is processed, the primary route re-enters the best-path decision and immediately wins, because it’s the preferred path by local-pref, AS-path length, or whatever policy made it primary in the first place. The router installs it in the RIB and FIB and forwards.
The advantage isn’t literally that only one message is processed. It’s what failback gets to skip:
- No failure-detection interval, because nothing failed.
- No urgent invalidation of the healthy backup; it simply loses the election and steps back into standby.
- No proof of absence required. A newly available preferred path announces itself, and best-path comparison happens as advertisements arrive.
Subtractive vs. additive: the general rule
This asymmetry generalizes beyond BGP. Any control plane that must distinguish “gone” from “quiet” pays a detection tax on silent failures, and any preference-based selection process converges quickly when the preferred option reappears:
# Failover (subtractive change) establish the path is unusable → withdraw state → re-decide → reprogram → propagate # cost: detection (silent failures) + convergence + propagation, end to end # Failback (additive change) receive preferred advertisements → best-path selection → RIB/FIB update # cost: processing advertisements that win on arrival
In protocol terms: a withdrawal forces the network to fall back on remaining information, while an advertisement hands the network new, better information. Falling back requires cleanup and, for silent failures, proof of absence. Accepting an upgrade requires neither.
What to do with this
Measure before you attribute. An end-to-end recovery number by itself doesn’t tell you where the time went. Router syslog timestamps (adjacency change, withdrawal, RIB install), a continuous timestamped probe, and visibility into the far side’s route tables let you assign the delay to a stage instead of guessing. Until then, treat explanations (including this article’s pipeline) as a model of where time can go, not a verdict on where it did go.
Budget availability around failover, not failback. Your outage window is defined by the slow direction. If your SLA math assumes 2-second recovery because that’s what you saw when the primary came back, you’ve measured the wrong direction.
Attack detection first, when detection is in play. For silent failures, detection is usually the largest single component. Deploying BFD (where supported) or tightening protocol timers shrinks it from tens of seconds to sub-second. Just remember that tests using administrative or interface shutdowns never exercised detection in the first place, so they can’t tell you what BFD bought you.
Then attack programming and propagation. On large tables or deep topologies, the later stages add up. BGP PIC, preinstalled backup paths, and next-hop tracking pre-compute the backup so the forwarding flip is nearly instantaneous. In provider and cloud environments, ask what the far side’s propagation behavior looks like, because it’s part of your recovery time whether you can see it or not.
Audit for features that intentionally alter convergence. Graceful restart can retain stale routes during a qualifying control-plane restart; dampening and hold-down policies can slow the return to a preferred path. These are legitimate design trades, but they should be deliberate and verified. For Cisco IOS-XE sessions where fast removal of a failed forwarding path is the primary objective, BFD is appropriate. Graceful restart should be enabled only when its interaction with BFD and the platform’s failure classification has been validated. AWS Direct Connect documents BGP and BFD timer behavior, but it does not establish a universal rule that BFD and graceful restart must never be configured together.
Consider whether you even want instant failback. Fast failback is usually welcome, but on a flapping primary it means traffic ping-pongs at full speed. Dampening or hold-down policies deliberately slow failback down to buy stability. The asymmetry can be a feature you tune, not just a quirk you observe.
// failover: invalidate, converge, propagate. failback: the better path shows up and wins. measure the rest.