BGP Master Class for Cisco Enterprise Networks

Kerry Cordero
Routing & Switching
33 min read

The Definitive Guide for Cisco Enterprise Networks (IOS-XE & NX-OS)

This guide covers Border Gateway Protocol from first principles through path selection, policy engineering, security hardening, and enterprise design patterns. Every configuration example is shown for Catalyst/ASR/ISR (IOS-XE) and Nexus (NX-OS). It’s written for the enterprise engineer: internet edge, multihoming, WAN, and the growing role of BGP inside the campus and data center.

Contents: Part 1: Why BGP Exists · Part 2: Fundamentals · Part 3: Route Reflectors · Part 4: Policy & Security · Part 5: Advanced Features · Part 6: Design Patterns · Part 7: Troubleshooting & Reference

The Minimum Safe Production Checklist

Before your first (or next) eBGP session carries production traffic, every box below should be checked. The rest of this guide explains each one.

Part 1: Why BGP Exists — and When an Enterprise Actually Needs It

1.1 A Different Kind of Routing Protocol

IGPs (OSPF, EIGRP, IS-IS) answer the question “what is the fastest path?” inside a network you fully control. BGP answers a different question: “what path am I allowed and willing to use, given business relationships?” It is a path-vector protocol built around policy, not speed:

  • It runs over TCP port 179 — no discovery, no multicast hellos. You explicitly configure every neighbor. If a peer isn’t configured on both sides, no session forms. This is a feature: BGP relationships mirror business relationships.
  • Its metric isn’t bandwidth or delay — it’s an ordered list of tie-breakers (the best-path algorithm) that you manipulate with policy.
  • It scales to the entire internet (~1 million IPv4 routes and growing) because it dampens the frequency of updates and doesn’t recompute a global topology the way link-state protocols do.

1.2 eBGP vs. iBGP

The same protocol, two very different behaviors depending on whether the peer is in your Autonomous System:

eBGP (different AS) iBGP (same AS)
Default TTL 1 (peers expected to be directly connected) 255
Next-hop on advertisement Changed to the advertising router Unchanged
AS-path Prepends own AS Not modified
Loop prevention AS-path (reject routes containing own AS) Split horizon: routes learned from one iBGP peer are NOT re-advertised to other iBGP peers
Administrative distance (IOS-XE) 20 200

Two of these rows generate 90% of real-world iBGP problems, so flag them now:

  1. The iBGP split-horizon rule means iBGP requires a full mesh of sessions (or route reflectors / confederations — Part 3).
  2. Next-hop is not changed over iBGP, so an internal router can learn a route whose next hop is an external IP it can’t reach. The fix is next-hop-self on the border routers or carrying the external subnet in your IGP — forgetting this is the classic “route is in the table but traffic blackholes” lab-and-production failure.

1.3 When Does an Enterprise Need BGP?

Historically the answer was “when you’re multihomed to the internet.” That’s still the canonical case — but it no longer describes how most enterprises actually encounter BGP. Today, for many organizations, BGP is the everyday WAN and data-center routing protocol, running at every site whether or not the internet edge is multihomed. The realistic list:

  • You are multihomed to the internet (two ISPs, or two links to one ISP with your own address space) and need to influence inbound/outbound path selection. Still the classic case, and the design focus of Part 6.
  • You have provider-independent (PI) address space and your own public ASN — advertising your space requires eBGP.
  • You have an MPLS L3VPN WAN. Every site runs an eBGP session to the carrier’s PE router. Each site advertises its local prefixes to the PE; the carrier’s MP-BGP VPN backbone carries them; every other site learns them from its PE. That eBGP session at each site is your site-to-site routing — the carrier’s cloud is invisible, and inter-site reachability lives or dies on those PE-CE sessions and their filters. For many enterprises this alone means BGP runs at 50, 200, or 1,000 sites that will never touch an internet routing table.
  • You connect privately to the cloud. Azure ExpressRoute and AWS Direct Connect speak BGP to you and only BGP — there is no static or IGP option on those handoffs. Each circuit is an eBGP session (Microsoft peers as AS 12076; AWS private VIFs peer from the Amazon-side ASN you configure), with provider-imposed prefix limits and provider-documented knobs for active/passive across redundant circuits.
  • Your data center fabric uses it as the control plane — VXLAN EVPN on Nexus uses BGP (the L2VPN EVPN address family) to distribute MAC/IP reachability, typically with spines as route reflectors (iBGP designs) or per-device ASNs (eBGP underlay designs). If you run modern DC fabrics, you run BGP whether you think of it that way or not.
  • You need policy-based control at an administrative boundary — extranets, mergers/acquisitions (interconnecting two networks with overlapping IGP designs), SD-WAN edge-to-core peering, or handoffs to entities you don’t control. BGP is the protocol of administrative boundaries even when both sides are private.

The practical takeaway: the question has shifted from “do we need BGP?” to “how many different roles is BGP already playing in our network, and are we operating all of them to the same standard?” The hardening and policy discipline in Part 4 applies to a PE-CE session or an ExpressRoute peering exactly as much as to an internet edge.

Conversely: a single-homed site with one default route still does not need BGP on its internet handoff, even if the ISP offers it. A static default with IP SLA tracking is simpler and converges as fast. Don’t deploy BGP for prestige — deploy it where the design above demands it.

1.4 Autonomous System Numbers

  • 16-bit ASNs occupy 0–65535. Public/global-use ASNs are allocated by RIRs, excluding special-purpose and reserved values: AS0, AS23456 (AS_TRANS), 64496–64511 (documentation), 64512–65534 (private use), and 65535 (reserved).
  • Private-use ASNs: 64512–65534 (2-byte) and 4200000000–4294967294 (4-byte) — for internal use, confederation members, and SP customers who don’t own an ASN. Strip them before routes reach the internet (remove-private-as).
  • 4-byte ASNs (RFC 6793): up to 4294967295, written in asplain (e.g., 401308). Universally supported on modern IOS-XE/NX-OS; new RIR assignments are typically 4-byte, so any “we only support 2-byte” assumption in scripts or filters is a bug.

Part 2: The Fundamentals

2.1 Session Establishment and the Finite State Machine

BGP neighbors walk a state machine you will read in show bgp summary for the rest of your career:

Idle → Connect → (Active) → OpenSent → OpenConfirm → Established

  • Idle: not trying, or just reset. Persistent Idle usually means no route to the neighbor or the neighbor isn’t configured.
  • Active: sounds healthy; isn’t. It means the TCP connection attempt failed and the router is retrying. Persistent Active = TCP can’t complete: wrong IP, ACL/firewall blocking 179, neighbor not configured on the far side, or update-source mismatch.
  • OpenSent / OpenConfirm: TCP works; BGP OPENs are being exchanged. Failures here are parameter mismatches — wrong remote-as, bad authentication, capability problems.
  • Established: the only state where routes flow. Anything cycling between states = flapping; check the log (%BGP-5-ADJCHANGE) and the Last read/write and notification history in show bgp neighbor.

Four message types run the protocol: OPEN (parameters/capabilities), UPDATE (routes added/withdrawn — the payload), KEEPALIVE (default every 60s, hold time 180s; a session dies if hold time passes with no message), and NOTIFICATION (fatal error, session torn down — the notification code in the log tells you exactly why).

Hold-time note: peers negotiate down to the lower of the two configured hold times. The 60/180 defaults are conservative internet-era values; for fast failure detection don’t crank timers down — use BFD (Part 5). Timer tuning below ~10s of hold time buys little and costs stability.

2.2 Path Attributes — the Vocabulary of Policy

Every BGP route carries attributes; policy is the art of reading and rewriting them:

Attribute Type Scope Used for
AS_PATH Well-known mandatory Global Loop prevention + path length tiebreak + prepending policy
NEXT_HOP Well-known mandatory Global Where to actually send traffic
ORIGIN Well-known mandatory Global Ancient tiebreak (IGP < EGP < Incomplete)
LOCAL_PREF Well-known discretionary Your AS only (iBGP) Outbound path preference — the enterprise workhorse
MED (metric) Optional non-transitive Adjacent AS Suggesting inbound preference to a directly connected AS
COMMUNITY Optional transitive As far as peers honor it Tagging routes for policy — the enterprise workhorse #2
Weight Cisco-proprietary Single router only Local override; not advertised anywhere

The mental model for the two you’ll use daily:

  • LOCAL_PREF controls how traffic leaves your AS. Higher wins. Set it inbound on routes you learn, and every router in your AS agrees on the exit.
  • Influencing how traffic enters your AS is fundamentally harder — you’re trying to affect someone else’s outbound decision. Your tools, in decreasing order of reliability: advertise more-specific prefixes (blunt but decisive), AS-path prepending (a suggestion), MED (only compared between routes from your AS by default, and many ISPs ignore it), and provider communities (the professional’s tool — most ISPs publish communities that let you set local-pref inside their network; read your ISP’s community guide).

2.3 The Best-Path Selection Algorithm

When multiple routes to the same prefix exist, Cisco BGP compares them in strict order. The list is long; these are the steps that decide real networks (memorize the first four and step 8–11 ordering):

  1. Highest Weight (Cisco-only, local to the router)
  2. Highest LOCAL_PREF
  3. Locally originated (network/aggregate/redistributed on this router)
  4. Shortest AS_PATH (prepending manipulates this)
  5. Lowest ORIGIN (IGP < EGP < Incomplete)
  6. Lowest MED (by default only when the first AS in the path is the same)
  7. eBGP over iBGP
  8. Lowest IGP metric to the next hop (“hot potato” — exit at the closest door)
  9. Oldest eBGP route (stability preference)
  10. Lowest router ID
  11. Lowest neighbor address

Before any of this, a route must be valid: next hop reachable, not failing loop check, not filtered. show bgp ipv4 unicast <prefix> shows every candidate path and flags the best with > — and on modern code tells you the reason the best path won. Learn to read that output; it’s the whole algorithm made visible.

Prefer LOCAL_PREF over Weight for policy, even though Weight is evaluated first: Weight exists on one router only and produces inconsistent AS-wide decisions; LOCAL_PREF is propagated through iBGP and gives the AS a single coherent routing policy.

2.4 Getting Routes Into BGP

Three doors, in order of preference:

  1. network statements — advertise a prefix that exists in the routing table, exactly (prefix and mask must match). Deliberate and self-documenting. The standard way to originate your public space — typically paired with a static route to Null0 for the aggregate:
ip route 203.0.113.0 255.255.255.0 Null0 name BGP-ANCHOR
router bgp 65001
 address-family ipv4 unicast
  network 203.0.113.0 mask 255.255.255.0
  1. aggregate-address — summarize routes already in the BGP table (with summary-only to suppress the specifics).
  2. Redistribution — from IGP into BGP. Use sparingly, always with a route-map filter, never redistribute BGP into IGP at the internet edge (a full table into OSPF is a résumé-generating event). Redistributed routes carry origin ? (incomplete).

2.5 A Minimal, Correct eBGP Session (Both Platforms)

IOS-XE:

router bgp 65001
 bgp log-neighbor-changes
 bgp router-id 10.255.0.1
 neighbor 192.0.2.1 remote-as 3356
 neighbor 192.0.2.1 description ISP-A
 !
 address-family ipv4 unicast
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 route-map ISP-A-IN in
  neighbor 192.0.2.1 route-map ISP-A-OUT out
  network 203.0.113.0 mask 255.255.255.0

NX-OS:

feature bgp
router bgp 65001
 router-id 10.255.0.1
 log-neighbor-changes
 address-family ipv4 unicast
  network 203.0.113.0/24
 neighbor 192.0.2.1
  remote-as 3356
  description ISP-A
  address-family ipv4 unicast
   route-map ISP-A-IN in
   route-map ISP-A-OUT out

Platform differences worth internalizing: NX-OS requires feature bgp; neighbor config nests under the neighbor rather than repeating neighbor x.x.x.x lines; network takes CIDR notation; and NX-OS activates the configured address-family under the neighbor by nesting it (no separate activate keyword). Also note the default-policy difference: NX-OS has long refused to exchange routes on eBGP sessions without explicit policy (RFC 8212 fail-closed behavior), while IOS-XE by default still propagates routes with no policy applied — from IOS-XE 17.2.1 you can enforce RFC 8212-style fail-closed behavior with bgp safe-ebgp-policy, and behavior varies by platform and release. Operationally, none of this matters if you follow the law: every eBGP neighbor gets an explicit inbound and outbound route-map regardless of platform defaults — Part 4 is about what goes in them.

Note on example values: AS 65001 (private-use) and 203.0.113.0/24 (documentation range) are lab/documentation values used throughout this guide. In production internet eBGP, use your assigned public ASN and your own prefixes — private-AS customer designs exist but require explicit coordination with your provider.

2.6 iBGP Session Hygiene

iBGP sessions should be built loopback-to-loopback, riding the IGP, so the session survives any single link failure:

! IOS-XE
router bgp 65001
 neighbor 10.255.0.2 remote-as 65001
 neighbor 10.255.0.2 update-source Loopback0
 address-family ipv4 unicast
  neighbor 10.255.0.2 activate
  neighbor 10.255.0.2 next-hop-self

! NX-OS
router bgp 65001
 neighbor 10.255.0.2
  remote-as 65001
  update-source loopback0
  address-family ipv4 unicast
   next-hop-self

next-hop-self on border routers solves the unreachable-external-next-hop problem from 1.2. eBGP sessions, by contrast, normally peer on the directly connected link addresses (multihop is the exception, used for loopback peering across L3 or with route servers — and requires ebgp-multihop <ttl> / NX-OS ebgp-multihop).

Part 3: Scaling iBGP — Route Reflectors

3.1 The Full-Mesh Problem

The iBGP split-horizon rule (learned-from-iBGP is never sent to iBGP) means every iBGP speaker must peer with every other: n(n−1)/2 sessions. Five routers = 10 sessions (fine). Twenty routers = 190 sessions (not fine). Two solutions exist; in the enterprise, one of them matters:

  • Route Reflectors (RR) — the standard. A router configured as RR is allowed to reflect iBGP routes between its clients.
  • Confederations — sub-AS partitioning. Powerful, complex, and essentially absent from enterprise networks. Know the name; deploy RRs.

3.2 Route Reflector Rules

  • Routes from a client are reflected to all clients and non-clients.
  • Routes from a non-client are reflected to clients only.
  • Routes from eBGP go to everyone (normal behavior).
  • The RR adds ORIGINATOR_ID and CLUSTER_LIST attributes for loop prevention — the iBGP analogs of AS_PATH.
  • Reflection does not change attributes or next hop — an RR is a control-plane mirror, not a traffic waypoint. This enables the key design freedom: the RR does not need to be in the data path. A pair of centrally located routers (or even dedicated small boxes/VMs in large designs) can reflect for the whole AS.

3.3 Configuration

Only the RR is configured; clients have no idea they’re clients.

! IOS-XE (on the RR):
router bgp 65001
 neighbor 10.255.0.10 remote-as 65001
 neighbor 10.255.0.10 update-source Loopback0
 address-family ipv4 unicast
  neighbor 10.255.0.10 activate
  neighbor 10.255.0.10 route-reflector-client

! NX-OS (on the RR):
router bgp 65001
 neighbor 10.255.0.10
  remote-as 65001
  update-source loopback0
  address-family ipv4 unicast
   route-reflector-client

3.4 Enterprise RR Design Rules

  1. Always two RRs for redundancy; every client peers with both. By default both RRs in the same cluster should use distinct cluster IDs (the default — router ID — is fine) so clients receive reflected routes even in corner-case failures; this doubles session count per client to exactly 2, which is the point.
  2. Place RRs where they’re topologically central and stable — core routers, or the WAN aggregation pair. Avoid making a busy internet edge router also the RR if you can; separation of roles simplifies maintenance.
  3. In a hierarchical WAN, RRs can be tiered (regional RRs are clients of national RRs), but most enterprises never need more than one RR pair per AS.
  4. When the RR is out of the data path, remember reflected routes keep their original next hop — your IGP must carry reachability to those next hops (or borders run next-hop-self, which they should anyway).

For a typical enterprise: two borders + a handful of internal BGP speakers can just full-mesh. The moment you find yourself adding the 6th–8th iBGP speaker or templating mesh config, deploy the RR pair and stop managing a mesh.

Part 4: Policy and Security — the Route-Map Toolkit

This is BGP’s equivalent of the STP guard features: the section that separates a working config from a safe one. The prime directive: an unfiltered BGP session is a loaded gun. Every eBGP neighbor gets an inbound and outbound policy, no exceptions, and the outbound policy’s job is to guarantee that only your prefixes ever leave your AS. The classic enterprise disaster — becoming a transit AS for the internet because you re-advertised ISP-A’s full table to ISP-B — is a filtering failure, and it can take your links (and your reputation, and briefly other people’s traffic) down.

4.1 The Building Blocks

Prefix lists — match prefixes with mask-length control. Faster and clearer than ACLs for route filtering; this is what you should use.

ip prefix-list OUR-PREFIXES seq 5 permit 203.0.113.0/24
! le/ge control matched mask lengths:
ip prefix-list NO-LONG-PREFIXES seq 5 permit 0.0.0.0/0 le 24   ! anything /24 or shorter

AS-path access lists — regex matching on AS_PATH:

ip as-path access-list 10 permit ^$            ! locally originated only
ip as-path access-list 20 permit ^3356_        ! learned directly from AS 3356
ip as-path access-list 30 permit _174$         ! originated by AS 174

^$ is the most important regex in enterprise BGP: an outbound filter permitting only ^$ guarantees you never announce anything you didn’t originate — structural anti-transit insurance, and a beautiful belt to wear with the prefix-list suspenders.

Community lists and route-maps compose these into policy. Standard communities are 32-bit values written ASN:value; enable friendly formatting everywhere: ip bgp-community new-format (IOS-XE; NX-OS default). Well-known communities you must know: no-export (don’t advertise to eBGP peers), no-advertise (don’t advertise at all), and RFC 7999 blackhole (65535:666, honored by many ISPs for DDoS victim-prefix blackholing).

One sizing caveat: standard communities are 32 bits total, historically read as ASN16:value16 — which means a 4-byte ASN cannot fit in the ASN half of a standard community. If your public ASN is 4-byte, build your policy namespace on Large Communities (RFC 8092): three 4-byte fields, e.g. 401308:100:10, supported on modern IOS-XE and NX-OS (send-community both / send-community extended semantics vary by release — verify). You’ll still honor providers’ published standard-community guides where that’s what they use.

4.2 The Enterprise Internet-Edge Policy Set (Complete, IOS-XE)

Inbound from an ISP — sanitize before accepting. (GLOBAL-V4-IN is a whitelist: it denies bogons and overly long prefixes, then permits the acceptable remainder — so the route-map permit clause matches good routes, not bogons.)

ip prefix-list GLOBAL-V4-IN seq 5  deny 0.0.0.0/8 le 32
ip prefix-list GLOBAL-V4-IN seq 10 deny 10.0.0.0/8 le 32
ip prefix-list GLOBAL-V4-IN seq 15 deny 100.64.0.0/10 le 32
ip prefix-list GLOBAL-V4-IN seq 20 deny 127.0.0.0/8 le 32
ip prefix-list GLOBAL-V4-IN seq 25 deny 169.254.0.0/16 le 32
ip prefix-list GLOBAL-V4-IN seq 30 deny 172.16.0.0/12 le 32
ip prefix-list GLOBAL-V4-IN seq 35 deny 192.0.2.0/24 le 32
ip prefix-list GLOBAL-V4-IN seq 40 deny 192.168.0.0/16 le 32
ip prefix-list GLOBAL-V4-IN seq 45 deny 198.18.0.0/15 le 32
ip prefix-list GLOBAL-V4-IN seq 50 deny 198.51.100.0/24 le 32
ip prefix-list GLOBAL-V4-IN seq 55 deny 203.0.113.0/24 le 32   ! adjust: this doc uses it as "our" space
ip prefix-list GLOBAL-V4-IN seq 60 deny 224.0.0.0/3 le 32
ip prefix-list GLOBAL-V4-IN seq 90 deny 0.0.0.0/0 ge 25        ! nothing longer than /24
ip prefix-list GLOBAL-V4-IN seq 100 permit 0.0.0.0/0 le 24

ip prefix-list OUR-PREFIXES seq 5 permit 203.0.113.0/24
ip as-path access-list 10 permit ^$

route-map ISP-A-IN deny 10
 match ip address prefix-list OUR-PREFIXES     ! never accept our own space from outside
route-map ISP-A-IN permit 20
 match ip address prefix-list GLOBAL-V4-IN
 set local-preference 200                      ! ISP-A preferred for outbound

route-map ISP-A-OUT permit 10
 match ip address prefix-list OUR-PREFIXES
 match as-path 10                              ! belt AND suspenders: our prefixes, locally originated
route-map ISP-A-OUT deny 99                    ! implicit, but explicit is kind to the next engineer

The NX-OS versions are nearly identical (ip prefix-list and route-map syntax carries over; as-path lists are ip as-path access-list NAME ... with names instead of numbers on modern code).

4.3 Session Protection

Maximum-prefix — the seatbelt against a peer (or your own filter mistake) flooding you. Set it on every eBGP session, sized to expectation:

! IOS-XE — full-table peer. Size from the neighbor's ACTUAL baseline plus
! growth headroom, and revisit regularly: the IPv4 table is ~1.06–1.1M routes
! (mid-2026) depending on vantage point, so 1.2M/90 already warns today.
 neighbor 192.0.2.1 maximum-prefix 1300000 90
! Partner extranet expected to send 20 routes:
 neighbor 198.51.100.9 maximum-prefix 50 80 restart 15

! NX-OS:
  address-family ipv4 unicast
   maximum-prefix 1300000 90

Default behavior tears the session down at the limit (correct for security); warning-only exists but defeats the purpose; restart <min> auto-retries.

Authentication — MD5 (neighbor x.x.x.x password ...) remains the deployed norm and is table stakes on every eBGP session. TCP-AO (RFC 5925) is its modern replacement, available on recent IOS-XE and NX-OS releases — use it where both ends support it, especially on long-lived ISP sessions.

GTSM / TTL security — eBGP packets from a directly connected peer should arrive with TTL 255; anything that traversed a hop can’t. This makes remote session-spoofing attacks structurally impossible:

! IOS-XE:
 neighbor 192.0.2.1 ttl-security hops 1
! NX-OS:
 neighbor 192.0.2.1
  ttl-security hops 1

(Mutually exclusive with ebgp-multihop; for multihop sessions, GTSM with the right hop count still beats plain multihop.)

Control-plane protection — an ACL/CoPP policy permitting TCP/179 only from configured peer addresses. NX-OS ships CoPP by default; on IOS-XE build a CoPP policy or at minimum an infrastructure ACL on edge interfaces.

4.4 RPKI Route Origin Validation

The modern edge also validates origins: RPKI lets you check whether the AS originating a prefix is cryptographically authorized to do so, killing a large class of route leaks and hijacks. You run (or subscribe to) one or two validator instances (Routinator, etc.); routers fetch validated data over the RTR protocol and mark every eBGP route Valid / Invalid / NotFound:

! IOS-XE:
router bgp 65001
 bgp rpki server tcp 10.10.10.50 port 3323 refresh 300
! then in policy:
route-map ISP-A-IN deny 5
 match rpki invalid

! NX-OS (release-dependent; verify support on your platform):
router bgp 65001
 rpki server 10.10.10.50
  transport tcp port 3323

Enterprise best practice as of 2026: drop Invalids where your edge policy and operations support it — many major upstreams perform some form of ROV, but adoption is not universal, so don’t assume they protect you — and create ROAs for your own address space at your RIR so the rest of the internet can protect you. The second half costs an hour in the RIR portal and is the highest-value BGP security action most enterprises haven’t taken.

4.5 Policy Anti-Patterns

  • Filtering with distribute-lists/ACLs instead of prefix lists — works, reads terribly, handles mask ranges badly.
  • clear ip bgp * (hard reset) to apply policy — tears down sessions and refloods full tables. Modern sessions negotiate the route refresh capability: clear ip bgp <peer> soft in|out (or just soft) reapplies policy with zero session impact. Hard resets are for session-parameter changes only.
  • Editing a live route-map by removing it from the neighbor first — just edit in place and soft-clear; removing the map on an RFC 8212-behaving box may mean no routes, and on older code may briefly mean all routes.
  • Prepending your way to inbound balance — one or two prepends is a legitimate nudge; five prepends usually means the real answer was more-specifics or provider communities. And some networks filter absurd prepend counts.
  • BGP dampening — penalizes flapping prefixes with suppression. Internet-scale operators largely abandoned default-parameter dampening years ago (it punished convergence after the flap ended). Don’t enable it in the enterprise.

Part 5: Advanced Features Worth Deploying

5.1 BFD — Fast Failure Detection Done Right

BGP’s own timers detect a dead-but-link-up neighbor in minutes-to-seconds at best. Bidirectional Forwarding Detection is a lightweight hello protocol (often hardware-offloaded) that detects forwarding-path failure in tens of milliseconds and signals BGP to tear down instantly. This is the correct answer to “how do I make BGP converge fast” — never timer tuning.

! IOS-XE:
interface TenGigabitEthernet0/0/0
 bfd interval 300 min_rx 300 multiplier 3
router bgp 65001
 neighbor 192.0.2.1 fall-over bfd

! NX-OS:
feature bfd
interface Ethernet1/1
 bfd interval 300 min_rx 300 multiplier 3
router bgp 65001
 neighbor 192.0.2.1
  bfd

300ms × 3 = ~900ms detection, a sane enterprise default; go faster only where hardware offload is confirmed. Deploy on: direct ISP links where the failure mode includes “link stays up but path dies” (common with carrier NIDs/metro Ethernet), DC fabric links, and any single-hop session you care about. Coordinate with the ISP — both ends must run it.

5.2 Graceful Restart, NSF, and NHT

  • Graceful Restart / NSF lets a peer keep forwarding on stale routes while your control plane restarts (supervisor switchover). Enable GR awareness/capability on dual-sup platforms and toward peers that support it; it’s default-on in NX-OS (graceful-restart under the BGP process) and configured per-neighbor or globally on IOS-XE (bgp graceful-restart).
  • Next-hop tracking (NHT) — default on modern code — registers BGP next hops with the RIB watcher so BGP reacts to IGP-detected next-hop loss in milliseconds instead of waiting for the scanner. Leave it on; know it exists when reading convergence behavior.
  • BGP PIC (Prefix Independent Convergence) — precomputes a backup path so failover time doesn’t scale with table size (crucial with full tables: repair in ~ms instead of per-prefix rewrites). bgp additional-paths install / platform-dependent PIC config; if you carry full tables on dual borders, turn it on.

5.3 Multipath — What maximum-paths Actually Does (and Doesn’t)

By default, BGP is emphatically single-path: for every prefix it selects exactly one best path, installs that one path in the routing table, and advertises that one path to peers. maximum-paths changes only the installation step:

! IOS-XE / NX-OS (under address-family):
 maximum-paths 4          ! eBGP multipath
 maximum-paths ibgp 4     ! iBGP multipath

With maximum-paths N, BGP may install up to N eligible paths into the RIB/FIB, and traffic is load-shared across them via ECMP. Three things it does not do, which architects regularly get burned assuming otherwise:

  1. It does not change best-path selection. There is still exactly one best path (marked >); the others are installed as multipath entries (show bgp ipv4 unicast <prefix> marks them).
  2. It does not change advertisement. Only the single best path is advertised to peers. (Advertising multiple paths is a separate feature — BGP Additional-Paths — used mostly in RR/PIC designs.)
  3. It does not balance by bandwidth or load. Forwarding is per-flow hashing (CEF/ECMP) across the installed paths — a 10G and a 1G path get equal flow distribution unless you deploy the DMZ link-bandwidth extended community (bgp dmzlink-bw and neighbor ... dmzlink-bw, supported on IOS-XE; link-bandwidth handling on NX-OS) to weight the hashing. Mixed-speed multipath without link-bandwidth is a subtle way to congest your smaller link at 50% total load.

Eligibility rules — a path becomes a multipath candidate only if it ties the best path on every step of the algorithm down through the IGP metric: same weight, same local-pref, same AS-path length, same origin, same MED, same type (eBGP paths bundle with eBGP, iBGP with iBGP — no mixing), and for iBGP multipath, equal IGP metric to the next hops. And one more rule that surprises everyone the first time:

By default, eBGP multipath also requires the candidate paths to have an identical AS_PATH — not just the same length. Two paths of length 3 through different neighbor ASes do not qualify.

5.3.1 bgp bestpath as-path multipath-relax — When Equal Length Is Equal Enough

That identical-AS_PATH rule is exactly what this knob relaxes: with it enabled, paths qualify for multipath if their AS paths are merely the same length, even when the ASes in them differ.

! IOS-XE:
router bgp 65001
 address-family ipv4 unicast
  bgp bestpath as-path multipath-relax
  maximum-paths 4

! NX-OS:
router bgp 65001
 bestpath as-path multipath-relax
 address-family ipv4 unicast
  maximum-paths 4

When you want it:

  • Dual MPLS carriers (or carrier + internet-VPN underlay) to the same remote sites. Each remote site’s prefix arrives with AS-path length 1 (or equal length) via two different provider ASNs. Without multipath-relax: active/standby only. With it: ECMP across both carriers.
  • eBGP-underlay data center fabrics (unique ASN per leaf or per rack). A leaf learns a spine-advertised prefix via multiple spines whose ASNs differ; multipath-relax is what makes leaf-spine ECMP work at all in per-device-ASN designs. It’s a standard line in every eBGP EVPN underlay template.
  • Dual-homed internet edges where you genuinely want active/active outbound across two upstreams for destinations at equal AS-path distance. Legitimate, but see the caution below.

When you don’t need it: dual circuits to the same neighboring AS — two links to one ISP, or dual ExpressRoute/Direct Connect circuits (both present the same provider ASN) — plain maximum-paths already covers those, because the AS paths are identical.

The caution: equal AS-path length is not equal path quality. Multipath-relax will happily hash half your flows onto a path through a congested or geographically absurd carrier as long as the hop count matches. It also creates deliberate per-flow asymmetry — fine for the internet edge’s outbound, but audit anything stateful (firewalls, NAT) in the path, and confirm your monitoring can see per-path utilization before you turn two carriers into one logical pipe. Enable it as a design decision for the specific address family and neighbors involved, not as a default line in every template.

5.4 Templates: Stop Repeating Yourself

Configuration repetition is where filter gaps are born. Both platforms have inheritance mechanisms — use them from day one:

IOS-XE peer templates:

router bgp 65001
 template peer-policy EBGP-ISP-POLICY
  route-map ISP-IN in
  route-map ISP-OUT out
  maximum-prefix 1300000 90
 template peer-session EBGP-ISP-SESSION
  password 7 <secret>
  ttl-security hops 1
  fall-over bfd
 neighbor 192.0.2.1 remote-as 3356
 neighbor 192.0.2.1 inherit peer-session EBGP-ISP-SESSION
 address-family ipv4 unicast
  neighbor 192.0.2.1 inherit peer-policy EBGP-ISP-POLICY

NX-OS peer templates:

router bgp 65001
 template peer EBGP-ISP
  password 3 <secret>
  ttl-security hops 1
  bfd
  address-family ipv4 unicast
   route-map ISP-IN in
   route-map ISP-OUT out
   maximum-prefix 1300000 90
 neighbor 192.0.2.1
  inherit peer EBGP-ISP
  remote-as 3356

5.5 Communities as an Internal Design Language

Mature BGP shops tag every route at ingress and match tags (never prefixes) in interior policy. A minimal enterprise scheme:

Community Meaning
65001:100 Learned from ISP-A
65001:200 Learned from ISP-B
65001:500 Our originated prefixes
65001:666 Blackhole request (map to RTBH / provider blackhole community)
65001:900 Do not export outside the AS

Set at the edge, send communities on iBGP (neighbor x send-community — IOS-XE requires this explicitly; NX-OS: send-community under the neighbor AF), and every downstream policy becomes one-line community matches instead of duplicated prefix lists. When you add ISP-C, interior policy doesn’t change at all.

The additive keyword — small word, big blast radius. set community in a route-map replaces the route’s entire community list by default. set community ... additive appends instead:

route-map ISP-A-IN permit 20
 match ip address prefix-list GLOBAL-V4-IN
 set local-preference 200
 set community 65001:100 additive     ! tag AND keep what the ISP sent

Without additive, that line silently strips every community the ISP attached — their geographic/customer-type tags (which you may want for TE later), any RTBH or TE community you’d set elsewhere in the policy, everything. The failure mode is nasty because nothing breaks immediately; you discover it months later when a community-based policy downstream matches nothing. The working rules:

  • Tagging at ingress or mid-policy: always additive. This should be muscle memory.
  • Omit additive only when you deliberately intend to scrub all existing communities — and even then, prefer the surgical tool: set comm-list <list> delete removes only the communities matching a community-list, leaving the rest intact (e.g., strip a previous owner’s tags after an acquisition without touching provider communities).
  • In multi-clause route-maps, remember clauses execute once per route on the first match — but a continue (IOS-XE) or a later policy touching the same route (e.g., an outbound map after an inbound map) will re-set; every one of those needs additive too.

5.6 Conditional Advertisement & RTBH (Know They Exist)

  • Conditional advertisement (neighbor x advertise-map ... non-exist-map ...) — advertise a prefix to one peer only if some other route is absent; the classic use is announcing your space to a backup ISP only when the primary path dies. Niche but occasionally the exact right tool.
  • Remotely Triggered Black Hole — pre-provision a static route to a discard next hop plus a trigger route-map; during a DDoS, inject the victim /32 tagged with your blackhole community (and your ISP’s), and the traffic dies at their edge, saving your links. Cheap to pre-build, priceless during an attack. Pair with the RFC 7999 well-known blackhole community where your ISP supports it.

Part 6: Enterprise Design Patterns

6.1 The Decision That Shapes Everything: How Many Routes to Take

From each ISP you can accept: default route only, partial routes (ISP’s own + customer routes, plus default), or the full table.

  • Default-only: minimal memory/CPU, works on almost any router, zero visibility — outbound path choice is blind. Right answer when both links are commodity internet and “any exit works.”
  • Full tables: best-path granularity per destination, real traffic engineering — at the cost of ~1M+ routes×2 in RAM/FIB (verify your platform’s FIB scale — many enterprise boxes and most L3 switches cannot hold full tables in hardware) and slower convergence without PIC.
  • Partial + default: the underrated middle path — near-optimal exit selection toward each ISP’s own customer cone, modest footprint.

Honest guidance: most enterprises are well-served by default + partial, or even dual-default with local-pref, and should not carry full tables just because it feels serious. Take full tables when you have real TE needs, capable hardware, and someone who’ll actually use the visibility.

6.2 Reference Design: Dual-Router, Dual-ISP Internet Edge

The canonical enterprise pattern: two border routers, each to a different ISP, iBGP between them, active/standby (or coarse active/active) by policy.

        ISP-A (AS 3356)        ISP-B (AS 174)
             |                       |
        [BORDER-1] ---- iBGP ---- [BORDER-2]
             \                      /
              \-- HSRP/IGP core --/

BORDER-1 (primary exit), IOS-XE, composing everything from Parts 4–5:

router bgp 65001
 bgp router-id 10.255.0.1
 bgp log-neighbor-changes
 neighbor 192.0.2.1 remote-as 3356
 neighbor 192.0.2.1 description ISP-A
 neighbor 192.0.2.1 password 7 <secret>
 neighbor 192.0.2.1 ttl-security hops 1
 neighbor 192.0.2.1 fall-over bfd
 neighbor 10.255.0.2 remote-as 65001
 neighbor 10.255.0.2 update-source Loopback0
 !
 address-family ipv4 unicast
  network 203.0.113.0 mask 255.255.255.0
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 route-map ISP-A-IN in
  neighbor 192.0.2.1 route-map ISP-A-OUT out
  neighbor 192.0.2.1 maximum-prefix 1300000 90
  neighbor 192.0.2.1 remove-private-as
  neighbor 10.255.0.2 activate
  neighbor 10.255.0.2 next-hop-self
  neighbor 10.255.0.2 send-community

Policy skeleton: ISP-A-IN sets local-preference 200 + community 65001:100; BORDER-2’s ISP-B-IN sets local-preference 100 + 65001:200. Result: all outbound traffic exits via ISP-A while it’s healthy; on failure, everything shifts to ISP-B with no reconfiguration. For inbound symmetry, BORDER-2’s ISP-B-OUT prepends (set as-path prepend 65001 65001) — or better, sets ISP-B’s published “lower local-pref” community.

Two failure-mode details that separate designs that work from designs that demo well:

  • Full-table + default anchoring: if you accept only a default from each ISP, track upstream health with IP SLA or accept at least the ISP’s partial routes — otherwise a failure beyond your directly connected link (ISP edge router alive, ISP core dead) leaves you happily forwarding into a black hole.
  • The iBGP link between borders is load-bearing: when ISP-A dies, BORDER-1’s traffic must reach BORDER-2. Make that path redundant (through the core, not just a single crosslink), and confirm your IGP and next-hop-self carry it.

6.3 Where Else BGP Appears in the Enterprise

  • MPLS L3VPN WAN: you run eBGP with the carrier PE at every site (usually private AS or your AS with as-override/allowas-in handled by the carrier). Treat each PE session with the full hardening set; filter what you send (site prefixes only) and cap what you accept.
  • Cloud interconnects: Direct Connect / ExpressRoute speak eBGP and nothing else. Constraints to design around: prefix-count limits on the cloud side (e.g., advertising too many routes drops the session), private ASNs, and the need for deterministic active/passive across redundant circuits (AS-path prepend and local-pref on your side; MED and communities per provider docs).
  • SD-WAN: the underlay/LAN side of SD-WAN edges commonly peers BGP with the core to exchange site routes; the same filtering discipline applies (never let the SD-WAN fabric become transit between your DC and internet edge by accident — communities and ^$-style origination filters again).
  • Data center VXLAN EVPN: BGP (EVPN address family) is the fabric control plane on Nexus. That’s a masterclass of its own, but the foundations here — templates, communities, RRs (spines as RRs), multipath — are exactly the primitives it’s built on.
  • Campus: mostly shouldn’t be BGP inside a single campus — IGPs and fabrics do that job. The boundary cases: very large multi-AS organizations, M&A interconnects, and SD-Access transit designs.

6.4 Sizing Guidance

Small (single site, dual ISP, one or two routers): Default (+ partial if offered) from each ISP, local-pref active/standby, full Part-4 hardening, BFD if the ISPs support it. One afternoon to deploy properly; the hardening template is 90% of the value.

Medium (multi-site WAN + internet edge): Dedicated border pair, iBGP over redundant internal paths, communities scheme from day one, peer templates, RPKI validation, RTBH pre-provisioned. MPLS/SD-WAN PE sessions templated identically.

Large (many sites, DC fabrics, cloud): RR pair for iBGP scale, full or partial tables with PIC on capable edge hardware, per-provider community-based TE, automation-rendered policy (the filters in Part 4 should be generated, not hand-typed — IRR/RPKI data can drive them), and route-table telemetry (prefix counts, best-path churn) into your monitoring stack.

Part 7: Verification, Troubleshooting, and Quick Reference

7.1 The Commands You’ll Live In

show bgp ipv4 unicast summary          ! session states + prefix counts (NX-OS: show ip bgp summary works too)
show bgp ipv4 unicast                  ! the table; > marks best path
show bgp ipv4 unicast 8.8.8.8          ! every path for a prefix + why best won
show bgp ipv4 unicast neighbors 192.0.2.1 advertised-routes
show bgp ipv4 unicast neighbors 192.0.2.1 routes        ! accepted after inbound policy
show bgp ipv4 unicast neighbors 192.0.2.1 received-routes  ! pre-policy (needs soft-reconfig inbound on IOS-XE)
show bgp ipv4 unicast regexp _174$
show ip route 8.8.8.8                  ! did the best path make it to the RIB/FIB?
show bgp ipv4 unicast neighbors 192.0.2.1 | i Last|state|Notif
clear bgp ipv4 unicast 192.0.2.1 soft  ! reapply policy, zero impact
debug ip bgp updates                   ! last resort; ACL-scope it or regret it

The single most important habit: check advertised-routes after every outbound policy change. The transit-leak disaster is invisible in your own routing table — it only shows in what you’re sending.

7.2 Troubleshooting Playbook

Stuck in Idle: no route to the peer, or neighbor unconfigured. Check reachability from the update-source (ping 192.0.2.1 source Lo0).

Stuck in Active: TCP failing. Wrong peer IP, ACL/firewall on 179, update-source mismatch (iBGP loopback peering configured on one side only), or eBGP peer >1 hop away without multihop/GTSM alignment.

Cycles to OpenSent/OpenConfirm then resets: parameter mismatch — read the NOTIFICATION in the log; it names the field (bad peer AS, auth failure, hold-time zero, capability).

Session up, no routes received: inbound policy denies everything (soft-clear and check received-routes vs routes), peer’s outbound policy, missing AF activation, or RFC 8212 default-deny on a session someone built without route-maps.

Routes received, not in routing table: next hop unreachable (the 1.2 classic — check show bgp ... <prefix> for “inaccessible”), better AD source (a static or IGP route winning), or RIB-failure (show bgp ipv4 unicast rib-failure).

Traffic exits the wrong ISP: walk the best-path algorithm against show bgp <prefix> — 95% of the time it’s a local-pref you didn’t set where you thought, or weight lingering from an old test.

Asymmetric flow breaks stateful things: inbound path is the other ISP (firewalls drop the return). Fix inbound TE (more-specifics/communities) or, pragmatically, ensure edge state devices handle asymmetry (or NAT at each border pins symmetry).

Peer flaps under load: hold timer expiry during CPU spikes or a link dropping keepalives — check Last reset reason, interface errors, and whether BFD (correctly tuned, hardware-offloaded) should replace aggressive timers someone configured.

7.3 Monitor These Proactively

  • Session state changes — every ADJCHANGE pages or tickets. A flapping ISP session is an SLA conversation waiting to happen.
  • Prefix counts per neighbor — alert well below max-prefix limits, and on drops (a sudden fall from 950k to 200 routes means your upstream leaked or filtered).
  • Advertised-route count — should equal the number of prefixes you own. Any increase is a potential leak; alert at N+1.
  • RPKI validity of your own prefixes from an external vantage (is someone else originating your space?). Free monitoring services exist; use one.
  • Best-path churn on edge routers — sustained churn indicates upstream instability or an internal policy fight.

7.4 Minimum Viable Hardening (paste-ready, per eBGP neighbor)

! IOS-XE — assume templates from 5.4 wrap these:
 neighbor X password <key>            ! or TCP-AO where supported
 neighbor X ttl-security hops 1
 neighbor X fall-over bfd
 address-family ipv4 unicast
  neighbor X route-map <PEER>-IN in       ! bogons + own-space + RPKI-invalid drops, LP set, community tag
  neighbor X route-map <PEER>-OUT out     ! own prefixes + ^$ as-path, nothing else, ever
  neighbor X maximum-prefix <sized> 90
  neighbor X remove-private-as
  neighbor X send-community

NX-OS mirrors every line under the neighbor / neighbor-AF hierarchy (Part 4 and 5.4 show the syntax).

7.5 The One-Paragraph Philosophy

BGP is policy wearing a routing protocol’s clothes. Decide your policy first — which exit, which routes accepted, which prefixes announced, who is trusted how far — then express it in filters that fail closed: explicit inbound sanitation, outbound announcements provably limited to what you originate, session protections (max-prefix, GTSM, auth, BFD) on every external peer, and RPKI both consumed and published. Use LOCAL_PREF for outbound will, communities as your internal language, more-specifics and provider communities for inbound persuasion, and templates so the hundredth neighbor is as safe as the first. Then monitor what you send as obsessively as what you receive — because in BGP, your worst outage is the one you cause for someone else.

Configurations verified against IOS-XE (ASR/Catalyst 8000/9000) and NX-OS (Nexus 9000) behavior. Syntax varies slightly by release — RPKI, TCP-AO, and RFC 8212 default-policy behavior in particular are release-dependent; confirm against your platform’s configuration guide. IPv6 sessions deserve the identical treatment under the ipv6 unicast address family.

© 2026 cordero.me • All Rights Reserved