vPC, or Virtual PortChannel, is a fundamental technology in modern networking that allows links to be aggregated across multiple physical switches, presenting them as a single logical link. The features associated with vPC serve to enhance performance, stability, and resilience in the network. This post dives deep into these features, offering insights into their configuration and advantages.
1. vPC PEER-GATEWAY
Concept: The vPC Peer-Gateway optimizes the way servers like storage devices, application servers, and load balancers handle return traffic.
How it Works: By default, nodes send return traffic to a shared gateway address, like an HSRP virtual IP. This causes latency. With the vPC Peer-Gateway, these nodes can bypass this hop, sending the return traffic directly to the source MAC address. This feature ensures that the peers forward packets locally instead of dropping them, thus maintaining swift connectivity.
Key Takeaway: vPC Peer-Gateway boosts performance for devices implementing fast-path by facilitating efficient return traffic flows.
Details:
The vPC Peer-Gateway enables certain server nodes like storage devices, application servers, and load balancers to implement “fast-path” forwarding for return traffic.
By default, nodes will send return traffic to a shared gateway address like an HSRP virtual IP. This extra hop can add latency.
With fast-path, nodes will send return traffic directly to the source MAC address of the initiator. This avoids the extra hop through the gateway.
However, the vPC loop avoidance rules normally drop traffic received on the peer-link if it is destined to the peer’s MAC address. This would break fast-path.
The vPC Peer-Gateway feature addresses this by enabling the peers to forward packets destined to the peer’s MAC on their local interfaces.
So a server can send a fast-path packet to the source MAC, the peer will forward it locally rather than dropping it, maintaining fast-path connectivity.
In summary, the vPC Peer-Gateway enables fast-path return traffic flows to work properly by allowing forwarded across the peer link. This improves performance for storage, servers, and load balancers that implement fast-path.
Example:
9K-A(config-vpc-domain) # peer-gateway
2. vPC PEER-SWITCH
Concept: vPC Peer-Switch extends the base vPC functionality, allowing two switches to be seen as one device.
How it Works: The vPC Peer-Switch achieves this illusion by synchronizing bridge IDs and STP priorities between the peers. This results in both switches appearing as one STP root. This uniform STP state ensures seamless operation even if one peer fails.
Key Takeaway: vPC Peer-Switch enhances network stability by emulating a single STP domain across peer switches.
Details:
The vPC Peer-Switch feature builds upon the base vPC functionality that combines two switches into one logical device for purposes like ECMP hashing and STP.
It takes this further to present the two vPC peer switches as a single device to the entire network, beyond just the directly connected links.
This is accomplished by synchronizing key identifiers between the peers:
- Bridge IDs: The peers use a common bridge ID for all vPC VLANs rather than independent IDs.
- STP priorities – The port priorities are synchronized so both peers appear as a single STP root.
This makes the two switch vPC domain emulate a single virtual switch from a spanning tree perspective.
The benefit is that if one vPC peer fails, the other maintains the same STP state. There is no topology change or re-convergence.
This prevents disruption of traffic flows due to STP changes, unnecessary MAC flushing, and other issues caused by the failed switch no longer being the STP root.
In summary, vPC Peer-Switch creates a single stable STP domain across the peers, preventing failures from impacting the topology. This improves resilience and stability.
Example:
9K-A(config)# vpc domain 1 9K-A(config-vpc-domain) # peer-switch 9K-A(config)# spanning-tree vlan 1-3967 priority 4096 9K-B(config)# vpc domain 1 9K-B(config-vpc-domain) # peer-switch 9K-B(config)# spanning-tree vlan 1-3967 priority 4096
3. SPANNING-TREE PSEUDO-INFO
Concept: This feature resolves the STP loop issue that arises due to peers using the same bridge ID on all vPC VLANs.
How it Works: By configuring pseudo-information or separate STP priorities for non-vPC VLANs, one peer can act as the STP root for these VLANs. This avoids loop detection and ensures stable traffic flow.
Key Takeaway: Pseudo-information maintains optimal redundancy in environments where vPC and non-vPC links coexist.
Details:
The vPC peer-switch feature synchronizes the root bridge ID and priorities between the peers. However, this can cause issues for non-vPC links into the vPC domain. Since the peers use the same bridge ID on all vPC VLANs, the peer link appears as a loop to Spanning Tree.
To address this, pseudo-information can be configured – separate STP priorities for non-vPC VLANs.
(example below) vPC VLANs 1-100 use the synchronized ID and priorities between peers.
For non-vPC VLANs 101-200, configure a different root priority on each peer.
This allows one peer to be the STP root for the non-vPC VLANs across the peer link. No loop is detected.
The pseudo-information provides the complete view of the topology for spanning tree protocol:
- vPC VLANs see a single STP root
- Non-vPC VLANs see the standard redundant STP roots
This keeps the network stable and converged in scenarios where vPC and non-vPC links coexist. Traffic flows properly without being blocked.
In summary, STP pseudo-information maintains optimal redundancy and stability for mixed vPC and non-vPC topologies by providing a full view of the topology.
Example:
9K-A: # vPC Peer-Switch Config vpc domain 1 peer-switch spanning-tree vlan 1-100 priority 16384 # Pseudo-Info for non-vPC VLANs spanning-tree pseudo-information vlan 101-200 designated priority 32768 vlan 101-200 root priority 4096 9K-B: # vPC Peer-Switch Config vpc domain 1 peer-switch spanning-tree vlan 1-100 priority 16384 # Pseudo-Info for non-vPC VLANs spanning-tree pseudo-information vlan 101-200 designated priority 32768 vlan 101-200 root priority 8192
4. ARP SYNCH
Concept: In a vPC domain, while the peers act as one device, they maintain individual ARP tables.
How it Works: ARP synchronization between peers ensures that both have consistent ARP tables. This facilitates rapid traffic convergence and reduces issues caused by asymmetric traffic flows.
Key Takeaway: ARP synchronization ensures fast failover times by providing consistent ARP tables across peers.
Details:
In a vPC domain, the two peer devices act as a single logical device with distributed packet forwarding.
However, they still maintain separate ARP tables by default. This can cause issues:
- If a peer goes offline and comes back, it needs to repopulate its ARP table. This delays traffic convergence.
- Asymmetric traffic flows can lead to incomplete ARP tables.
To address this, ARP synchronization can be enabled between the vPC peers.
This causes the peers to automatically synchronize their ARP tables, similar to how STP parameters are synchronized.
When a vPC peer comes back online after a failure, its ARP table is already populated – no delay to relearn ARP entries.
Synchronized ARP tables also eliminate asymmetric traffic problems that can cause incomplete ARP entries.
So just like maintaining one STP topology, one consistent ARP table across the vPC peers improves redundancy and stability. Traffic loss is minimized when peers go offline and return.
In summary, ARP synchronization provides consistent neighbor tables across the vPC domain for faster failover convergence times.
Example:
9K-A(config)vpc domain 1 9K-A(config-vpc-domain) ip arp synchronize
5. VPC DOMAIN ID
Concept: The uniqueness of the vPC domain ID ensures isolation and optimal operation of vPC.
How it Works: The vPC domain ID determines the system MAC addresses for the peer devices. Any repetition of this ID can lead to MAC address conflicts and disrupt connectivity.
Key Takeaway: Assigning unique vPC domain IDs ensures flawless vPC operation by guaranteeing unique system MACs.
Details:
The vPC domain ID is a crucial setting that should be unique in a Layer 2 network topology.
This is because the vPC domain ID forms the unique system MAC addresses for the two vPC peer devices.
For example, if the domain ID is 100, the vPC peers may use MACs 00:23:04:ee:be:00 and 00:23:04:ee:be:01.
If another separate vPC domain used the same domain ID 100, it would result in MAC address conflicts for the peers.
This would break connectivity and cause traffic leaks across the vPC domains. The domains would not isolate properly.
Therefore, each vPC domain in a contiguous L2 network should use a unique domain ID, even across different NPIV environments.
This ensures the system MACs will be unique and isolation will function correctly. It prevents leaks and connectivity issues.
In summary, unique vPC domain IDs guarantee unique system MACs across the network topology, which is crucial for proper vPC operation and isolation.
6. VPC PEER LINK
Concept: The vPC peer link is vital for maintaining synchronization between two vPC devices.
How it Works: This link carries essential heartbeat traffic and MAC address tables. To ensure robustness and prevent interference, a point-to-point direct connection between peers is recommended.
Key Takeaway: Using dedicated links for the vPC peer link is pivotal for reliable failover detection and overall stability.
Details:
The vPC peer link carries vital heartbeat traffic between the two vPC devices to ensure accurate failover detection. It also forwards MAC addresses and other tables between the peers.
If the peer link is not point-to-point, or if it carries unrelated traffic, it introduces risks:
- Traffic flooding over the peer link could consume bandwidth needed for heartbeats and synchronization.
- Forwarding loops are possible if peering traffic leaks out other interfaces.
- Failures anywhere on a shared link could impact peer connectivity.
Therefore, Cisco strongly recommends using a point-to-point direct connection between peers for the vPC peer link.
Additionally, the peer link should use dedicated ports on the vPC devices rather than a shared interface.
This prevents any other traffic from impacting or interfering with the vital peering communication.
In summary, point-to-point dedicated links are required for the vPC peer link in order to provide guaranteed isolation and performance for the heartbeat/synchronization traffic. This ensures accurate failover detection and stability.
7. VPC PEER KEEP ALIVE
Concept: This secondary communication link between vPC peers aids in graceful failover.
How it Works: In case of a primary link failure, the Keepalive Link maintains connectivity, ensuring continued communication. It’s best to connect these through a dedicated management network link.
Key Takeaway: The Keepalive Link enhances vPC resiliency by offering an alternative communication pathway during failures.
Details:
The vPC Peer Keepalive Link provides a secondary communication path between the vPC peers, separate from the main vPC peer link.
This is important in a dual-supervisor environment. If one supervisor fails, the peer link may go down with it.
The Keepalive Link allows the second supervisor to maintain connectivity for graceful failover.
Cisco recommends dedicating ports on the management interfaces of the supervisors for the Keepalive Link.
These ports should connect via a dedicated link through the management switch.
Never connect the Keepalive Link directly between supervisors – it should only communicate through the management network.
This provides total separation from the main data plane peer link. Traffic flooding or failures cannot impact the Keepalive Link.
In summary, dedicating ports on the supervisors connected through the management network ensures a highly available and resilient secondary channel for graceful vPC failover detection.
8. BPDU GUARD and BPDU FILTER
Concept: These features, recommended in Cisco’s best practices guide, help prevent bridging loops in vPC setups.
How it Works: BPDUGUARD shuts down ports receiving BPDUs, and BPDUFILTER prevents BPDU exchanges. Together, they shield the network from unintentional bridging loops and misconfigurations.
Key Takeaway: Use both BPDUGUARD and BPDUFILTER on edge ports for layered vPC protection.
Details:
Cisco recommends using both BPDUGUARD and BPDUFILTER together on edge ports in their Virtual PortChannel (VPC) best practices guide for the following reasons:
BPDUGUARD provides a protection mechanism to shut down an edge port if a BPDU is received, indicating an unauthorized device is connected or there is a misconfiguration. This helps prevent bridging loops.
BPDUFILTER prevents a port from sending or receiving BPDUs. This is useful on edge ports to prevent any BPDU exchange that could trigger topology changes.
Using both together provides layered protection. BPDUFILTER prevents BPDU exchange altogether. BPDUGUARD acts as a safety net in case a BPDU still manages to reach an edge port due to misconfiguration or malfunction, shutting it down to prevent issues.
In a VPC configuration, the dual-homed connections to endpoints could accidentally create a bridging loop if BPDUGUARD is not used. If an endpoint starts sending BPDUs, it could cause STP to detect a loop and block one of the VPC peer links.
So in summary, BPDUFILTER provides proactive protection and BPDUGUARD acts as a failsafe. Their combined usage on edge ports helps prevent bridging loops and misconfigurations in the VPC setup.
Example:
Miami(config)spanning-tree port type network default Miami(config)spanning-tree port type edge bpduguard default Miami(config)spanning-tree port type edge bpdufilter default
9. PORT-CHANNEL DISTRIBUTION
Concept: Within a vPC peer chassis, it’s best to spread out the port channel member interfaces.
How it Works: Distributing member interfaces across line cards enhances bandwidth utilization, provides redundancy, and facilitates operations like upgrades.
Key Takeaway: Distribute port channel interfaces across line cards for optimal performance and redundancy.
Details:
In a Cisco Nexus vPC environment, the port channels from the upstream devices will be dual-homed across the two vPC peers.
Within each vPC peer chassis, the best practice is to distribute the port channel member interfaces across multiple line cards.
The main reasons for this recommendation are:
- Improves bandwidth utilization – traffic load is spread evenly across line cards.
- Prevents overload on any one line card or supervisor.
- Provides redundancy against failure of a line card or supervisor.
- Allows rolling upgrades – single line cards can be upgraded without full impact.
So in summary, distributing the member interfaces of the dual-homed vPC port channels across line cards improves performance, redundancy, and graceful operations like upgrades.
It prevents outages if a specific line card or supervisor fails, and also ensures even utilization of chassis resources by balancing traffic across all available interfaces.
10. OVERSUBSCRIPTION ALIGNMENT
Concept: Oversubscription ratios guide the sizing of uplink and downlink speeds in the switch.
How it Works: Aligning ratios ensure that vPC nodes can handle traffic demands even during peak periods, preventing bottlenecks.
Key Takeaway: Monitor and maintain aligned oversubscription ratios for peak performance under all traffic conditions.
Details:
Oversubscription ratios refer to how much larger the total possible throughput is versus the actual bandwidth provided.
For example, a 20:1 ratio means the total possible traffic could be up to 20x the actual bandwidth.
Cisco recommends different oversubscription targets depending on the network location:
- Access layer: A higher oversubscription of 20:1 is typical here. There is a very large amount of ports compared to uplink bandwidth.
- Distribution layer: An oversubscription of 4:1 to 6:1 is recommended here to account for some aggregation.
- Core layer: The lowest oversubscription of 2:1 or 3:1 should be maintained here since this is the backbone.
The key is mapping out the expected traffic on each part of the network, both current needs and future growth. Determine the total connections and required bandwidth.
Then apply the oversubscription ratios above to properly size each area’s uplink bandwidth. Monitor utilization and adjust if needed.
Following Cisco’s guidelines will provide sufficient bandwidth with expected oversubscription. The goal is to avoid congestion during peak usage now and in the future.
11. vPC AUTO-RECOVERY
Concept: vPC Auto-recovery enhances system resilience by ensuring automatic restoration after a vPC peer-link failure.
How it Works: In scenarios where the vPC peer-link fails, the secondary switch will suspend its vPC member ports as a preventive measure against potential split-brain scenarios. The auto-recovery feature, once triggered, revokes this action after a specific timeout, allowing traffic to flow without the need for manual intervention.
Key Takeaway: vPC Auto-recovery offers an automated recovery solution for vPC systems, reducing the downtime and minimizing the administrative overhead during peer-link failures.
Details:
- The primary purpose of the auto-recovery command is to restore operations in situations where both the vPC peer-link and vPC keepalive link are down simultaneously.
- If only the peer-link goes down, the secondary switch will deactivate its vPC member ports to prevent dual-active scenarios, regardless of the auto-recovery setting.
- The main risk with auto-recovery is the dual-active or split-brain scenario, where both peers believe they are the active device. This can happen if both the peer-link and keepalive link fail, and then the peer-link recovers before the keepalive link.
- Proper configuration is essential to mitigate the risks. The recovery delay should be set to allow sufficient time for network protocols to stabilize, and the vPC keepalive link should be robust and reliable.
Example:
9K-A(config-vpc-domain) # auto-recovery reload-delay {60-3600 seconds}
12. vPC LAYER3 PEER-ROUTER
Concept: The Layer3 peer-router feature enhances the vPC peer link by supporting Layer 3 routing, facilitating more efficient path selections and traffic distribution.
How it Works: Initially, the vPC peer link serves as a Layer 2 trunk. With the layer3 peer-router command, this link transforms into a routing link, allowing the vPC peers to form BGP adjacencies over the peer link, exchanging BGP routes.
Key Takeaway: By evolving the vPC peer link from a Layer 2 connection to a Layer 3 routing link, the layer3 peer-router feature ensures optimal traffic distribution and provides support for features like IP multicast routing across the vPC.
Details:
- Typically, the vPC peer link is a Layer 2 conduit for forwarding traffic between the peers. The layer3 peer-router command evolves this functionality, enabling the establishment of BGP peering relationships over the peer link.
- This transformation ensures synchronization of routing tables and next-hop selections across the vPC peer switches.
- Traffic is routed between the peer switches via the vPC peer link, eliminating the need for traffic to traverse another interface, thus avoiding potential bottlenecks or suboptimal paths.
- Benefits include quicker convergence during route changes or peer failures, better traffic distribution, and the support for advanced features like multicast routing across the vPC.
Example:
9K-A(config-vpc-domain) # layer3 peer-router