Cisco  – Manual Tracing of Layer 2 Path on Cisco Nexus Switches

1. Identify the issue: Do your best to get all the data possible to help troubleshoot the issue.

2. Determine the forwarding path of the frames: Use a manual Layer 2 trace by examining the MAC address tables when no Layer 2 traceroute tool is compatible with your network’s OS, such as Cisco’s NX-OS.

3. Obtain necessary information: Get the source and destination MAC addresses, along with the corresponding IP addresses from the server team.

4. Identify the source switch and interface: Use the command sh mac address-table | inc SSSS.SSSS.SSSS replacing SSSS.SSSS.SSSS with the source MAC address. The output will be something similar to:

    SWITCH-B# sh mac address-table | inc SSSS.SSSS.SSSS
    VLAN     MAC Address      Type      age  Secure NTFY   Ports
    * 300      SSSS.SSSS.SSSS    dynamic   15      F    F     Eth201/2/3

5. Determine the path to the destination: Use the command sh mac address-table | inc DDDD.DDDD.DDDD replacing DDDD.DDDD.DDDD with the destination MAC address. The output will look like:

    SWITCH-B# sh mac address-table | inc DDDD.DDDD.DDDD
    VLAN     MAC Address      Type      age  Secure NTFY   Ports
    * 300      DDDD.DDDD.DDDD    dynamic   15      F    F     Po2

6. Find the physical interfaces in the port-channel: Execute show port-channel summary. The output should be something like:

    SWITCH-B# sh port-channel sum
    Flags:  D - Down        P - Up in port-channel (members)
            I - Individual  H - Hot-standby (LACP only)
            s - Suspended   r - Module-removed
            S - Switched    R - Routed
            U - Up (port-channel)
            M - Not in use. Min-links not met
    Group Port-Channel  Type     Protocol  Member Ports
    2     Po2(SU)       Eth      LACP       Eth2/1(P)    Eth2/2(P)

7. Consult the CDP neighbor table: Find out which neighbors these interfaces connect to by running show cdp neighbor. The output will be something like:

    SWITCH-B# sh cdp ne
    Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
    S - Switch, H - Host, I - IGMP, r - Repeater,
    V - VoIP-Phone, D - Remotely-Managed-Device,
    s - Supports-STP-Dispute
    Device-ID Local Intrfce Hldtme Capability Platform     Port ID
              Eth2/1        130    S I s      N5K-C5548    Eth2/1
              Eth2/2        133    S I s      N5K-C5548    Eth2/2

8. Determine the forwarding physical interface: Use the command sh port-channel load-balance forwarding-path int port-channel 2 vlan 200 src-ip dst-ip The output will indicate the physical interface forwarding the packets and the load-balancing algorithm the port-channel is using:

    SWITCH-B# sh port-channel load-balance forwarding int port-channel 2 vlan 250 src-ip dst-ip
    Missing params will be substituted by 0's.
    Load-balance Algorithm on switch: source-dest-ip
    crc8_hash: 22 Outgoing port id: Ethernet2/2
    Param(s) used to calculate load-balance:
    dst-mac: 0000.0000.0000
    src-mac: 0000.0000.0000

9. Repeat the process: Continue this process for each subsequent switch until the MAC address-table entry points to a physical interface with only one MAC address in the MAC address-table. This is the end of the one-way Layer 2 path.

10. Trace the return path: Do not assume the return path will be symmetrical to the outgoing path. Tracing the return path can help uncover asymmetries potentially causing the problem.

Recommendations for Successful Troubleshooting:

1. Consider all possibilities: Always consider both symmetrical and asymmetrical paths when troubleshooting network issues.
2. Ensure consistent configuration: All devices in the forwarding path should be configured to handle the type of data frames being used.
3. Utilize available tools: Make full use of the utilities provided by your hardware and OS for troubleshooting.
4. Be patient and meticulous: Network issues can be complex and time-consuming to resolve. Patience and attention to detail can lead to successful problem-solving.