1. Identify the issue: Do your best to get all the data possible to help troubleshoot the issue.
2. Determine the forwarding path of the frames: Use a manual Layer 2 trace by examining the MAC address tables when no Layer 2 traceroute tool is compatible with your network’s OS, such as Cisco’s NX-OS.
3. Obtain necessary information: Get the source and destination MAC addresses, along with the corresponding IP addresses from the server team.
4. Identify the source switch and interface: Use the command sh mac address-table | inc SSSS.SSSS.SSSS replacing SSSS.SSSS.SSSS with the source MAC address. The output will be something similar to:
SWITCH-B# sh mac address-table | inc SSSS.SSSS.SSSS
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+------+------+----+------------
* 300 SSSS.SSSS.SSSS dynamic 15 F F Eth201/2/3
5. Determine the path to the destination: Use the command sh mac address-table | inc DDDD.DDDD.DDDD replacing DDDD.DDDD.DDDD with the destination MAC address. The output will look like:
SWITCH-B# sh mac address-table | inc DDDD.DDDD.DDDD
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+------+------+----+-----------
* 300 DDDD.DDDD.DDDD dynamic 15 F F Po2
6. Find the physical interfaces in the port-channel: Execute show port-channel summary. The output should be something like:
SWITCH-B# sh port-channel sum
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
-------------------------------------------------------------------
2 Po2(SU) Eth LACP Eth2/1(P) Eth2/2(P)
7. Consult the CDP neighbor table: Find out which neighbors these interfaces connect to by running show cdp neighbor. The output will be something like:
SWITCH-B# sh cdp ne
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
SWITCH-C
Eth2/1 130 S I s N5K-C5548 Eth2/1
SWTICH-D
Eth2/2 133 S I s N5K-C5548 Eth2/2
8. Determine the forwarding physical interface: Use the command sh port-channel load-balance forwarding-path int port-channel 2 vlan 200 src-ip 3.3.3.3 dst-ip 4.4.4.4. The output will indicate the physical interface forwarding the packets and the load-balancing algorithm the port-channel is using:
SWITCH-B# sh port-channel load-balance forwarding int port-channel 2 vlan 250 src-ip 3.3.3.3 dst-ip 4.4.4.4
Missing params will be substituted by 0's.
Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 22 Outgoing port id: Ethernet2/2
Param(s) used to calculate load-balance:
dst-ip: 4.4.4.4
src-ip: 3.3.3.3
dst-mac: 0000.0000.0000
src-mac: 0000.0000.0000
9. Repeat the process: Continue this process for each subsequent switch until the MAC address-table entry points to a physical interface with only one MAC address in the MAC address-table. This is the end of the one-way Layer 2 path.
10. Trace the return path: Do not assume the return path will be symmetrical to the outgoing path. Tracing the return path can help uncover asymmetries potentially causing the problem.
Recommendations for Successful Troubleshooting:
1. Consider all possibilities: Always consider both symmetrical and asymmetrical paths when troubleshooting network issues.
2. Ensure consistent configuration: All devices in the forwarding path should be configured to handle the type of data frames being used.
3. Utilize available tools: Make full use of the utilities provided by your hardware and OS for troubleshooting.
4. Be patient and meticulous: Network issues can be complex and time-consuming to resolve. Patience and attention to detail can lead to successful problem-solving.