I ran these on our Nexus 7K switches.
Display the CPU utilization
TPA-C12-L2-LEAF101# show processes cpu
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
1 784907 14217801 55 0.0% init
2 3 294 10 0.0% kthreadd
3 7494 1466724 5 0.0% migration/0
4 3051755 778000403 3 0.0% ksoftirqd/0
5 259812 9291924 27 0.0% watchdog/0
6 6029 1150831 5 0.0% migration/1
7 3220043 822340807 3 0.0% ksoftirqd/1
8 29439 9291924 3 0.0% watchdog/1
9 8112419 112796026 71 0.0% events/0
10 2668389 91944131 29 0.0% events/1
11 0 56 16 0.0% khelper
12 24 3468 7 0.0% kblockd/0
13 19 1853 10 0.0% kblockd/1
14 0 2 0 0.0% kacpid
15 0 2 0 0.0% kacpi_notify
16 28 9 3141 0.0% ksuspend_usbd
17 9 275 33 0.0% khubd
18 55276 7486540 7 0.0% pdflush
19 40522 5495462 7 0.0% pdflush
20 0 2 2 0.0% kswapd0
21 0 2 1 0.0% aio/0
22 0 2 1 0.0% aio/1
23 0 2 1 0.0% nfsiod
24 0 2 1 0.0% kide/0
25 0 2 1 0.0% kide/1
28 0 2 0 0.0% scsi_eh_0
29 122281 19239317 6 0.0% usb-storage
31 0 17 9 0.0% rpciod/0
32 0 2 1 0.0% rpciod/1
33 106717 23873879 4 0.0% kirqd
477 3 2 1902 0.0% jffs2_gcd_mtd2
583 54963 2270699 24 0.0% kjournald
588 31840 2153265 14 0.0% kjournald
1253 28592 2063500 13 0.0% kjournald
1262 29873 2000840 14 0.0% kjournald
1755 1 30 41 0.0% portmap
1764 4 6 743 0.0% rpc.statd
1779 0 2 3 0.0% lockd
1780 65 16660 3 0.0% nfsd
1781 63 16662 3 0.0% nfsd
1782 62 16668 3 0.0% nfsd
1783 63 16675 3 0.0% nfsd
1784 64 16660 3 0.0% nfsd
1785 62 16654 3 0.0% nfsd
1786 66 16666 3 0.0% nfsd
1787 68 16662 4 0.0% nfsd
1792 0 1 119 0.0% rpc.mountd
1815 5322251 136397414 39 0.0% sysmgr
2496 0 1 45 0.0% mping-thread
2731 31885 5992170 5 0.0% insmod
2859 0 2 2 0.0% bond0
3152 8 2 4183 0.0% xinetd
3153 5 28 195 0.0% tftpd
3154 10715894 67595779 158 0.0% syslogd
3155 140 643 218 0.0% sdwrapd
3156 521695433 189861271 2747 0.0% pfma
3163 130866 29963846 4 0.0% ls-notify-mts-t
3169 6044747 137075461 44 0.0% usd_mts_kthread
3176 0 1 10 0.0% pfm_gem_hp_kern
3180 1134583 33772087 33 0.0% vshd
3181 93 12 7828 0.0% smm
3182 561672 60342908 9 0.0% psshelper
3183 48754 1024189 47 0.0% lmgrd
3184 375078 13930383 26 0.0% licmgr
3185 68498 1242147 55 0.0% fs-daemon
3186 374383 14013862 26 0.0% feature-mgr
3187 182806 12127579 15 0.0% fcfwd
3188 1057 11660 90 0.0% confcheck
3189 186474 9150877 20 0.0% capability
3190 565623 60345112 9 0.0% psshelper_gsvc
3200 349976 2169854 161 0.0% cisco
3201 180096 6399991 28 0.0% clis
3203 88 234 378 0.0% klogd
3213 124210932 111061977 1118 0.0% port-profile
3216 205040 9098134 22 0.0% provision
3218 391479 18193906 21 0.0% zschk
3219 312034 20174304 15 0.0% xmlma
3250 1603650683 2147483647 603 1.9% carmelusd
3251 300005 30209163 9 0.0% bootvar
3252 138736 12054114 11 0.0% bios_daemon
3253 307054 15968158 19 0.0% bcm_usd
3254 109274 5986517 18 0.0% athertonusd
3255 113112 7575356 14 0.0% ascii-cfg
3256 656607 60397458 10 0.0% securityd
3257 644711 60468710 10 0.0% cert_enroll
3258 13070229 133247478 98 0.0% aaa
...
CPU util : 1.5% user, 6.0% kernel, 92.5% idle
Please note that only processes from the requested vdc are shown above
Same command but sort it by Utilization
TPA-C12-L2-LEAF101# show processes cpu sort (sort Show processes CPU Info (Sorted by Cpu Util))
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3250 1603668699 2147483647 603 1.9% carmelusd
3156 521698718 189863058 2747 0.9% pfma
3445 96586350 1041812161 92 0.9% cfs
1 784912 14217893 55 0.0% init
2 3 294 10 0.0% kthreadd
3 7494 1466732 5 0.0% migration/0
4 3051792 778012878 3 0.0% ksoftirqd/0
5 259814 9291985 27 0.0% watchdog/0
6 6029 1150843 5 0.0% migration/1
7 3220083 822354324 3 0.0% ksoftirqd/1
8 29440 9291985 3 0.0% watchdog/1
9 8112475 112796825 71 0.0% events/0
10 2668406 91944741 29 0.0% events/1
11 0 56 16 0.0% khelper
12 24 3468 7 0.0% kblockd/0
13 19 1853 10 0.0% kblockd/1
14 0 2 0 0.0% kacpid
15 0 2 0 0.0% kacpi_notify
16 28 9 3141 0.0% ksuspend_usbd
17 9 275 33 0.0% khubd
18 55276 7486540 7 0.0% pdflush
19 40523 5495547 7 0.0% pdflush
20 0 2 2 0.0% kswapd0
21 0 2 1 0.0% aio/0
22 0 2 1 0.0% aio/1
23 0 2 1 0.0% nfsiod
24 0 2 1 0.0% kide/0
25 0 2 1 0.0% kide/1
28 0 2 0 0.0% scsi_eh_0
29 122281 19239438 6 0.0% usb-storage
31 0 17 9 0.0% rpciod/0
32 0 2 1 0.0% rpciod/1
33 106718 23874035 4 0.0% kirqd
477 3 2 1902 0.0% jffs2_gcd_mtd2
583 54964 2270715 24 0.0% kjournald
588 31841 2153279 14 0.0% kjournald
1253 28593 2063513 13 0.0% kjournald
1262 29873 2000852 14 0.0% kjournald
1755 1 30 41 0.0% portmap
1764 4 6 743 0.0% rpc.statd
1779 0 2 3 0.0% lockd
1780 65 16660 3 0.0% nfsd
1781 63 16662 3 0.0% nfsd
1782 62 16668 3 0.0% nfsd
1783 63 16675 3 0.0% nfsd
1784 64 16660 3 0.0% nfsd
...
CPU util : 1.0% user, 0.5% kernel, 98.5% idle
Please note that only processes from the requested vdc are shown above
Check VPC process
TPA-C12-L2-LEAF101# show processes cpu | egrep "PID|--|vpc" PID Runtime(ms) Invoked uSecs 1Sec Process ----- ----------- -------- ----- ------ ----------- 29214 6669934 129029587 51 0.0% vpc
PID Lookup
TPA-C12-L2-LEAF101# show system internal sysmgr service pid 29214
Service "vpc" ("vpc", 142):
UUID = 0x251, PID = 29214, SAP = 450
State: SRV_STATE_HANDSHAKED (entered at time Mon Aug 24 13:55:18 2009).
Restart count: 1
Time of last restart: Mon Aug 24 13:55:17 2009.
The service never crashed since the last reboot.
Tag = N/A
Plugin ID: 0
Name Lookup
TPA-C12-L2-LEAF101# show system internal sysmgr service name vpc
Service "vpc" ("vpc", 142):
UUID = 0x251, PID = 29214, SAP = 450
State: SRV_STATE_HANDSHAKED (entered at time Mon Aug 24 13:55:18 2009).
Restart count: 1
Time of last restart: Mon Aug 24 13:55:17 2009.
The service never crashed since the last reboot.
Tag = N/A
Plugin ID: 0
CPU – Traffic caused High CPU utilization and control-plane instability
TPA-C1-L2# show hardware internal cpu-mac inband stats | egrep " Rx|Tx|counters|Throttle|Tick|rate|total|good|XOFF p|XON p" RMON counters Rx Tx total packets 907390555 654768557 good packets 907390555 654768557 good octets 149103253191 100263694360 total octets 35 23 XON packets 0 0 XOFF packets 0 0 Interrupt counters Tx packet timer 0 Tx absolute timer 907390555 Tx queue empty 654768557 Tx descr thresh low 0 Error counters Tx no CRS ..................... 0 FC Rx unsupported .............. 0 Tx TCP segmentation context .... 0 Tx TCP segmentation context fail 0 Throttle statistics Throttle interval ........... 2 * 100ms Packet rate limit ........... 64000 pps Tick counter ................ 23832007 Rx packet rate (current/max) 302 / 2686 pps Tx packet rate (current/max) 186 / 2581 pps Tx packets ..... 654768557 Tx queue depth . 10000 Tx src_p stamp . 32 CPU bound traffic current pps /maximum pps reached
Look at the packet rate as well as type of traffic (unicast or multicast) for CPU-punted traffic from an interface
TPA-C1-L2# show system internal pktmgr interface vlan 64
Vlan64, ordinal: 288 Hash_type: 1
SUP-traffic statistics: (sent/received)
Packets: 36773042 / 37818975
Bytes: 3863673713 / 4529696953
Instant packet rate: 2 pps / 2 pps
Packet rate limiter (Out/In): 0 pps / 0 pps
Average packet rates(1min/5min/15min/EWMA):
Packet statistics:
Tx: Unicast 7221723, Multicast 29541346
Broadcast 9973
Rx: Unicast 6310556, Multicast 31305574
Broadcast 202845
Check CoPP and HWRL (hardware rate limiter) in order to help identify the type of traffic.
show policy-map interface control-plane module 2 class copp-system-class-malicious