Cisco Nexus Show Commands for High CPU Usage

I ran these on our Nexus 7K switches.

Display the CPU utilization

TPA-C12-L2-LEAF101# show processes cpu

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process
-----  -----------  --------  -----  ------  -----------
    1       784907  14217801     55    0.0%  init
    2            3       294     10    0.0%  kthreadd
    3         7494   1466724      5    0.0%  migration/0
    4      3051755  778000403      3    0.0%  ksoftirqd/0
    5       259812   9291924     27    0.0%  watchdog/0
    6         6029   1150831      5    0.0%  migration/1
    7      3220043  822340807      3    0.0%  ksoftirqd/1
    8        29439   9291924      3    0.0%  watchdog/1
    9      8112419  112796026     71    0.0%  events/0
   10      2668389  91944131     29    0.0%  events/1
   11            0        56     16    0.0%  khelper
   12           24      3468      7    0.0%  kblockd/0
   13           19      1853     10    0.0%  kblockd/1
   14            0         2      0    0.0%  kacpid
   15            0         2      0    0.0%  kacpi_notify
   16           28         9   3141    0.0%  ksuspend_usbd
   17            9       275     33    0.0%  khubd
   18        55276   7486540      7    0.0%  pdflush
   19        40522   5495462      7    0.0%  pdflush
   20            0         2      2    0.0%  kswapd0
   21            0         2      1    0.0%  aio/0
   22            0         2      1    0.0%  aio/1
   23            0         2      1    0.0%  nfsiod
   24            0         2      1    0.0%  kide/0
   25            0         2      1    0.0%  kide/1
   28            0         2      0    0.0%  scsi_eh_0
   29       122281  19239317      6    0.0%  usb-storage
   31            0        17      9    0.0%  rpciod/0
   32            0         2      1    0.0%  rpciod/1
   33       106717  23873879      4    0.0%  kirqd
  477            3         2   1902    0.0%  jffs2_gcd_mtd2
  583        54963   2270699     24    0.0%  kjournald
  588        31840   2153265     14    0.0%  kjournald
 1253        28592   2063500     13    0.0%  kjournald
 1262        29873   2000840     14    0.0%  kjournald
 1755            1        30     41    0.0%  portmap
 1764            4         6    743    0.0%  rpc.statd
 1779            0         2      3    0.0%  lockd
 1780           65     16660      3    0.0%  nfsd
 1781           63     16662      3    0.0%  nfsd
 1782           62     16668      3    0.0%  nfsd
 1783           63     16675      3    0.0%  nfsd
 1784           64     16660      3    0.0%  nfsd
 1785           62     16654      3    0.0%  nfsd
 1786           66     16666      3    0.0%  nfsd
 1787           68     16662      4    0.0%  nfsd
 1792            0         1    119    0.0%  rpc.mountd
 1815      5322251  136397414     39    0.0%  sysmgr
 2496            0         1     45    0.0%  mping-thread
 2731        31885   5992170      5    0.0%  insmod
 2859            0         2      2    0.0%  bond0
 3152            8         2   4183    0.0%  xinetd
 3153            5        28    195    0.0%  tftpd
 3154     10715894  67595779    158    0.0%  syslogd
 3155          140       643    218    0.0%  sdwrapd
 3156    521695433  189861271   2747    0.0%  pfma
 3163       130866  29963846      4    0.0%  ls-notify-mts-t
 3169      6044747  137075461     44    0.0%  usd_mts_kthread
 3176            0         1     10    0.0%  pfm_gem_hp_kern
 3180      1134583  33772087     33    0.0%  vshd
 3181           93        12   7828    0.0%  smm
 3182       561672  60342908      9    0.0%  psshelper
 3183        48754   1024189     47    0.0%  lmgrd
 3184       375078  13930383     26    0.0%  licmgr
 3185        68498   1242147     55    0.0%  fs-daemon
 3186       374383  14013862     26    0.0%  feature-mgr
 3187       182806  12127579     15    0.0%  fcfwd
 3188         1057     11660     90    0.0%  confcheck
 3189       186474   9150877     20    0.0%  capability
 3190       565623  60345112      9    0.0%  psshelper_gsvc
 3200       349976   2169854    161    0.0%  cisco
 3201       180096   6399991     28    0.0%  clis
 3203           88       234    378    0.0%  klogd
 3213    124210932  111061977   1118    0.0%  port-profile
 3216       205040   9098134     22    0.0%  provision
 3218       391479  18193906     21    0.0%  zschk
 3219       312034  20174304     15    0.0%  xmlma
 3250   1603650683  2147483647    603    1.9%  carmelusd
 3251       300005  30209163      9    0.0%  bootvar
 3252       138736  12054114     11    0.0%  bios_daemon
 3253       307054  15968158     19    0.0%  bcm_usd
 3254       109274   5986517     18    0.0%  athertonusd
 3255       113112   7575356     14    0.0%  ascii-cfg
 3256       656607  60397458     10    0.0%  securityd
 3257       644711  60468710     10    0.0%  cert_enroll
 3258     13070229  133247478     98    0.0%  aaa
...

CPU util  :   1.5% user,   6.0% kernel,   92.5% idle
Please note that only processes from the requested vdc are shown above

Same command but sort it by Utilization

TPA-C12-L2-LEAF101# show processes cpu sort (sort Show processes CPU Info (Sorted by Cpu Util))

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process
-----  -----------  --------  -----  ------  -----------
 3250   1603668699  2147483647    603    1.9%  carmelusd
 3156    521698718  189863058   2747    0.9%  pfma
 3445     96586350  1041812161     92    0.9%  cfs
    1       784912  14217893     55    0.0%  init
    2            3       294     10    0.0%  kthreadd
    3         7494   1466732      5    0.0%  migration/0
    4      3051792  778012878      3    0.0%  ksoftirqd/0
    5       259814   9291985     27    0.0%  watchdog/0
    6         6029   1150843      5    0.0%  migration/1
    7      3220083  822354324      3    0.0%  ksoftirqd/1
    8        29440   9291985      3    0.0%  watchdog/1
    9      8112475  112796825     71    0.0%  events/0
   10      2668406  91944741     29    0.0%  events/1
   11            0        56     16    0.0%  khelper
   12           24      3468      7    0.0%  kblockd/0
   13           19      1853     10    0.0%  kblockd/1
   14            0         2      0    0.0%  kacpid
   15            0         2      0    0.0%  kacpi_notify
   16           28         9   3141    0.0%  ksuspend_usbd
   17            9       275     33    0.0%  khubd
   18        55276   7486540      7    0.0%  pdflush
   19        40523   5495547      7    0.0%  pdflush
   20            0         2      2    0.0%  kswapd0
   21            0         2      1    0.0%  aio/0
   22            0         2      1    0.0%  aio/1
   23            0         2      1    0.0%  nfsiod
   24            0         2      1    0.0%  kide/0
   25            0         2      1    0.0%  kide/1
   28            0         2      0    0.0%  scsi_eh_0
   29       122281  19239438      6    0.0%  usb-storage
   31            0        17      9    0.0%  rpciod/0
   32            0         2      1    0.0%  rpciod/1
   33       106718  23874035      4    0.0%  kirqd
  477            3         2   1902    0.0%  jffs2_gcd_mtd2
  583        54964   2270715     24    0.0%  kjournald
  588        31841   2153279     14    0.0%  kjournald
 1253        28593   2063513     13    0.0%  kjournald
 1262        29873   2000852     14    0.0%  kjournald
 1755            1        30     41    0.0%  portmap
 1764            4         6    743    0.0%  rpc.statd
 1779            0         2      3    0.0%  lockd
 1780           65     16660      3    0.0%  nfsd
 1781           63     16662      3    0.0%  nfsd
 1782           62     16668      3    0.0%  nfsd
 1783           63     16675      3    0.0%  nfsd
 1784           64     16660      3    0.0%  nfsd
...

CPU util  :   1.0% user,   0.5% kernel,   98.5% idle
Please note that only processes from the requested vdc are shown above

Check VPC process

TPA-C12-L2-LEAF101# show processes cpu | egrep "PID|--|vpc"
PID    Runtime(ms)  Invoked   uSecs  1Sec    Process
-----  -----------  --------  -----  ------  -----------
29214      6669934  129029587     51    0.0%  vpc

PID Lookup

TPA-C12-L2-LEAF101# show system internal sysmgr service pid 29214
Service "vpc" ("vpc", 142):
        UUID = 0x251, PID = 29214, SAP = 450
        State: SRV_STATE_HANDSHAKED (entered at time Mon Aug 24 13:55:18 2009).
        Restart count: 1
        Time of last restart: Mon Aug 24 13:55:17 2009.
        The service never crashed since the last reboot.
        Tag = N/A
        Plugin ID: 0

Name Lookup

TPA-C12-L2-LEAF101# show system internal sysmgr service name vpc
Service "vpc" ("vpc", 142):
        UUID = 0x251, PID = 29214, SAP = 450
        State: SRV_STATE_HANDSHAKED (entered at time Mon Aug 24 13:55:18 2009).
        Restart count: 1
        Time of last restart: Mon Aug 24 13:55:17 2009.
        The service never crashed since the last reboot.
        Tag = N/A
        Plugin ID: 0

CPU – Traffic caused High CPU utilization and control-plane instability

TPA-C1-L2# show hardware internal cpu-mac inband stats | egrep " Rx|Tx|counters|Throttle|Tick|rate|total|good|XOFF p|XON p"
RMON counters                            Rx                   Tx
total packets                     907390555            654768557
good packets                      907390555            654768557
good octets                    149103253191         100263694360
total octets                             35                   23
XON packets                               0                    0
XOFF packets                              0                    0
Interrupt counters
Tx packet timer     0
Tx absolute timer   907390555
Tx queue empty      654768557
Tx descr thresh low 0
Error counters
Tx no CRS  ..................... 0
FC Rx unsupported .............. 0
Tx TCP segmentation context .... 0
Tx TCP segmentation context fail 0
Throttle statistics
Throttle interval ........... 2 * 100ms
Packet rate limit ........... 64000 pps
Tick counter ................ 23832007
Rx packet rate (current/max)  302 / 2686 pps
Tx packet rate (current/max)  186 / 2581 pps
Tx packets ..... 654768557
Tx queue depth . 10000
Tx src_p stamp . 32

CPU bound  traffic current pps /maximum pps reached

Look at the packet rate as well as type of traffic (unicast or multicast) for CPU-punted traffic from an interface

TPA-C1-L2#  show system internal pktmgr interface vlan 64
Vlan64, ordinal: 288  Hash_type: 1
  SUP-traffic statistics: (sent/received)
    Packets: 36773042 / 37818975
    Bytes: 3863673713 / 4529696953
    Instant packet rate: 2 pps / 2 pps
    Packet rate limiter (Out/In): 0 pps / 0 pps
    Average packet rates(1min/5min/15min/EWMA):
    Packet statistics:
      Tx: Unicast 7221723, Multicast 29541346
          Broadcast 9973
      Rx: Unicast 6310556, Multicast 31305574
          Broadcast 202845

Check CoPP and HWRL (hardware rate limiter) in order to help identify the type of traffic.

show policy-map interface control-plane module 2 class copp-system-class-malicious
More Stories
Data Center Temperature Best Practices