Comware Based
1748209 Members
3001 Online
108759 Solutions
New Discussion

Re: High kdrvfwdd1 cpu usage and network slow down dramatically

 
MJaat
Occasional Advisor

High kdrvfwdd1 cpu usage and network slow down dramatically

Hi,

 

On normally:

<>display process cpu

54 0.5% 0.4% 8.0% [kdrvfwdd1]

 

When this problem occurs and i notice it(too late):

<>display process cpu

54 45.8% 47.3% 48.9% [kdrvfwdd1]

 

Normally ping to the router gives <1ms and google.com ~27ms.

When this problem occurs ping to router is about >200ms and outside world, example google.com gives >1s.

What is kdrvfwdd1 and where could i start to searching/mitigating this problem.

Occurs about 4times per month.

 

Thank you for your post.

 

 

4 REPLIES 4
Ivan_B
HPE Pro

Re: High kdrvfwdd1 cpu usage and network slow down dramatically

Hi @MJaat !

This is a kernel driver forwarding process. If it takes a lot of CPU cycles, it means the transit traffic it receives needs CPU processing instead of forwarding in ASIC (hardware forwarding).

What is the device model number and s/w version running on it?

 

I am an HPE employee

Accept or Kudo

MJaat
Occasional Advisor

Re: High kdrvfwdd1 cpu usage and network slow down dramatically

Hi,

 

Sorry late reply.

Model MSR2003.

The affected version was version 7.1.064, Release 0605P13.

Updated id to version 7.1.064, Release 0809P27.

 

Lets see if the problem continues.

MJaat
Occasional Advisor

Re: High kdrvfwdd1 cpu usage and network slow down dramatically

This is persistent one:

<normain>display process
       JID        PID %CPU %MEM STAT PRI     TTY HH:MM:SS COMMAND
         1          1  1.4  0.0   S  120      -  00:00:37 scmd
         2          2  0.0  0.0   S  115      -  00:00:00 [kthreadd]
         3          3  0.0  0.0   S   99      -  00:00:00 [migration/0]
         4          4  0.0  0.0   S  115      -  00:00:25 [ksoftirqd/0]
         5          5  0.0  0.0   S   99      -  00:00:05 [watchdog/0]
         6          6  0.0  0.0   S   99      -  00:00:00 [migration/1]
         7          7  0.0  0.0   S  115      -  00:00:00 [ksoftirqd/1]
         8          8  0.0  0.0   R   99      -  00:00:05 [watchdog/1]
         9          9  0.0  0.0   S  115      -  00:00:11 [events/0]
        10         10  0.0  0.0   R  115      -  00:00:12 [events/1]
        11         11  0.0  0.0   S  115      -  00:00:00 [khelper]
        12         12  0.0  0.0   S  115      -  00:00:00 [kblockd/0]
        13         13  0.0  0.0   S  115      -  00:00:00 [kblockd/1]
        14         14  0.0  0.0   S  115      -  00:00:00 [khubd]
        15         15  0.0  0.0   S  120      -  00:00:00 [vzmond]
        16         16  0.0  0.0   S  120      -  00:00:00 [pdflush]
        17         17  0.0  0.0   S  120      -  00:00:05 [pdflush]
        18         18  0.0  0.0   S  115      -  00:00:00 [kswapd0]
        19         19  0.0  0.0   S  115      -  00:00:00 [aio/0]
        20         20  0.0  0.0   S  115      -  00:00:00 [aio/1]
        21         21  0.0  0.0   S  115      -  00:00:00 [mtdblockd]
        43         43  0.0  0.0   S  115      -  00:00:14 [SPI_HOST.0]
        46         46  0.0  0.0   S  115      -  00:00:00 [BTTY_DT]
        47         47  0.0  0.0   S  115      -  00:00:00 [BTTY_CT]
        48         48  0.0  0.0   S  115      -  00:00:52 [BTTY_TXT]
        60         60  0.0  0.0   S  115      -  00:00:00 [DRV_MboardInt]
        61         61  0.0  0.0   S  115      -  00:00:17 [commontask]
        62         62  0.0  0.0   D  115      -  00:02:47 [drv_txdone]
        63         63  0.0  0.0   S  115      -  00:00:00 [STTY_DT]
        64         64  0.0  0.0   S  115      -  00:00:00 [STTY_CT]
        67         67 49.3  0.0   R  115      -  07:11:09 [kdrvfwdd1]
        72         72  0.0  0.0   D  115      -  00:00:41 [DRV_STACK]
        73         73  0.0  0.0   S  115      -  00:00:00 [kdrvmac]
        74         74  0.0  0.0   S  115      -  00:00:19 [kdrvlswtmr]
        77         77  0.0  0.0   D  100      -  00:00:00 [lipc_topology]
        79         79  0.0  0.0   S  120      -  00:00:00 [LOAD]
        80         80  0.0  0.0   S  120      -  00:00:00 [LOADProc]
        81         81  0.0  0.0   S  120      -  00:00:00 [timesyncs]
        82         82  0.0  0.0   S  115      -  00:00:00 [lipc_portevt]
        83         83  0.0  0.0   S  115      -  00:00:00 [kfastlog]
        84         84  0.0  0.0   S  115      -  00:00:01 [kuserlog]
        85         85  0.0  0.0   S  105      -  00:00:00 [kevent]
        86         86  0.0  0.0   S  125      -  00:00:00 [kifupdown]
        87         87  0.0  0.0   S  120      -  00:03:21 [kmac/1]
        88         88  0.0  0.0   S  120      -  00:00:00 [kipfs/1]
        89         89  0.0  0.0   S  120      -  00:00:00 [kip6fs/1]
        90         90  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/0]
        91         91  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/2]
        92         92  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/3]
        93         93  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/4]
        94         94  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/5]
        95         95  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/6]
        96         96  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/7]
        97         97  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/8]
        98         98  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/9]
        99         99  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/10]
       100        100  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/11]
       101        101  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/12]
       102        102  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/13]
       103        103  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/14]
       104        104  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/15]
       105        105  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/16]
       106        106  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/17]
       107        107  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/18]
       108        108  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/19]
       109        109  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/20]
       110        110  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/21]
       111        111  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/22]
       112        112  0.0  0.0   S  115      -  00:00:00 [ktcp_proxy/1/23]
       113        113  0.0  0.0   S  115      -  00:00:00 [DRV_BisEpoll_LS]
       114        114  0.0  0.0   S  115      -  00:00:00 [DRV_BisEpoll_LG]
       115        115  0.0  0.0   S  120      -  00:00:03 [mbuf_main]
       116        116  0.0  0.0   S  105      -  00:03:59 [sock/1]
       117        117  0.0  0.0   S  120      -  00:00:00 [kevimac/1]
       118        118  0.0  0.0   S  115      -  00:00:00 [DIM_KTHREAD1]
       119        119  0.0  0.0   S  115      -  00:00:00 [kips/1]
       120        120  0.0  0.0   S  115      -  00:00:00 [kntop_AVCntop/1]
       121        121  0.0  0.0   S  115      -  00:00:01 [kntop_ntop/1]
       122        122  0.0  0.0   S  115      -  00:00:00 [kntop_Auditntop]
       125        125  0.0  0.0   S  110      -  00:00:00 cioctld
       126        126  0.0  0.0   S  120      -  00:00:00 fsd
       128        128  0.0  0.0   S  120      -  00:00:02 licd --mode=3 --did=1
       129        129  0.0  0.4   S  100      -  00:02:41 dbmd
       132        132  4.3  0.0   S  120      -  04:56:41 diagd -dflash --memale
       134        134  0.0  0.0   S  105      -  00:00:00 had
       135        135  0.0  0.0   S  120      -  00:06:04 syslogd --logfile=1 --
       136        136  0.0  0.0   S  120      -  00:00:00 pkg_licd
       137        137  0.0  0.0   S  120      -  00:02:07 cryptomgrd
       138        138  0.0  0.0   S  120      -  00:12:21 devd --TimeSourceInter
       142        142  0.0  0.0   S  120      -  00:02:05 ifmgr --mpmode=0 --mfr
       143        143  0.0  0.0   S  120      -  00:00:00 edev --FanDirection --
       150        150  0.0  0.0   S  115      -  00:00:07 [DRV0/0]
       151        151  0.0  0.0   S  115      -  00:02:35 [DRV0/1]
       152        152  0.0  0.0   S  115      -  00:02:38 [DRV0/2]
       153        153  0.0  0.0   S  115      -  00:00:07 [DRV0/3]
       156        156  0.0  0.0   S  120      -  00:00:34 goldd
       157        157  0.0  0.0   S  120      -  00:01:06 moduleproxyd
       160        160  0.0  0.0   S  120      -  00:00:27 ttymgrd --MaxVTYNum=64
       161        161  0.0  0.0   S  120      -  00:00:00 sysmand
       162        162  0.0  0.2   S  120      -  00:00:57 lauthd --MaxManageUser
       163        163  0.0  0.1   S  120      -  00:00:00 cfad
       164        164  0.0  0.0   S  120      -  00:00:58 aaad
       165        165  0.0  0.0   S  110      -  00:09:36 laggd
       166        166  0.0  0.0   S  120      -  00:00:00 tranged
       167        167  0.0  0.0   S  120      -  00:00:00 vland --QinQ --Etherne
       175        175  0.0  0.1   S  120      -  00:00:42 aclmgrd
       180        180  0.0  0.0   S  120      -  00:01:36 ethd --MACCFG=0x19 --e
       181        181  0.0  0.0   S  110      -  00:00:00 coppd
       188        188  0.0  0.1   S  120      -  00:00:01 qosd --LogicIfType=63
       189        189  0.0  1.1   S  120      -  00:01:01 comsh server
       194        194  0.0  0.1   S  120      -  00:03:18 urlcfgd --HttpEnable
       195        195  0.0  0.0   S  120      -  00:00:01 cloudagentd
       196        196  0.0  0.0   S  120      -  00:00:32 wsald
       197        197  0.0  0.7   S  120      -  00:03:54 xmlcfgd --WebStyle=11
       202        202  0.0  0.1   S  125      -  00:00:08 httpd -w 2 -c 1024
       205        205  0.0  0.0   S  100      -  00:00:00 [loop0]
       208        208  0.0  0.0   S  100      -  00:00:00 [loop1]
       210        210  0.0  0.0   S  100      -  00:00:00 [loop2]
       212        212  0.0  0.0   S  100      -  00:00:00 [loop3]
       214        214  0.0  0.0   S  100      -  00:00:00 [loop4]
       216        216  0.0  0.0   S  100      -  00:00:00 [loop5]
       218        218  0.0  0.0   S  100      -  00:00:00 [loop6]
       220        220  0.0  0.0   S  100      -  00:00:00 [loop7]
       222        222  0.0  0.0   S  100      -  00:00:00 [loop8]
       224        224  0.0  0.0   S  100      -  00:00:00 [loop9]
       226        226  0.0  0.0   S  100      -  00:00:00 [loop10]
       228        228  0.0  0.0   S  100      -  00:00:00 [loop11]
       230        230  0.0  0.0   S  100      -  00:00:00 [loop12]
       232        232  0.0  0.0   S  100      -  00:00:00 [loop13]
       234        234  0.0  0.0   S  100      -  00:00:00 [loop14]
       236        236  0.0  0.0   S  100      -  00:00:00 [loop15]
       239        239  0.0  0.0   S  100      -  00:00:00 [loop16]
       242        242  0.0  0.0   S  100      -  00:00:00 [loop17]
       244        244  0.0  0.0   S  100      -  00:00:00 [loop18]
       246        246  0.0  0.0   S  100      -  00:00:00 [loop19]
       248        248  0.0  0.0   S  100      -  00:00:00 [loop20]
       250        250  0.0  0.0   S  100      -  00:00:00 [loop21]
       252        252  0.0  0.0   S  100      -  00:00:00 [loop22]
       254        254  0.0  0.0   S  100      -  00:00:00 [loop23]
       256        256  0.0  0.0   S  100      -  00:00:00 [loop24]
       258        258  0.0  0.0   S  100      -  00:00:00 [loop25]
       260        260  0.0  0.0   S  100      -  00:00:00 [loop26]
       262        262  0.0  0.0   S  100      -  00:00:00 [loop27]
       264        264  0.0  0.0   S  100      -  00:00:00 [loop28]
       266        266  0.0  0.0   S  100      -  00:00:00 [loop29]
       268        268  0.0  0.0   S  100      -  00:00:00 [loop30]
       270        270  0.0  0.0   S  100      -  00:00:00 [loop31]
       272        272  0.0  0.0   S  100      -  00:00:00 [loop32]
       274        274  0.0  0.0   S  100      -  00:00:00 [loop33]
       276        276  0.0  0.0   S  100      -  00:00:00 [loop34]
       278        278  0.0  0.0   S  100      -  00:00:00 [loop35]
       280        280  0.0  0.0   S  100      -  00:00:00 [loop36]
       282        282  0.0  0.0   S  100      -  00:00:00 [loop37]
       284        284  0.0  0.0   S  100      -  00:00:00 [loop38]
       286        286  0.0  0.0   S  100      -  00:00:00 [loop39]
       288        288  0.0  0.0   S  100      -  00:00:00 [loop40]
       290        290  0.0  0.0   S  100      -  00:00:00 [loop41]
       292        292  0.0  0.0   S  100      -  00:00:00 [loop42]
       294        294  0.0  0.0   S  100      -  00:00:00 [loop43]
       296        296  0.0  0.0   S  100      -  00:00:00 [loop44]
       298        298  0.0  0.0   S  100      -  00:00:00 [loop45]
       300        300  0.0  0.0   S  100      -  00:00:00 [loop46]
       302        302  0.0  0.0   S  100      -  00:00:00 [loop47]
       304        304  0.0  0.0   S  100      -  00:00:00 [loop48]
       306        306  0.0  0.0   S  100      -  00:00:00 [loop49]
       308        308  0.0  0.0   S  100      -  00:00:00 [loop50]
       310        310  0.0  0.0   S  100      -  00:00:00 [loop51]
       312        312  0.0  0.0   S  100      -  00:00:00 [loop52]
       314        314  0.0  0.0   S  100      -  00:00:00 [loop53]
       316        316  0.0  0.0   S  100      -  00:00:00 [loop54]
       318        318  0.0  0.0   S  100      -  00:00:00 [loop55]
       320        320  0.0  0.0   S  100      -  00:00:00 [loop56]
       322        322  0.0  0.0   S  100      -  00:00:00 [loop57]
       324        324  0.0  0.0   S  100      -  00:00:00 [loop58]
       326        326  0.0  0.0   S  100      -  00:00:00 [loop59]
       328        328  0.0  0.0   S  100      -  00:00:00 [loop60]
       330        330  0.0  0.0   S  100      -  00:00:00 [loop61]
       332        332  0.0  0.0   S  100      -  00:00:00 [loop62]
       334        334  0.0  0.0   S  100      -  00:00:00 [loop63]
       336        336  0.0  0.0   S  100      -  00:00:00 [loop64]
       338        338  0.0  0.0   S  100      -  00:00:00 [loop65]
       340        340  0.0  0.0   S  100      -  00:00:00 [loop66]
       342        342  0.0  0.0   S  100      -  00:00:00 [loop67]
       344        344  0.0  0.0   S  100      -  00:00:00 [loop68]
       346        346  0.0  0.0   S  100      -  00:00:00 [loop69]
       348        348  0.0  0.0   S  100      -  00:00:00 [loop70]
       350        350  0.0  0.0   S  100      -  00:00:00 [loop71]
       352        352  0.0  0.0   S  100      -  00:00:00 [loop72]
       354        354  0.0  0.0   S  100      -  00:00:00 [loop73]
       356        356  0.0  0.0   S  100      -  00:00:00 [loop74]
       358        358  0.0  0.0   S  100      -  00:00:00 [loop75]
       360        360  0.0  0.0   S  100      -  00:00:00 [loop76]
       367        367  0.2  0.1   S  120      -  00:41:26 pppd
       372        372  0.0  0.0   S  125      -  00:00:50 ipstackd --ARPMaxStati
       373        373  0.0  0.0   S  115      -  00:12:13 [karp/1]
       374        374  0.0  0.0   S  115      -  00:01:41 [kwadj/1]
       375        375  0.0  0.0   S  115      -  00:00:03 [kfib/1]
       378        378  0.0  0.0   S  120      -  00:00:44 tunneld
       379        379  0.0  0.0   S  115      -  00:00:00 [ktunnel/1]
       387        387  0.0  0.1   S  120      -  00:00:01 routed --ipv4 --ftnsta
       397        397  0.0  0.0   S  120      -  00:00:01 staticrtd --ipv4
       400        400  0.0  0.0   S  120      -  00:00:00 sessiond --Hotbackup -
       403        403  0.0  0.3   S  120      -  00:00:04 aprd
       412        412  0.0  0.0   S  120      -  00:03:25 natd --Nat444BindIfIp
       413        413  0.0  0.1   S  125      -  00:08:57 dhcpd --maxDhcprDynami
       414        414  0.0  0.0   S  125      -  00:00:00 ipcimd
       419        419  0.1  0.0   S  120      -  00:17:27 dnsd
       421        421  0.0  0.1   S  120      -  00:00:39 ipsecd --NoReoptimizat
       422        422  0.0  0.0   S  115      -  00:00:00 [kipsec/1]
       425        425  0.0  0.0   S  120      -  00:00:00 objpd
       427        427  0.0  0.0   S  120      -  00:00:00 seczoned --membertype=
       430        430  0.0  0.0   S  120      -  00:00:02 sshd
       431        431  0.0  0.0   S  125      -  00:10:49 ntpd
       433        433  0.0  0.1   S  120      -  00:06:49 l2tpv2d
       437      97473  0.0  0.3   S  120      -  00:00:59 iked --NoReoptimizatio
       442        442  0.4  0.4   S  120      -  01:15:48 dpid --ShareMemBlockNu
       443        443  0.0  0.0   S  120      -  00:00:00 appprofiled
       454        454  0.0  0.9   S  120      -  00:03:22 ntopd
       463        463  0.0  0.0   S  120      -  00:00:00 avcd
       471        471  0.0  0.0   S  119      -  00:00:00 telnetd -I
    178886     178886  0.1  0.1   S  120      -  00:00:01 sshd
    178888     178888  0.0  0.1   S  120      -  00:00:00 login -f --ipv4 192.16
    178890     178890  0.0  0.0   S  120      -  00:00:00 comshc shell level 0
    178891     178891  0.5  0.7   S  120   pts/0 00:00:01 comsh server
    179082     179082  0.0  0.6   S  120   pts/0 00:00:00 comsh server
    179083     179083  0.0  0.6   S  120   pts/0 00:00:00 comsh server
<normain>

 

Ivan_B
HPE Pro

Re: High kdrvfwdd1 cpu usage and network slow down dramatically

Hi @MJaat !

Certainly this situation is not normal. I suggest you to capture traffic from all interfaces for 5 mins. Better one interface by one. And check if there is any anomaly in the transit traffic. There should be some traffic and a lot of it that requires CPU processing - IP packets with Options in the header, fragments etc.. Also, check the output of "display ip statistics", repeat this command 5-10 times with interval of 1-2 minutes and then see what counter increases much faster than others.

 

I am an HPE employee

Accept or Kudo