ESXi5x-HPSA P410i WARNING: LinScsi: SCSILinuxAbortCommands:1843 and Lost access to volume

[DigitalNomadIL]

Datastore / Disk latency problems with HP ProLiant G7 - HP Smart Array P410i controller " WARNING: LinScsi: SCSILinuxAbortCommands:1843: " and "Lost access to volume" (Still an issue with hpsa 5.5.0.114-1OEM )

We've been tracking this issue in the VMWare forums which I encountered fairly recently on my G7 series VMWare hosts when updating to Cookbook Release September 2015 for details you can follow that post here: https://communities.vmware.com/message/2562449#2562449

After updating a mixture of G7, G8 and G9 VMHosts to 5.5 update 3a and Cookbook release September 2015 (SPP JUN 15). I started having host errors specifically on my G7 hardware. One host went so far as to disconnect from VCenter.

 I went the full boat update on the OS to bring it fully in line with HP's recipe. This was the first time hitting drivers in quite a while. So when the first errors started coming in, I immediately suspected the HPSA v106 ( Version:5.5.0.106-1OEM ) which was updated from v50 ( Version:5.5.0.50-1OEM)

VCenter is reporting errors

Lost access to volume 56424481-7f094eb0-8ee6-

80c16e6e15e0 (VMHost_local) due to

connectivity issues. Recovery attempt is in

progress and outcome will be reported shortly.

info 2/2/2015 9:00:45 AM (VMHost_local)

VMKernel.log was reporting some conflict's with claim rules between PowerPath and the NMP for the local disk but that was cleared.

2015-11-29T07:53:04.162Z cpu22:33327)WARNING: LinScsi: SCSILinuxAbortCommands:1843: Failed, Driver hpsa, for vmhba1

Hostd.log

2015-11-30T19:37:48.614Z [248C4B70 info 'Vimsvc.ha-eventmgr'] Event 1820 : Lost access to volume 4ffd89b4-760e9689-81e3-e83935a81a45 (vmhost_local) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

2015-11-30T19:37:48.615Z [248C4B70 info 'Vimsvc.ha-eventmgr'] Event 1821 : Successfully restored access to volume 4ffd89b4-760e9689-81e3-e83935a81a45 (vmhost_local) following connectivity issues.

I opened tickets with both HP and VMWare

VMWare came back with the first fix which was to upgrade the hpsa driver(hpsa 5.5.0.114-1OEM) which can be downloaded from https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-HP-HPSA-550114-1OEM&productId=353. however overnight , the errors returned. Same occurred with posted version 116

HP support suggested back revving to version 84 then 74 of the driver which failed. In the end we have tried the following versions of the driver which have failed 116, 114,84,74 and the last stable version was 60. This has been a long standing issue.

So in Summary

HPSA driver later than v60 driving a HP Smart Array p410 controller of any firmware vintage with a spare drive configured. The spare drive configuration seems to cause an i/o back-feed into the driver blowing it up at irregular intervals. This issue will not surface unless a spare drive is configured on the array controller.

 Existing HP case # from VMWare Engineering is 4651641937   Feel free to jump on the boat this is a long standing issue

 Kindest Regards,

DGN