Internal Storage Enclosure Device Failure on HP ProLiant ML310e Gen8 v2

[Davjava]

Greetings,
I have HP ProLiant ML310e Gen8 v2 server running with two SSD Hard Drive in RAID 1 on Smart Array B120i SATA RAID Embedded controller that working fine for few days but recently sometimes the server is failing with 'Internal Storage Enclosure Device Failure' listed in ILO event log (sometimes on one disk and sometimes on the other).
Rebooting the server and recovery the drive array resolve the problem.
In the ADU report the Last Failure Reason indicate is Timeout (0x07) or Aborted Command (0x0e).
Could you help me understand what is causing the problem and how can resolve it?
Thank you for your help!

Smart Array B120i SATA RAID in Embedded Slot : Internal Drive Cage at Port 1I : Box 1 : Physical Drive (960 GB SATA SSD) 1I:1:4 : Physical Drive Status

SCSI Bus 0 (0x00)
SCSIID 3 (0x03)
Logical Block Size 0x0200
Total Blocks 0x6fc81ab0
Reserved Blocks 0x00010000
Drive Model ATA KINGSTON SA400S3
Drive Serial Number ***removed***
Drive Firmware Revision SBFKZ1.3
SCSI Inquiry Bits 0x00
Compaq Drive Stamped Stamped For Monitoring (0x01)
Last Failure Reason Timeout (0x07)
Device Flags 0x01
Position Device Flag Status
-------- ----------------------------- -------------
0 Drive Present And Operational Informational

More Device Flags 0x25
Position Device Flag Status
-------- ------------------------------------- -------------
0 SMART Predictive Failure Supported Informational
2 SMART Predictive Failure Is Enabled Informational
4 Device Attached To Internal Connector Informational
5 Configured Data Drive Informational

Smart Array B120i SATA RAID in Embedded Slot : Internal Drive Cage at Port 1I : Box 1 : Physical Drive (960 GB SATA SSD) 1I:1:3 : Physical Drive Status

SCSI Bus 0 (0x00)
SCSIID 2 (0x02)
Logical Block Size 0x0200
Total Blocks 0x6fc81ab0
Reserved Blocks 0x00010000
Drive Model ATA KINGSTON SA400S3
Drive Serial Number ***removed***
Drive Firmware Revision SBFKZ1.3
SCSI Inquiry Bits 0x00
Compaq Drive Stamped Stamped For Monitoring (0x01)
Last Failure Reason Aborted Command (0x0e)
Device Flags 0x01
Position Device Flag Status
-------- ----------------------------- -------------
0 Drive Present And Operational Informational

More Device Flags 0x25
Position Device Flag Status
-------- ------------------------------------- -------------
0 SMART Predictive Failure Supported Informational
2 SMART Predictive Failure Is Enabled Informational
4 Device Attached To Internal Connector Informational
5 Configured Data Drive Informational

Moderator Edit: Removed the Serial number for Privacy. 

[support_s]

System recommended content:

1. HPE ProLiant ML310e Gen8 v2 Server User Guide

2. HPE ProLiant ML310e Gen8 v2 Server Maintenance and Service Guide

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Davjava]

Thank you  very much for suggest but I had just read the guide.

In the following ADU report there aren't read/write errors but the count of  Other Time Outs are not equal to zero.

Is there the possible cause of failure?

Smart Array B120i SATA RAID in Embedded Slot : Internal Drive Cage at Port 1I : Box 1 : Physical Drive (960 GB SATA SSD) 1I:1:4 : Monitor and Performance Statistics (Since Factory)

Serial Number: confidential information erased. 
Firmware Revision SBFKZ1.3
Product Revision ATA KINGSTON SA400S3
Reference Time 0x0003ad5f
Sectors Read 0x000000062cac2734
Read Errors Hard 0x00000000
Read Errors Retry Recovered 0x00000000
Read Errors ECC Corrected 0x0000000000000000
Sectors Written 0x00000001b46529f6
Write Errors Hard 0x00000000
Write Errors Retry Recovered 0x00000000
Seek Count 0xffffffffffffffff
Seek Errors 0xffffffffffffffff
Spin Cycles 0x00000000
Spin Up Time 0x0000
Performance Test 1 0x0000
Performance Test 2 0xffff
Performance Test 3 0xffff
Performance Test 4 0xffff
Reallocation Sectors 0xffffffff
Reallocated Sectors 0xffffffff
DRQ Time Outs 0xffff
Other Time Outs 0x003c
Drive Rebuild Count 2 (0x0002)
Spin Retries 65535 (0xffff)
Recovers Failed Read 0x0000
Recovers Failed Write 0x0000
Format Errors 0x0000
Self Test Failures 0xffff
Not Ready Failures 0x00000000
Remap Abort Failures 0xffffffff
IRQ Deglitch Count 4294967295 (0xffffffff)
Bus Faults 0x00000000
Hot Plug Count 0 (0x00000000)
Track Rewrite Errors 0xffff
Write Errors After Remap 0x0000
Background Firmware Revision 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Media Failures 0x0000
Hardware Errors 0x0000
Aborted Command Failures 0x0000
Spin Up Failures 0x0000
Bad Target Count 0 (0x0000)
Predictive Failure Errors 0x00000000

[BPSingh]

Greetings!

Please ensure the B120i firmware (BIOS) and the driver are running on the latest revision. 
Reseat the cable connections between LFF non-hot-plug drive bracket and the controller. 
Item#13 https://support.hpe.com/hpesc/public/docDisplay?docId=c03911648&docLocale=en_US
If feasible, please test with compatible HPE branded drives as they will have firmware embedded on them. 

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[Davjava]

Greetings!

The B120i firmware (BIOS) version is P78 03282014 and the driver on ESXI 5.5.0 is scsi-hpvsa 5.5.0-84OEM.550.0.0.1198611 

Please could you verify for me if there are newer versions?

Thank you very much in advance for the support.

[BPSingh]

Greetings!

BIOS Version: 2014.03.28 is very old version. 

Please find the links to the latest BIOS version for HP ProLiant ML310e Gen8 v2 (P78) Server. Please click on revision history and perform incremental update (skipping 1-2 versions).

https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_b719a0a53b4142f8a205ed0c8c#tab1 

Below is the link for latest driver version (scsi-hpvsa-5.5.0.100-1OEM.550.0.0.1331820)

https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_ab4b45dc0c8f49fc8585571ea2&tab=Fixes 

Disclaimer : Please note data integrity is customer responsibility and HPE recommends having data backup frequently and before performing any activity on the server so that it can be restored during disaster recovery. 

[Davjava]

Greetings!

can I direct upgrade BIOS to last version: 2019.04.0 (12 Jun 2019) ?

Another question, on the installation istruction are indicated:

IMPORTANT: This component requires that the CRU driver be loaded before the component can run. 

The minimum CRU version for 6.0 is 6.0.8. 
The minimum CRU version for 5.5 is 5.5.4.1.
The minimum CRU version for 5.1 is 5.0.3.9.

Could you expain me how can verify the CRU version?

Thank you very much in advance for the support.

[BPSingh]

Greetings!

can I direct upgrade BIOS to last version: 2019.04.0 (12 Jun 2019) ?

Since the BIOS on the server is running on very older revision, hence its recommended to upgrade incrementally. 

Could you expain me how can verify the CRU version?

"# esxcli software vib list" should list cru and the version installed.

Or try with 

# esxcli software vib list | grep hpcru

To check if hpcru driver is loaded on server by executing following command:

~ # esxcli system module list | grep hpcru

Example:

~ # vmware version -l
VMware ESXi 5.0.0 Update 2
~ # esxcli software vib list | grep hpcru
char-hpcru 5.0.3.09-1OEM.500.0.0.434156 Hewlett-Packard PartnerSupported 2014-01-30
~ # esxcli system module list | grep hpcru
hpcru true true

[Davjava]

Greetings!

Below the result of CRU verify on my server...there is only a difference from your example "HP" to "HPE" is correct?

~ # vmware version -l
VMware ESXi 5.5.0 Update 3
~ # esxcli software vib list | grep cru
hpe-cru 5.5.8.8-1OEM.550.0.0.1198610 HPE PartnerSupported 2018-12-03
~ # esxcli system module list | grep cru
hpcru true true
~ #

[BPSingh]

Greetings!

Yes, hpe-cru 5.5.8.8-1OEM.550.0.0.1198610 HPE PartnerSupported 2018-12-03 is the cru version installed on the server.

[Davjava]

Greetings!

using the links that you have suggest I have upgrade to the latest BIOS version without a problem. Now the BIOS version is P78 04/04/2019. After that I have upgrade driver at version (scsi-hpvsa-5.5.0.100-1OEM.550.0.0.1331820) but post installation the server became very slow with high access time to the datastore. 
Therefore I have downgrade the driver at version (scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820) and the server it's back to work fast but the original problem that sometimes the server is failing with 'Internal Storage Enclosure Device Failure' listed in ILO event log remains.

Thank you very much for your support.

[BPSingh]

Greetings!

As informed earlier, you may reseat the cable connections between LFF non-hot-plug drive bracket and the controller and check. 
Item#13 https://support.hpe.com/hpesc/public/docDisplay?docId=c03911648&docLocale=en_US 

Before you replace any part or log a support case, you can also test with HPE branded drives.

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[Davjava]

Greetings!

The server is physical installed in one remote office. Next week I will can reseat the cable connections between LFF non-hot-plug drive bracket and the controller and check.
Alter that I will update you
Thank you very much in advance for the support.

[Davjava]

Greetings!

as suggested I repositioned the cable connections between the non-hot plug LFF drive bracket and the controller. Now I am monitoring the operating status.

I will keep you updated in the next days.
Thank you very much in advance for the support.

[Davjava]

Greetings!

unfortunately after five days the issue occurred again. What can I do again?

Thank you very much in advance for the support.

[BPSingh]

Greetings!

As advised earlier, please check with HPE branded drive to narrow down the issue. If the issue persists, I would advise you to log a support ticket so that the events in the AHS logs can be analyzed to further investigate this issue.