HP ProLiant MicroServer Gen8 - B120i controller failing

[JimiT]

Hi All,

I have HP ProLiant MicroServer Gen8 running windows 2012 r2, one of my drives recently failed (drive2) and was replaced and all seemed to go well.  Not sure when but I think just after replacing that drive, I began to see some eratic peformance problems - the system would lock up for a few seconds then continue.  This is still occuring and its been a couple of weeks behaving like this.

Now that I've had the chance to sit and go through the windows event logs, I see no errors in the application event logs however the windows system event logs are littered with these errors, but nothing being reported in SEL from ilo which is puzzling ??  And they appear to eventually recover with an informational event (I would see about 20 or so of the events directly below informing of a disk block error, then the following event is repeated about 100 times and then finally recovers with the informational - see below)

Log Name:      System
Source:        disk
Date:          5/21/2017 7:54:33 PM
Event ID:      7
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      thebox
Description:
The device, \Device\Harddisk0\DR0, has a bad block.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="49156">7</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-21T09:54:33.365742700Z" />
    <EventRecordID>430528</EventRecordID>
    <Channel>System</Channel>
    <Computer>thebox</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk0\DR0</Data>
    <Binary>030080000100000000000000070004C0000100009C0000C000000000000000000094675D2000000084CC010000000000FFFFFFFF01000000580000840200040063200AFF42072040000C00003C000000000000000000000020383C4B00E0FFFF0000000000000000D0E7BE4800E0FFFF0000000000000000CAB32E10000000002800102EB3CA00000600000000000000700003000000000A00000000110000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        Cissesrv
Date:          5/21/2017 7:54:34 PM
Event ID:      24606
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      thebox
Description:
Logical drive 1 of array controller B120i [Embedded] returned a fatal error during a read/write request from/to the volume.  

 Logical block address 271496138, block count 6 and command 32 were taken from the failed logical I/O request.  

 Array controller B120i [Embedded] is also reporting that the last physical drive to report a fatal error condition (associated with this logical request), is located  in bay 0 of box 0 connected to port ??.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Cissesrv" />
    <EventID Qualifiers="50229">24606</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-21T09:54:34.000000000Z" />
    <EventRecordID>430563</EventRecordID>
    <Channel>System</Channel>
    <Computer>thebox</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Cissesrv</Data>
    <Data>B120i [Embedded]</Data>
    <Data>1</Data>
    <Data>271496138</Data>
    <Data>6</Data>
    <Data>32</Data>
    <Data>0</Data>
    <Data>0</Data>
    <Data>??</Data>
  </EventData>
</Event>

Log Name:      System
Source:        Cissesrv
Date:          5/21/2017 7:54:34 PM
Event ID:      24579
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      thebox
Description:
An event queue overflow has occurred for the array controller B120i [Embedded]. At least one event has been lost.  This controller will only queue up to 100 events.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Cissesrv" />
    <EventID Qualifiers="17461">24579</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-21T09:54:34.000000000Z" />
    <EventRecordID>430564</EventRecordID>
    <Channel>System</Channel>
    <Computer>thebox</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Cissesrv</Data>
    <Data>B120i [Embedded]</Data>
  </EventData>
</Event>

I don't beleive this is a problem with the disk, though more to do with the controller B120i.  I am unable to use the Array Manger to connect to the controller, it locks up when I do and eventually the app crashes with:

Log Name:      Application
Source:        Application Error
Date:          5/21/2017 7:54:33 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      thebox
Description:
Faulting application name: hpssaclient.exe, version: 1.60.17.0, time stamp: 0x52a15d19
Faulting module name: ntdll.dll, version: 6.3.9600.17936, time stamp: 0x55a68e0c
Exception code: 0xc0000005
Fault offset: 0x000000000005474b
Faulting process id: 0xca0
Faulting application start time: 0x01d2d218053a15c9
Faulting application path: C:\Program Files\hp\hpssa\bin\hpssaclient.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 7d62e8a3-3e0b-11e7-80c7-645106d804e1
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-21T09:54:33.000000000Z" />
    <EventRecordID>3437</EventRecordID>
    <Channel>Application</Channel>
    <Computer>thebox</Computer>
    <Security />
  </System>
  <EventData>
    <Data>hpssaclient.exe</Data>
    <Data>1.60.17.0</Data>
    <Data>52a15d19</Data>
    <Data>ntdll.dll</Data>
    <Data>6.3.9600.17936</Data>
    <Data>55a68e0c</Data>
    <Data>c0000005</Data>
    <Data>000000000005474b</Data>
    <Data>ca0</Data>
    <Data>01d2d218053a15c9</Data>
    <Data>C:\Program Files\hp\hpssa\bin\hpssaclient.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>7d62e8a3-3e0b-11e7-80c7-645106d804e1</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Windows Error Reporting
Date:          5/21/2017 8:18:40 PM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      thebox
Description:
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: hpssaclient.exe
P2: 1.60.17.0
P3: 52a15d19
P4: ntdll.dll
P5: 6.3.9600.17936
P6: 55a68e0c
P7: c0000005
P8: 000000000005474b
P9: 
P10: 

Attached files:

These files may be available here:
C:\Users\Administrator\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_hpssaclient.exe_ae8f42aad8c1883f9a1375d127a4ec2d68ed2c2_47e07d47_07e631ab

Analysis symbol: 
Rechecking for solution: 0
Report Id: 7d62e8a3-3e0b-11e7-80c7-645106d804e1
Report Status: 2048
Hashed bucket: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-05-21T10:18:40.000000000Z" />
    <EventRecordID>3438</EventRecordID>
    <Channel>Application</Channel>
    <Computer>thebox</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>APPCRASH</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>hpssaclient.exe</Data>
    <Data>1.60.17.0</Data>
    <Data>52a15d19</Data>
    <Data>ntdll.dll</Data>
    <Data>6.3.9600.17936</Data>
    <Data>55a68e0c</Data>
    <Data>c0000005</Data>
    <Data>000000000005474b</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:\Users\Administrator\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_hpssaclient.exe_ae8f42aad8c1883f9a1375d127a4ec2d68ed2c2_47e07d47_07e631ab</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>7d62e8a3-3e0b-11e7-80c7-645106d804e1</Data>
    <Data>2048</Data>
    <Data>
    </Data>
  </EventData>
</Event>

[JimiT]

Here are some more details on the drives and controller f/w

Storage Information


The Logical view shows configured logical drives and associated physical drives. It 
does not show physical drives which are not configured as part of an array, or spare drives.

The Physical view does not show configured logical drives.


Collapse All 

-Controller on System Board

Logical View   Physical View 

Controller Status  OK  
Serial Number N/A 
Model Dynamic Smart Array B120i Controller 
Firmware Version 6.00 
Controller Type HPE Smart Array 
Cache Module Status  OK 
Cache Module Serial Number N/A 
Cache Module Memory 65536 KB 


-Physical Drive in Port 1I Bay 1
Status  OK 
Serial Number 9VP9ZRBF 
Model ST31000528AS 
Media Type HDD 
Capacity 1000 GB 
Location Port 1I Bay 1 
Firmware Version CC38 
Drive Configuration Configured 
Encryption Status Not Encrypted 


-Physical Drive in Port 2I Bay 2
Status  OK 
Serial Number Z1E8M6YG 
Model ST2000DM001-1CH1 
Media Type HDD 
Capacity 2000 GB 
Location Port 2I Bay 2 
Firmware Version CC27 
Drive Configuration Configured 
Encryption Status Not Encrypted 


-Physical Drive in Port 3I Bay 3
Status  OK 
Serial Number Z1E8MBYF 
Model ST2000DM001-1CH1 
Media Type HDD 
Capacity 2000 GB 
Location Port 3I Bay 3 
Firmware Version CC27 
Drive Configuration Configured 
Encryption Status Not Encrypted 


-Physical Drive in Port 4I Bay 4
Status  OK 
Serial Number Z1E8MBVG 
Model ST2000DM001-1CH1 
Media Type HDD 
Capacity 2000 GB 
Location Port 4I Bay 4 
Firmware Version CC27 
Drive Configuration Configured 
Encryption Status Not Encrypted 



-----------------------------------------
-Physical Drive in Port 5I Bay 5
Status  OK 
Serial Number S1DHNSAF761354A 
Model Samsung 
Media Type SSD 
Capacity 500 GB 
Location Port 5I Bay 5 
Firmware Version EXT0BB6Q 
Drive Configuration Configured 
Encryption Status Not Encrypted 
      

[PiterParker]

Hi Jimi, 

Did you configure any of your drives in RAID 1? 

Are you using HPE SSA Software? I would recommend to install it and run diagnostic tool. 

I would suspect one of the drive could have bad sectors or are going to predictive failed. 

If you would have SSA installed you would be clearly able to see it. 

Let me know if it helps. 

[JimiT]

Hi PiterPiker,

Thanks for the response - I do have HP SSA Software but it fails to load in windows, just locks up and need to kill it from task manager.  I've run the SSA software from post, diagnostic reports come back all clean, disks look good and controller faultless!

After boot, seems the server is now cactus! It still boots into windows, but once I'm finally logged in, it locks up too frequently becoming frustratingly unusable. 

Are there any other offline diagnostic tools I can run from in ilo perhaps or other to do a thorough test on the controller before i decide whether to buy a new disk or controller?

Jim

[JimiT]

Interestingly, when I use the HP SSA manager from post, I can view the Logical and Physical Drives in each array, the diag reports btw are all good!

However, in Bay1, (the apparent problem drive) transfer speed is set to 3Gbps, as for Bay2 it is set to 6Gbps, and Bay3 and 4 are 3Gbps. Bay5 (SSD drive in the DVDRom port) is 3Gpbs.

I don't understand why Bay1 drive is 3Gbps??? I beleive Bay3 and 4 are 3Gbps, but I believe Bay1 & 2 should be 6Gbps.

I've attaced screenshots,..

[JimiT · Accepted solution]

I just verified the 1TB drive is SATA2, so this is why only 3Gbps - I think I'll get a new SATA3 drive and replace it to see how it goes.

Jim