DL Series Server, P822 raid and D3600 disk shelves... disk replacement

[adamdb_uk]

Hi,

    we have an ageing G8 DL Series server with a couple of D3600 shelves attached (populated with MB4000FCWDK 4TB SAS drives). The storage is configured as a 72TB raid 50 volume which is presented to windows (drives 1-11 on first D3600 shelf are parity group 1 with drive 12 as the spare and drives 1-11 on second D3600 shelf are parity group 2 with the 12th as the spare).

Drive frmware varies between HPD2. and HPD7. Controller firmware is 6.34 (yes its old). Recently we began to see a lot of drive failures. These have been replaced one at time and so far no issues. However I now have a problem where I had a proper failure in bay 9 on one D3600 and two predictive failures on bays 5 and 7 on the other shelf/parity group. Swapping the failed drive in bay 9 triggered a parity rebuild for that parity group but when i attempted to replace one of the 'predictive failures' in the other parity group  i got the error 'incorrect drive replaced' and the logical volume would not start. Looks like the firmware on the latest drive was HPDB which I am guessing is pretty recent.

Is it likely i need to update the controller firmware and drive firmware or is this error down to how you replace a predictive failure? I was under the impression you can replace up to 2 drives at a time so long as they are in different parity groups which mine are...

If the error above is likely related to firmware revisions I'll plan to update the controller f/w and the disks. Assume it's controller first then disks... I believe the packages for this would be 
Controller: https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_4851cf9c22964a3bb3a4704bad&tab=Compatibility
Disks : https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_b7209ba80b4c4f46b464681a39&tab=Installation+Instructions

One thing of note is the f/w revision listed for the Smart update is HPDA (which it appears has since been superceded by HPDB as I can see from the drive I was supplied with....) is there a later version of this Smart update tool which contains HPDB or later firmware? I can't seem to find it!

Below is the current status (with the parity in recovery back to 2E:2:9)

=> ctrl slot=2 ld 2 show detail

Smart Array P822 in Slot 2

array B

Logical Drive: 2
Size: 72.8 TB
Fault Tolerance: 50
Number of Parity Groups: 2
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 2560 KB
Status: OK
MultiDomain Status: OK
Caching: Enabled
Parity Initialization Status: In Progress
Parity Initialization Progress: 85% complete
Unique Identifier: 600508B1001C473A86D0EEBE234ADE26
Disk Name: \\.\PhysicalDrive1 (Disk 1) (Bus: 0,Target: 1,Lun: 1)
Mount Points: S:\ 72.8 TB Partition Number 3
Logical Drive Label: 056AC517PDVTF0BRH6Y54R6541
Parity Group 1:
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SAS, 4000.7 GB, OK)
Parity Group 2:
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SAS, 4000.7 GB, Predictive Failure)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SAS, 4000.7 GB, Predictive Failure)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SAS, 4000.7 GB, OK)
Drive Type: Data
LD Acceleration Method: Controller Cache

but if anyone can confirm my thoughts it would be appreciated.
thanks.

Adam.

[BPSingh]

Greetings!

For the 'incorrect drive replaced' alert. 

1. Please check the drive type and the capacity that you are attempting to replace. 
2. Also please ensure that the drive is a compatible drive. 
https://www.hpe.com/psnow/doc/c04227611.html?jumpid=in_pdp-psnow-qs 

3. Do you get the same error on both bay 5 and bay 7?

Regarding the firmware, 8.32 is the latest firmware revision available for P822. Since its currently on 6.34, it is recommended to perform an incremental upgrade. 

https://support.hpe.com/connect/s/softwaredetails?softwareId=MTX_dc0fa2598792465d8d5eaab1e2&language=en_US&tab=revisionHistory 

One thing of note is the f/w revision listed for the Smart update is HPDA (which it appears has since been superceded by HPDB as I can see from the drive I was supplied with....) is there a later version of this Smart update tool which contains HPDB or later firmware?---> We need to check with the model number of the drive? Every drive model will have different firmware revision. 

Disclaimer : Please note data integrity is customer responsibility and HPE recommends to have data backup frequently and before performing any activity on the server so that it can be restored during disaster recovery.   

[adamdb_uk]

Response to "1. Please check the drive type and the capacity that you are attempting to replace. " - to appears to show the drive replacement as genuine HPE (which it should be) but the drive firmware appears to be very recent - it IS showing the same model number as the other drives in the chassis. (this drive was subsequently removed and the old drive put in the slot as the volume would not come online).

physicaldrive 2E:1:5
Port: 2E
Box: 1
Bay: 5
Active Path:
Port: 1E Box: 1 Bay: 5
Redundant Path(s):
Port: 2E Box: 2 Bay: 5
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPDB
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 30
Maximum Temperature (C): 47
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

2. Also please ensure that the drive is a compatible drive.  - it's the same model as all the others.

3. 3. Do you get the same error on both bay 5 and bay 7?- This is not been checked so far. I have been waiting for the parity rebuild to complete on the other parity group and will then revisit the predictive failures again.

In response to the question " We need to check with the model number of the drive? Every drive model will have different firmware revision. " -  - please see the PD output above.

Can you confirm I will be ok to update from 6.34 -> 8.32 for the P822 or does I need to go through interim versions?

Also assume the smart update tool will apply this update to any other controllers in the chassis (of which there are two others).

Is it also worth apply the drive firmware package?

I will askour onsite DC teams to send me photos of the replacement drive labels which may yield more information.

Also it appears that overnight while the partity rebuild was happening in parity group 1 (which had a REAL failure at 2E:2:9 previously) which was due to finish last night, one of the predictive failures (partity group 2E:1:5) has actuallty died now so the system has now begin rebuild onto the spare. Seems crazy logic that it builds rebuilds parity onto the spare but when you put a drive in to fix the failure to rebuilds parity BACK to the replacement drive! Why? Surely you should just mark the replacement as the new spare. Bad design. Oh well.

So this was the status last night.

=> ctrl slot=2 ld 2 show detail

Smart Array P822 in Slot 2

array B

Logical Drive: 2
Size: 72.8 TB
Fault Tolerance: 50
Number of Parity Groups: 2
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 2560 KB
Status: OK
MultiDomain Status: OK
Caching: Enabled
Parity Initialization Status: In Progress
Parity Initialization Progress: 88% complete
Unique Identifier: 600508B1001C473A86D0EEBE234ADE26
Disk Name: \\.\PhysicalDrive1 (Disk 1) (Bus: 0,Target: 1,Lun: 1)
Mount Points: S:\ 72.8 TB Partition Number 3
Logical Drive Label: 056AC517PDVTF0BRH6Y54R6541
Parity Group 1:
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SAS, 4000.7 GB, OK)
Parity Group 2:
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SAS, 4000.7 GB, Predictive Failure)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SAS, 4000.7 GB, Predictive Failure)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SAS, 4000.7 GB, OK)
Drive Type: Data
LD Acceleration Method: Controller Cache

=> ctrl slot=2 ld 2 show detail

AND THIS IS THE STATUS THIS MORNING

Smart Array P822 in Slot 2

array B

Logical Drive: 2
Size: 72.8 TB
Fault Tolerance: 50
Number of Parity Groups: 2
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 2560 KB
Status: OK
MultiDomain Status: OK
Caching: Enabled
Parity Initialization Status: In Progress
Parity Initialization Progress: 5% complete
Unique Identifier: 600508B1001C473A86D0EEBE234ADE26
Disk Name: \\.\PhysicalDrive1 (Disk 1) (Bus: 0,Target: 1,Lun: 1)
Mount Points: S:\ 72.8 TB Partition Number 3
Logical Drive Label: 056AC517PDVTF0BRH6Y54R6541
Parity Group 1:
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SAS, 4000.7 GB, OK)
Parity Group 2:
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SAS, 4000.7 GB, Failed)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SAS, 4000.7 GB, Predictive Failure)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SAS, 4000.7 GB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SAS, 4000.7 GB, OK)
Drive Type: Data
LD Acceleration Method: Controller Cache

=>

So its now doing yet ANOTHER parity rebuild onto the spare for parity group 2 which is in bay 12. I assume the spares are tied to the parity group? If the other drive fails during the parity rebuild for the failure at 2E:1:5 I guess we'll lose the data (unless spares are valid for the whole thing?).

So I guess I will not have to wait for this to complete (usually about 40 hours).

This thing certainly seems to have hit the wall for MTBF on the drives. Just hope we can resolve the issue around why it does not like the newly shipped ones.

Here's a final output showing all the PDs for ARRAY B. I've removed array A as this is just some internal SSDs used for the server. I can also provide you with ADU reports if needed just need your details. thanks

=> ctrl slot=2 pd all show detail

Smart Array P822 in Slot 2

array B

physicaldrive 2E:1:1
Port: 2E
Box: 1
Bay: 1
Active Path:
Port: 1E Box: 1 Bay: 1
Redundant Path(s):
Port: 2E Box: 2 Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD9
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 55
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:2
Port: 2E
Box: 1
Bay: 2
Active Path:
Port: 2E Box: 2 Bay: 2
Redundant Path(s):
Port: 1E Box: 1 Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPDB
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 46
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:3
Port: 2E
Box: 1
Bay: 3
Active Path:
Port: 1E Box: 1 Bay: 3
Redundant Path(s):
Port: 2E Box: 2 Bay: 3
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPDA
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 44
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:4
Port: 2E
Box: 1
Bay: 4
Active Path:
Port: 2E Box: 2 Bay: 4
Redundant Path(s):
Port: 1E Box: 1 Bay: 4
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 38
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:5
Port: 2E
Box: 1
Bay: 5
Active Path:
Port: 1E Box: 1 Bay: 5
Redundant Path(s):
Port: 2E Box: 2 Bay: 5
Status: Failed
Last Failure Reason: Mark bad failed
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Maximum Temperature (C): 39
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: Not Applicable

physicaldrive 2E:1:6
Port: 2E
Box: 1
Bay: 6
Active Path:
Port: 2E Box: 2 Bay: 6
Redundant Path(s):
Port: 1E Box: 1 Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 39
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:7
Port: 2E
Box: 1
Bay: 7
Active Path:
Port: 1E Box: 1 Bay: 7
Redundant Path(s):
Port: 2E Box: 2 Bay: 7
Status: Predictive Failure
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:8
Port: 2E
Box: 1
Bay: 8
Active Path:
Port: 2E Box: 2 Bay: 8
Redundant Path(s):
Port: 1E Box: 1 Bay: 8
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:9
Port: 2E
Box: 1
Bay: 9
Active Path:
Port: 1E Box: 1 Bay: 9
Redundant Path(s):
Port: 2E Box: 2 Bay: 9
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 38
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:10
Port: 2E
Box: 1
Bay: 10
Active Path:
Port: 2E Box: 2 Bay: 10
Redundant Path(s):
Port: 1E Box: 1 Bay: 10
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 36
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:11
Port: 2E
Box: 1
Bay: 11
Active Path:
Port: 1E Box: 1 Bay: 11
Redundant Path(s):
Port: 2E Box: 2 Bay: 11
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:1
Port: 2E
Box: 2
Bay: 1
Active Path:
Port: 2E Box: 1 Bay: 1
Redundant Path(s):
Port: 1E Box: 2 Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 38
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:2
Port: 2E
Box: 2
Bay: 2
Active Path:
Port: 1E Box: 2 Bay: 2
Redundant Path(s):
Port: 2E Box: 1 Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:3
Port: 2E
Box: 2
Bay: 3
Active Path:
Port: 2E Box: 1 Bay: 3
Redundant Path(s):
Port: 1E Box: 2 Bay: 3
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 38
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:4
Port: 2E
Box: 2
Bay: 4
Active Path:
Port: 1E Box: 2 Bay: 4
Redundant Path(s):
Port: 2E Box: 1 Bay: 4
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 32
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:5
Port: 2E
Box: 2
Bay: 5
Active Path:
Port: 2E Box: 1 Bay: 5
Redundant Path(s):
Port: 1E Box: 2 Bay: 5
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 38
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:6
Port: 2E
Box: 2
Bay: 6
Active Path:
Port: 1E Box: 2 Bay: 6
Redundant Path(s):
Port: 2E Box: 1 Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD7
Serial Number: **removed**
Model: HP MB4000FCZGL
Current Temperature (C): 29
Maximum Temperature (C): 49
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:7
Port: 2E
Box: 2
Bay: 7
Active Path:
Port: 2E Box: 1 Bay: 7
Redundant Path(s):
Port: 1E Box: 2 Bay: 7
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:8
Port: 2E
Box: 2
Bay: 8
Active Path:
Port: 1E Box: 2 Bay: 8
Redundant Path(s):
Port: 2E Box: 1 Bay: 8
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 30
Maximum Temperature (C): 39
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:9
Port: 2E
Box: 2
Bay: 9
Active Path:
Port: 2E Box: 1 Bay: 9
Redundant Path(s):
Port: 1E Box: 2 Bay: 9
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 51
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:10
Port: 2E
Box: 2
Bay: 10
Active Path:
Port: 1E Box: 2 Bay: 10
Redundant Path(s):
Port: 2E Box: 1 Bay: 10
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 30
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:11
Port: 2E
Box: 2
Bay: 11
Active Path:
Port: 2E Box: 1 Bay: 11
Redundant Path(s):
Port: 1E Box: 2 Bay: 11
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD2
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 30
Maximum Temperature (C): 47
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:1:12
Port: 2E
Box: 1
Bay: 12
Active Path:
Port: 2E Box: 2 Bay: 12
Redundant Path(s):
Port: 1E Box: 1 Bay: 12
Status: OK
Drive Type: Spare Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 31
Maximum Temperature (C): 35
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

physicaldrive 2E:2:12
Port: 2E
Box: 2
Bay: 12
Active Path:
Port: 1E Box: 2 Bay: 12
Redundant Path(s):
Port: 2E Box: 1 Bay: 12
Status: OK
Drive Type: Spare Drive
Interface Type: SAS
Size: 4000.7 GB
Native Block Size: 512
Rotational Speed: 7200
Firmware Revision: HPD5
Serial Number: **removed**
Model: HP MB4000FCWDK
Current Temperature (C): 26
Maximum Temperature (C): 37
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

=>

Thanks.

Moderator Edit: Confedential information removed. 

[BPSingh]

Greetings!

HPDB for MB4000FCWDK seems to be factory installed, online we see only HPDA as the latest firmware revision. 

Can you confirm I will be ok to update from 6.34 -> 8.32 for the P822 or does I need to go through interim versions?

Please do incremental update 6.34-> 7.02 -> 8.00-> 8.32

Also assume the smart update tool will apply this update to any other controllers in the chassis (of which there are two others). 

Yes, if the other controllers are any of these models "Smart Array P220i, P222, P420i, P420, P421, P721m, and P822" as mentioned on the firmware page. 

Is it also worth apply the drive firmware package?

Are you referring to the controller driver? 

Further you may also test with same specification but other drive model to narrow down. Since the slot works with the original driives, could be something to do with the new drive.

[adamdb_uk]

Thanks for the upgrade path information for the raid controllers. I wil raise some paperwork to get that scheduled in. Am I ok to perform the firmware updates during the partiy rebuild onto the spare or should I wait until this is completed? Also when applying the updates to via smart update does the server require a reboot after each update or can they all be applied one after the other and then a reboot? thanks.

Is it also worth apply the drive firmware package?

Are you referring to the controller driver?  - NO - There is a drive firmware update package for this model of drive to bring them all up to HPDA. Is it worth installing this as well to make sure all the drives are on the same firmware?

I suspect it's not worth doing much more until the raid controllers have been updated as the issue with the HPDB drive may be resolved, If not I guess I'll have to send them back and request drives with older firmware.

Are there any equivalent drive models I could request if they cannot supply this one with older firmware?

thanks.

[BPSingh]

Ideally we can, however I would advise for the parity initialization to complete before upgrading the firmware.. 

Yes, kindly reboot the server after each firmware upgrade for the changes to take place. 

Its recommended that we keep drive firmware on the latest revision as new and interim versions have various fixes, if you refer the "Revision History" on the download page. - this is not related to the issue reported but as a best practice. 

Compatible drives with same specification are listed here: https://www.hpe.com/psnow/doc/c04227611.html?jumpid=in_pdp-psnow-qs however we won't have control over the firmware versions the drives are shipped with. I would advise to discuss this with the sales team. 

[adamdb_uk]

@BPSingh 

Hi,

        Just to advise I've updated the RAID controllers to 8.32 now (via the update path you recommended). The drive which had HPDB firmware is now being accepted as valid and the array is currently undergoing yet another parity rebuild.

I was going to attempt the drive firwmare update next but it does not seem to be working. THis is to perform the update on the  MB4000FCWDK drives using the smart update tool listed in my original post to bring all the  other drives up to HPDA firmware. However when I run the tool it fails. Executing cpqsetup.exe as 'Administrator' under Windows 2012R2 just gives an error "HPED Setup for Windows has Stopped working". Reason - APPCRASH.

Does this tool need to be run with non-admin rights? Or run in compatability mode? Any advice?

thanks

[BPSingh]

Greetings!

Glad to see that original issue has been resolved.

For the drive firmware, please try incremental update instead of directly going to HPDA. If that too fails, you may attempt an offline upgrade by booting to Service Pack for ProLiant.

I have checked the contents for SPP_Gen8.1 and can see the HPDA firmware listed MB4000FCWDK.

Please refer below web-link to view the information.

https://downloads.hpe.com/pub/softlib2/software1/publishable-catalog/p615636943/v138588/SPP_Gen8.1_Content_Notes.pdf

You can boot to this SPP ISO image in interactive mode and update only the drive firmware.

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00026884en_us#N100E6

Disclaimer : Please note data integrity is customer responsibility and HPE recommends to have data backup frequently and before performing any activity on the server so that it can be restored during disaster recovery.