HP P410i “Parity Initialization Status: Initialization Failed” How to fix?

[schellr]

Hello,

Having a little problem with a Smart Array P410i in a ProLiant DL380G6.
It's RAID5 consisting of 5 disks with an additional 6th disk as spare.

One of the disks had a failure. The disks has been replaced and in hpacucli ctrl slot=0 show config everything seems to be OK, the new disk is marked as spare.

But in the output of hpacucli ctrl slot=0 ld 1 show I can see that the Parity Initialization has the status Initialization Failed.

What can I do to reinitialize parity initialization?

Thank you in advance.

ADUReport is attached (.png extension needs to be removed).
Output of hpacucli ctrl all show config detail:

Spoiler

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: Confidential Info Erased
   Cache Serial Number: 
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 2.50
   Rebuild Priority: High
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 100% Read / 0% Write
   Drive Write Cache: Disabled
   Total Cache Size: 256 MB
   Total Cache Memory Available: 144 MB
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0  MB
      Status: OK
      Array Type: Data
      Spare Type: dedicated



      Logical Drive: 1
         Size: 546.8 GB
         Fault Tolerance: 5
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 64 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Parity Initialization Status: Initialization Failed
         Unique Identifier: 600508B1001037383941424344450300
         Disk Name: /dev/cciss/c0d0
         Mount Points: / 537.0 GB
         OS Status: LOCKED
         Logical Drive Label: A012704D50123456789ABCDE2842
         Drive Type: Data

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 10000
         Firmware Revision: HPDD
         Serial Number: 
         Model: HP      EG0146FAWHU
         Current Temperature (C): 40
         Maximum Temperature (C): 53
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown

      physicaldrive 1I:1:2
         Port: 1I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 10000
         Firmware Revision: HPDD
         Serial Number: 
         Model: HP      EG0146FAWHU
         Current Temperature (C): 42
         Maximum Temperature (C): 57
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown

      physicaldrive 1I:1:3
         Port: 1I
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 10000
         Firmware Revision: HPDD
         Serial Number: 
         Model: HP      EG0146FAWHU
         Current Temperature (C): 41
         Maximum Temperature (C): 57
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown

      physicaldrive 1I:1:4
         Port: 1I
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 10000
         Firmware Revision: HPDD
         Serial Number: 
         Model: HP      EG0146FAWHU
         Current Temperature (C): 39
         Maximum Temperature (C): 54
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown

      physicaldrive 2I:1:6
         Port: 2I
         Box: 1
         Bay: 6
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 10000
         Firmware Revision: HPDD
         Serial Number: 
         Model: HP      EG0146FAWHU
         Current Temperature (C): 37
         Maximum Temperature (C): 49
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown

      physicaldrive 2I:1:5
         Port: 2I
         Box: 1
         Bay: 5
         Status: OK
         Drive Type: Spare Drive
         Interface Type: SAS
         Size: 146 GB
         Rotational Speed: 15000
         Firmware Revision: HPD9
         Serial Number:         
         Model: HP      EH0146FARWD
         Current Temperature (C): 32
         Maximum Temperature (C): 43
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown


   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250
      Device Number: 250
      Firmware Version: RevC
      WWID: 50123456789ABCED
      Vendor ID: PMCSIERA
      Model:  SRC 8x6G

[f_silva]

Hi friend

You will need check statys of the all disks, because if one HDDs have read / write errors it that can cause the parity initialization failure of the logical volume. Other thing that can cause that is the Firmware version of the Smart Array controller.

So, I recommend that you check the firmware version and if is out of date, upgrade and check all disks in the Array Diagnostic Log and to isolate the read/write error.

If you collect the log, send to me and I can check for you.

Regards

[schellr]

Thank you for the quick response.

This is the drive status information from hpacucli:

Spoiler

[root@srv01 ~]#hpacucli ctrl slot=0 pd all show status

   physicaldrive 1I:1:1 (port 1I:box 1:bay 1, 146 GB): OK
   physicaldrive 1I:1:2 (port 1I:box 1:bay 2, 146 GB): OK
   physicaldrive 1I:1:3 (port 1I:box 1:bay 3, 146 GB): OK
   physicaldrive 1I:1:4 (port 1I:box 1:bay 4, 146 GB): OK
   physicaldrive 2I:1:6 (port 2I:box 1:bay 6, 146 GB): OK
   physicaldrive 2I:1:5 (port 2I:box 1:bay 5, 146 GB, spare): OK

Since the output from hpacucli ctrl slot=0 physicaldrive all show detail is already in the spoiler section of the original post, I am guessing you are asking for further information.
These are the SMART infos of the devices:

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,0 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EG0146FAWHU
Revision:             HPDD
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000c5001d874e73
Serial number:        Confidential Info Erased
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:54:55 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     41 C
Drive Trip Temperature:        65 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 1151560196
  Blocks received from initiator = 2284803932
  Blocks read from cache and sent to initiator = 3602218600
  Number of read and write commands whose size <= segment size = 2438343158
  Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 68146,43
  number of minutes until next internal SMART test = 16

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        4         0  2222162435          0     356792,712           0
write:         0        0         0         0          0      40958,411           0
verify:        0        1         0        11          0          0,000           1

Non-medium error count:       22
No self-tests have been logged
Long (extended) Self Test duration: 1570 seconds [26,2 minutes]

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,1 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EG0146FAWHU
Revision:             HPDD
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000c5001d863d3b
Serial number:        Confidential Info Erased
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:55:27 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     43 C
Drive Trip Temperature:        65 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 4038725961
  Blocks received from initiator = 3096985576
  Blocks read from cache and sent to initiator = 1255288545
  Number of read and write commands whose size <= segment size = 2470309119
  Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 68145,90
  number of minutes until next internal SMART test = 15

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        2         0  2515466927          0     356071,917           0
write:         0        0         0         0          0      41382,885           0

Non-medium error count:       33
No self-tests have been logged
Long (extended) Self Test duration: 1570 seconds [26,2 minutes]

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,2 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EG0146FAWHU
Revision:             HPDD
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000c5001d86820f
Serial number:       
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:55:53 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     42 C
Drive Trip Temperature:        65 C
Elements in grown defect list: 3
Vendor (Seagate) cache information
  Blocks sent to initiator = 3348607910
  Blocks received from initiator = 1878218604
  Blocks read from cache and sent to initiator = 3278458544
  Number of read and write commands whose size <= segment size = 2265000248
  Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 68145,72
  number of minutes until next internal SMART test = 15

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        8         0  2264406328          0     349121,507           6
write:         0        0         0         0          0      40740,787           0

Non-medium error count:       23
No self-tests have been logged
Long (extended) Self Test duration: 1570 seconds [26,2 minutes]

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,3 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EG0146FAWHU
Revision:             HPDD
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000c5001d868207
Serial number:       
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:56:14 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     40 C
Drive Trip Temperature:        65 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 904170883
  Blocks received from initiator = 3356894921
  Blocks read from cache and sent to initiator = 160196265
  Number of read and write commands whose size <= segment size = 2126313209
  Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 67214,03
  number of minutes until next internal SMART test = 14

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0  2643149352          0     336874,839           0
write:         0        0         0         0          0      39290,481           0

Non-medium error count:        1
No self-tests have been logged
Long (extended) Self Test duration: 1570 seconds [26,2 minutes]

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,4 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EH0146FARWD
Revision:             HPD9
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000cca00b807380
Serial number:                
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:56:48 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     32 C
Drive Trip Temperature:        65 C
Manufactured in week 08 of year 2012
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  40
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 9464056620515328

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0   638539         0    638539          0      68727,372           0
write:         0  1118329         0   1118329          0      13103,049           0

Non-medium error count:      469

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   42684                 - [-   -    -]
# 2  Background short  Completed                   -       5                 - [-   -    -]
# 3  Background short  Completed                   -       1                 - [-   -    -]

Long (extended) Self Test duration: 1394 seconds [23,2 minutes]

Spoiler

[root@srv01 ~]#smartctl -a -d cciss,5 /dev/cciss/c0d0
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.18-308.1.1.v5PAE] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               HP
Product:              EG0146FAWHU
Revision:             HPDD
User Capacity:        146.815.737.856 bytes [146 GB]
Logical block size:   512 bytes
Logical Unit id:      0x5000c5001d866f13
Serial number:       
Device type:          disk
Transport protocol:   SAS
Local Time is:        Wed Jan  3 14:57:07 2018 CET
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature:     38 C
Drive Trip Temperature:        65 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 2585548455
  Blocks received from initiator = 1696680852
  Blocks read from cache and sent to initiator = 3245013025
  Number of read and write commands whose size <= segment size = 2285795565
  Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 68146,00
  number of minutes until next internal SMART test = 14

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0  338451552          0     350921,254           0
write:         0        0         0         0          0      40648,424           0

Non-medium error count:        6
No self-tests have been logged
Long (extended) Self Test duration: 1570 seconds [26,2 minutes]

Please let me know, if you were expecting a different kind of log.

As for the Firmware Update: I would prefer not having to do an update, as the system is off site.

Though if I have to do an update, I am guessing this is the right one:
https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=3883933&swItemId=MTX_5e52f965d84f41c2bb65d33b58&swEnvOid=4184#tab-history

That would be version 6.64 (B)(1 Oct 2015). Currently installed is version 2.50

[root@srv01 ~]#hpacucli ctrl slot=0 show config detail
[...]
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 2.50
[...]

[DGOLAN]

Did you get a response back on this?  I am experiencing the same issues.  Drives look good but parity initialization failed on Array 1.