3PAR disk replacement failed for multiple disk

[Mohamad_elmasri]

Hello Expert,

I have 3par out of support with 5 disk failure, we replace the first disk 1:6 and the replacment went fine without any issue, before the reallocation finish we replace disk 5:8 disk, disk 5:8 show failed and after 1 day 1:6 also failed.

AEADDCSAN01 cli% showpd -failed
--Size(MB)-- ----Ports----
Id CagePos Type RPM State Total Free A B Capacity(GB)
24 1:6:0? FC 10 failed 838656 0 ----- ----- 900
71 1:13:0 FC 10 failed 838656 0 ----- ----- 900
72 5:8:0? FC 10 failed 838656 0 ----- ----- 900
76 5:12:0 FC 10 failed 838656 0 ----- ----- 900
85 5:8:0 FC 10 failed 838656 0 1:0:2* 0:0:2 900
------------------------------------------------------------------
5 total 4193280 0

AEADDCSAN01 cli% movepdtospare -f -vacate -nowait 72
There are no chunklets to move.

AEADDCSAN01 cli% movepdtospare -f -vacate -nowait 85
There are no chunklets to move.

then run 

Cage 5, magazine 8:
A servicemag resume command failed on this magazine.
The command completed at Fri Nov 1 09:39:03 2024.
The output of the servicemag resume was:
servicemag resume 5 8
... mag 5 8 already onlooped
... firmware is current on pd WWN [5000CCA0719B20DF] Id [72]
... firmware is current on pd WWN [5000CCA0164C7DE3] Id [85]
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 5 8
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 5 8
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 5 8
... checking for valid disks...
Failed --
disk WWN [5000CCA0164C7DE3] Id [85] is not normal. Please use showpd -s to see details of disk state
servicemag resume 5 8 -- Failed

the same error for both 1:6 and 5:8 please advice and approciate your support as usual

[support_s]

System recommended content:

1. HPE 3PAR StoreServ Alerts Reference: Customer Edition (HPE 3PAR OS 3.3.1 Technology Release T05)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Ramesh_Kumar_P]

@Mohamad_elmasri,

I have sent you a private note. Please check and share the details.

Thank you

[Mohamad_elmasri]

I did already. Please check. Appreciate your support

[Ramesh_Kumar_P]

Hi Mohamad_elmasri,

I have shared the FTP details over private message. You can please share the log and keep me posted.

Thank you!

[Ramesh_Kumar_P]

@Mohamad_elmasri,

Thank you for sharing the log. I have reviewed them and here are the details.

- The PD 1:6:0(PDID24) were failed and replaced. It has taken new PDID 0
- The new PD also marked failed due to bad media chunklets

➤ cat showpd_-c.out | grep -i 1:6:0
 0 1:6:0   FC   failed   819     0    0    0      0     738   81    0    0    0      0    0
24 1:6:0?  FC   failed   819     0    0    0    701       0  118    0    0    0      0    0
                                                                                                                                               ✔
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

- This PD failed due to excessive smart errors on the drive.

➤ cat showpd_-c.out | grep -i 1:13:0
71 1:13:0  FC   failed   819     0    0    0      0     763   51    0    0    0      0    5

- The PD 5:8:0(PDID 72) were failed and replaced. It has taken new PDID 85
- The new PD also marked failed due to bad media chunklets. Also this has path errors too.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➤ cat showpd_-c.out | grep -i 5:8:0
72 5:8:0?  FC   failed   819     0    0    0    526       0  293    0    0    0      0    0
85 5:8:0   FC   failed   819     0    0    0      0     794   25    0    0    0      0    0
                                                                                                                                               
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
- This PD detected as slow drive. This needs a replacement as it may impact the array performance while read / write IOs on this drive.

➤ cat showpd_-c.out | grep -i 5:12:0
76 5:12:0  FC   failed   819     0    0    0      0     629  190    0    0    0      0    0

Expect the PDID 76, all the above drive is a genuine hardware failure.
This needs a replacement to fix the issue.

[Ramesh_Kumar_P]

@Mohamad_elmasri, Hope you got the required information. I will be closing this case now. Please feel free to open new case if further help needed.

Thank you!