Getting disk admitted error when replace failed disk in 3par storage.

[DSinghp]

Hi Team,

Getting below error when I am trying to replace failed disk in 3Par.PFB,

DC_S4_6U_3PAR_8400 cli% serivrvicemag resure me 0 23
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 0 23
... onlooping mag 0 23
... firmware is current on pd WWN [5001173100ADF17C]
... firmware is current on pd WWN [50011731009084BC] Id [23]
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 0 23
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 0 23
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 0 23
... checking for valid disks...
Failed --
disk WWN [5001173100ADF17C] not admitted
Failed --
disk WWN [5001173100ADF17C] is not normal. Please use showpd -s to see details of disk state
servicemag resume 0 23 -- Failed
Command failed

I have also ran admithw/admitpd wwn of PD but still it not work. PFB,

DC_S4_6U_3PAR_8400 cli% admintpd 5001173100ADF17C
Checking for drive table upgrade packages.
Package check completed
admitted 0 out of 1 disks

We have 4 node cluster and node 0 is failed state, can anyone confirm due node 0 failed we are getting admitted failed error???

DC_S4_6U_3PAR_8400 cli% shownode
Control Data Cache
Node ----Name---- -State-- Master InCluster -Service_LED ---LED--- Mem(MB) Mem(MB) Available(%)
0 Failed No No Unknown Unknown 0 0 0
1 SGH615W45T-1 Degraded Yes Yes Off AmberBlnk 16384 16384 100
2 SGH615W45T-2 OK No Yes Off GreenBlnk 16384 16384 100
3 SGH615W45T-3 OK No Yes Off GreenBlnk 16384 16384 100

[StorageMike]

Hi

What version of Inform are you running? Also what mode disk is it (showpd -i)

[DSinghp]

@StorageMike 

DC_S4_6U_3PAR_8400 cli% showversion
Release version 3.3.1 (MU2)
Patches: P32,P40,P45,P51,P52,P60,P76,P78,P80,P90,P96,P103

Component Name Version
CLI Server 3.3.1 (P96)
CLI Client 3.3.1
System Manager 3.3.1 (P96)
Kernel 3.3.1 (MU2)
TPD Kernel Code 3.3.1 (MU2)
TPD Kernel Patch 3.3.1 (P103)

Eariler output 

DC_S4_6U_3PAR_8400 cli% showpd -i
Id CagePos State ----Node_WWN---- --MFR-- -----Model------ -Serial- -FW_Rev- Protocol MediaType -----AdmissionTime-----
--- 1:7:0 degraded 5000CCA02D1682FB HGST HCBF1200S5xeN010 06GDBTSA 3P05 SAS Magnetic -----------------------
0 0:0:0 degraded 5000CCA02D3B03F7 HGST HCBF1200S5xeN010 06H1G4MA 3P05 SAS Magnetic 2016-05-20 18:01:59 IST
1 0:1:0 degraded 5000CCA02D3A9CEF HGST HCBF1200S5xeN010 06H178XA 3P05 SAS Magnetic 2016-05-20 18:01:59 IST
2 5:2:0 normal 5000C500B8A76A78 SEAGATE STHB1200S5xeN010 W402D491 3P03 SAS Magnetic 2018-12-26 18:07:24 IST
3 0:3:0 degraded 5000CCA02D3B0A27 HGST HCBF1200S5xeN010 06H1GKDA 3P05 SAS Magnetic 2016-05-20 18:02:00 IST
4 0:4:0 degraded 5000CCA02D3AA0C3 HGST HCBF1200S5xeN010 06H17JUA 3P05 SAS Magnetic 2016-05-20 18:02:00 IST
5 0:5:0 degraded 5000CCA02D3AED27 HGST HCBF1200S5xeN010 06H1DMJA 3P05 SAS Magnetic 2016-05-20 18:02:00 IST
6 0:6:0 degraded 5000CCA02D3B05F7 HGST HCBF1200S5xeN010 06H1G8SA 3P05 SAS Magnetic 2016-05-20 18:02:01 IST
7 0:7:0 degraded 5000CCA02D3B225F HGST HCBF1200S5xeN010 06H1J5DA 3P05 SAS Magnetic 2016-05-20 18:02:01 IST
8 0:8:0 degraded 5000CCA02D3AEBAB HGST HCBF1200S5xeN010 06H1DJGA 3P05 SAS Magnetic 2016-05-20 18:02:02 IST
9 0:9:0 degraded 5000CCA02D3B330B HGST HCBF1200S5xeN010 06H1K8UA 3P05 SAS Magnetic 2016-05-20 18:02:02 IST
10 0:10:0 degraded 5000CCA02D3B2EF7 HGST HCBF1200S5xeN010 06H1K0DA 3P05 SAS Magnetic 2016-05-20 18:02:02 IST
11 0:11:0 degraded 5000CCA02D3B2E2B HGST HCBF1200S5xeN010 06H1JYSA 3P05 SAS Magnetic 2016-05-20 18:02:03 IST
12 0:12:0 degraded 5001173100907648 SanDisk DOPE1920S5xnNMRI A0003F54 3P0B SAS cMLC 2016-05-20 18:02:03 IST
13 0:13:0 degraded 5001173100909C0C SanDisk DOPE1920S5xnNMRI A0004925 3P0B SAS cMLC 2016-05-20 18:02:03 IST
14 0:14:0 degraded 5001173100907428 SanDisk DOPE1920S5xnNMRI A0003FFB 3P0B SAS cMLC 2016-05-20 18:02:03 IST
15 0:15:0 degraded 5001173100907658 SanDisk DOPE1920S5xnNMRI A0004067 3P0B SAS cMLC 2016-05-20 18:02:03 IST
16 0:16:0 degraded 5001173100907400 SanDisk DOPE1920S5xnNMRI A0003FD8 3P0B SAS cMLC 2016-05-20 18:02:03 IST
17 0:17:0 degraded 50011731009076AC SanDisk DOPE1920S5xnNMRI A0004003 3P0B SAS cMLC 2016-05-20 18:02:03 IST
18 0:18:0 degraded 5001173100906A38 SanDisk DOPE1920S5xnNMRI A0003D17 3P0B SAS cMLC 2016-05-20 18:02:03 IST
19 0:19:0 degraded 5001173100906A4C SanDisk DOPE1920S5xnNMRI A0003D55 3P0B SAS cMLC 2016-05-20 18:02:03 IST
20 0:20:0 degraded 5001173100909340 SanDisk DOPE1920S5xnNMRI A0004670 3P0B SAS cMLC 2016-05-20 18:02:03 IST
21 0:21:0 failed 5001173100908F6C SanDisk DOPE1920S5xnNMRI A0004529 3P0B SAS MLC 2016-05-20 18:02:03 IST
22 0:22:0 degraded 500117310090868C SanDisk DOPE1920S5xnNMRI A00043C1 3P0B SAS cMLC 2016-05-20 18:02:03 IST
23 0:23:0 failed 50011731009084BC SanDisk DOPE1920S5xnNMRI A0004358 3P0B SAS cMLC 2016-05-20 18:02:03 IST
24 1:0:0 degraded 5000CCA02D3B29B7 HGST HCBF1200S5xeN010 06H1JNKA 3P05 SAS Magnetic 2016-05-20 18:02:03 IST
25 1:1:0 degraded 5000CCA02D3B05DF HGST HCBF1200S5xeN010 06H1G8KA 3P05 SAS Magnetic 2016-05-20 18:02:04 IST
26 1:2:0 degraded 5000CCA02D3AFAFF HGST HCBF1200S5xeN010 06H1EK3A 3P05 SAS Magnetic 2016-05-20 18:02:04 IST
27 1:3:0 degraded 5000CCA02D3B0543 HGST HCBF1200S5xeN010 06H1G79A 3P05 SAS Magnetic 2016-05-20 18:02:04 IST
28 1:4:0 degraded 5000CCA02D3AF4AB HGST HCBF1200S5xeN010 06H1E41A 3P05 SAS Magnetic 2016-05-20 18:02:05 IST
29 1:5:0 degraded 5000CCA02D3B0537 HGST HCBF1200S5xeN010 06H1G76A 3P05 SAS Magnetic 2016-05-20 18:02:05 IST
30 1:6:0 degraded 5000CCA02D3B003B HGST HCBF1200S5xeN010 06H1EWXA 3P05 SAS Magnetic 2016-05-20 18:02:05 IST
31 1:7:0? failed 5000CCA02D3B2CD7 HGST HCBF1200S5xeN010 06H1JW0A 3P05 SAS Magnetic 2016-05-20 18:02:06 IST
32 1:8:0 degraded 5000CCA02D3B2EFF HGST HCBF1200S5xeN010 06H1K0GA 3P05 SAS Magnetic 2016-05-20 18:02:06 IST

After repalced drive 0 23 , i am getting below output.

DC_S4_6U_3PAR_8400 cli% showpd -status
Id CagePos Type -State-- -----------------------------------------Detailed_State------------------------------------------ -SedState--
--- 0:23:0 SSD degraded missing_A_port not_capable
--- 1:7:0 FC degraded missing_A_port not_capable
0 0:0:0 FC degraded missing_A_port not_capable
1 0:1:0 FC degraded missing_A_port not_capable
2 5:2:0 FC normal normal not_capable
3 0:3:0 FC degraded missing_A_port not_capable
4 0:4:0 FC degraded missing_A_port not_capable
5 0:5:0 FC degraded missing_A_port not_capable
6 0:6:0 FC degraded missing_A_port not_capable
7 0:7:0 FC degraded missing_A_port not_capable
8 0:8:0 FC degraded missing_A_port not_capable
9 0:9:0 FC degraded missing_A_port not_capable
10 0:10:0 FC degraded missing_A_port not_capable
11 0:11:0 FC degraded missing_A_port not_capable
12 0:12:0 SSD degraded missing_A_port not_capable
13 0:13:0 SSD degraded missing_A_port not_capable
14 0:14:0 SSD degraded missing_A_port not_capable
15 0:15:0 SSD degraded missing_A_port not_capable
16 0:16:0 SSD degraded missing_A_port not_capable
17 0:17:0 SSD degraded missing_A_port not_capable
18 0:18:0 SSD degraded missing_A_port not_capable
19 0:19:0 SSD degraded missing_A_port not_capable
20 0:20:0 SSD degraded missing_A_port not_capable
21 0:21:0 SSD failed vacated,missing_A_port,disabled_A_port,disabled_B_port,invalid_media,invalid,read_capacity_failed not_capable
22 0:22:0 SSD degraded missing_A_port not_capable
23 0:23:0? SSD degraded not_available_for_allocations,prolonged_missing unknown
24 1:0:0 FC degraded missing_A_port not_capable
25 1:1:0 FC degraded missing_A_port not_capable
26 1:2:0 FC degraded missing_A_port not_capable
27 1:3:0 FC degraded missing_A_port not_capable
28 1:4:0 FC degraded missing_A_port not_capable
29 1:5:0 FC degraded missing_A_port not_capable
30 1:6:0 FC degraded missing_A_port not_capable
31 1:7:0? FC failed vacated,missing,invalid_media,smart_threshold_exceeded unknown
32 1:8:0 FC degraded missing_A_port not_capable

As per above no id allocated both drive  0:23:0 and 1:7:0

[DSinghp]

any one have solution for this issue ?

[sbhat09]

Hello,

Please run 'admithw wwn' is not the right command.

Please run either of the below commands:

'admithw -notune' or 'admitpd <wwn>' where wwn is the 16 digit wwn of the PD.

If the issue still persists, please let me know. The error may also be due to firmware.

Regards,

Srinivas Bhat

[veeyarvi]

Hello Dsinghp,

The drives in cages 0 and 1 showing "missing_A_Port". Is node 0 down? Need to fix the hardware issues before admitting the disk

Regards,

[DSinghp · Accepted solution]

After replace node0 issue has been resolved