Servicemag resume failed

[PCD]

We replace a failed hard disk on our 3PAR Inserv E200 SAN.

We initiated the command "servicemag start -pdid 34 and the command was successful.

Next we physically replace the new hard drive.

Then we inititated the command "servicemag resume 0 9"  but this command failed and we got the following error:

CLD1-3PAR-InServ-01 cli% servicemag status -d
Cage 0, magazine 9:
A servicemag resume command failed on this magazine.
The command completed at Mon Jul 15 09:02:12 2013.
The output of the servicemag resume was:
servicemag resume 0 9
... mag 0 9 already onlooped
... firmware is current on pd WWN [2210000A33008420] Id [34]
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
Failed --
disk WWN [2210000A33008420] Id [34] is not in valid state
servicemag resume 0 9 -- Failed

We try to search for this error but cannot find a solution for it. Please advise.

Also, we want to cancel this process  on cage 0 and magazine 9 and restart the disk replacement process again. And would like your advise if this command:

    "servicemag resume unmark -cg 0 -mg 9"  would do the job?

Thanks in advance for your comments.

[Torsten.]

Did you try to re-seat the disk?

Is it connected to both loops (showpd)?

[Dennis Handly]

What do these commands show?

showversion

servicemag status -d

showpd 34

showpd -i 34

showpd -s 34

If ID 34 doesn't exist, then replace by: -p -cg 0 -mg 9

showcage -d cage0

>firmware is current on pd WWN [2210000A33008420] Id [34]

The OUI for this WWN is Emulex.

[PCD]

Hi,

Thank you for your quick respond.

I re-seated the disk but it still show the disk as failed (prior to reseat it said not ready)

Below is the showpd command after I re-seated the disk (ID 34 is the bad disk):

login as: 3paradm
Using keyboard-interactive authentication.
Password:
CLD1-3PAR-InServ-01 cli% showpd
ID CagePos   SizeMB  Chunk  Free Spare ----Node_WWN----  State  APort  BPort LdA
 0  0:0:0    380928   1488   377    93 20000024B601C8B9  valid 0:0:1* 1:0:1    Y
 1  0:3:0    380928   1488   375    93 20000024B601F561  valid 0:0:1  1:0:1*   Y
 2  0:4:0    380928   1488   375    93 20000024B601CEDA  valid 0:0:1  1:0:1*   Y
 3  0:7:0    380928   1488   376    93 20000024B601F569  valid 0:0:1* 1:0:1    Y
 4  0:8:0    380928   1488   377    93 20000024B601CE20  valid 0:0:1* 1:0:1    Y
 5 0:11:0    380928   1488   376    93 20000024B601F51D  valid 0:0:1  1:0:1*   Y
 6 0:12:0    380928   1488   376    93 20000024B601CA73  valid 0:0:1  1:0:1*   Y
 7 0:15:0    380928   1488   376    93 20000024B601F558  valid 0:0:1* 1:0:1    Y
 8  1:0:0    380928   1488   377    93 2000001862583D04  valid 0:0:2* 1:0:2    Y
 9  1:3:0    380928   1488   376    93 20000024B60138E1  valid 0:0:2  1:0:2*   Y
10  1:4:0    380928   1488   375    93 2000001862583B77  valid 0:0:2  1:0:2*   Y
11  1:7:0    380928   1488   376    93 20000024B6012FFE  valid 0:0:2* 1:0:2    Y
12  1:8:0    380928   1488   377    93 20000024B6014239  valid 0:0:2* 1:0:2    Y
13 1:11:0    380928   1488   376    93 20000024B6014204  valid 0:0:2  1:0:2*   Y
14 1:12:0    380928   1488   376    93 2000001862583CC4  valid 0:0:2  1:0:2*   Y
15 1:15:0    380928   1488   377    93 20000024B601401E  valid 0:0:2* 1:0:2    Y
16  2:0:0    380928   1488   377    93 20000024B60EF3E4  valid 0:3:1* 1:3:1    Y
17  2:3:0    380928   1488   376    93 20000024B601F539  valid 0:3:1  1:3:1*   Y
18  2:4:0    380928   1488   375    93 20000024B60EF3BB  valid 0:3:1  1:3:1*   Y
19  2:7:0    380928   1488   376    93 20000024B601F567  valid 0:3:1* 1:3:1    Y
20  2:8:0    380928   1488   377    93 20000024B60EF409  valid 0:3:1* 1:3:1    Y
21 2:11:0    380928   1488   376    93 20000024B601F551  valid 0:3:1  1:3:1*   Y
22 2:12:0    380928   1488   376    93 20000024B60EF351  valid 0:3:1  1:3:1*   Y
23 2:15:0    380928   1488   377    93 20000024B601F527  valid 0:3:1* 1:3:1    Y
24  3:0:0    380928   1488   378    93 20000024B601394B  valid 0:3:2* 1:3:2    Y
25  3:3:0    380928   1488   376    93 20000024B601CABA  valid 0:3:2  1:3:2*   Y
26  3:4:0    380928   1488   376    93 20000024B60134C2  valid 0:3:2  1:3:2*   Y
27  3:7:0    380928   1488   376    93 20000024B601CA28  valid 0:3:2* 1:3:2    Y
28  3:8:0    380928   1488   377    93 20000024B6013AA9  valid 0:3:2* 1:3:2    Y
29 3:11:0    380928   1488   376    93 20000024B601CA41  valid 0:3:2  1:3:2*   Y
30 3:12:0    380928   1488   376    93 20000024B60141DC  valid 0:3:2  1:3:2*   Y
31 3:15:0    380928   1488   377    93 20000024B601CE2A  valid 0:3:2* 1:3:2    Y
33 1:13:0    953344   3724     5   280 2210000A3300C3CA  valid 0:0:2* 1:0:2    Y
34  0:9:0    953344   3724   225   280 2210000A33008420 failed 0:0:1- 1:0:1-   N
35 0:13:0    953344   3724     5   280 2210000A330083A5  valid 0:0:1* 1:0:1    Y
36  1:1:0    953344   3724    14   280 2210000A33008F33  valid 0:0:2  1:0:2*   Y
37  1:5:0    953344   3724     5   280 2210000A3300837A  valid 0:0:2* 1:0:2    Y
38  1:9:0    953344   3724    14   280 2210000A33008449  valid 0:0:2  1:0:2*   Y
39  0:1:0    953344   3724    14   280 2210000A3300BCBD  valid 0:0:1  1:0:1*   Y
40  2:1:0    953344   3724   705   279 2210000A3300C3D6  valid 0:3:1  1:3:1*   Y
41  2:5:0    953344   3724  1112   279 2210000A3300C3B7  valid 0:3:1* 1:3:1    Y
42  2:9:0    953344   3724   706   279 2210000A3300C519  valid 0:3:1  1:3:1*   Y
43 2:13:0    953344   3724  1113   279 2210000A3300C3D5  valid 0:3:1* 1:3:1    Y
44  0:5:0    953344   3724     5   280 2210000A3300C198  valid 0:0:1* 1:0:1    Y
45  3:5:0    953344   3724  1112   279 2210000A3300C500  valid 0:3:2* 1:3:2    Y
46  3:9:0    953344   3724   705   279 2210000A3300C542  valid 0:3:2  1:3:2*   Y
47 3:13:0    953344   3724  1113   279 2210000A3300C4D0  valid 0:3:2* 1:3:2    Y
48  3:1:0    953344   3724   706   279 2210000A3300C7EC  valid 0:3:2  1:3:2*   Y
--------------------------------------------------------------------------------
48         27443200 107200 19599  7448
CLD1-3PAR-InServ-01 cli%

[Torsten.]

Was this disk used before?

I would expect a new disk ID at the same position 0:9:0 but there is none?

The "new" disk is inserted now?

[PCD]

Hi,

Thank you for your prompt respond.

Below is the command info of Showversion, servicemag status -d, showpd 34 and showcage -d cage0

I ran the command below after I re-seated the disk and try to do a servicemag start -pdid 34 :

showversion:

CLD1-3PAR-InServ-01 cli% showversion Release version 2.2.4 (MU4) Patches:  None

Component Name                   Version CLI Server                       2.2.4 (MU4) CLI Client                       2.2.4 (MU4) GUI Server                       2.2.4 (MU4) System Manager                   2.2.4 (MU4) Kernel                           2.2.4 (MU4) TPD Kernel Code                  2.2.4 (MU4) CLD1-3PAR-InServ-01 cli%

CLD1-3PAR-InServ-01 cli% servicemag status -d
Cage 0, magazine 9:
A servicemag start command failed on this magazine.
The command completed at Wed Jul 17 06:36:14 2013.
The output of the servicemag start was:
Failed --
Unable to run servicemag command Start at the moment, a start/resume may already ongoing.
servicemag start -pdid 34 -- Failed
CLD1-3PAR-InServ-01 cli%

CLD1-3PAR-InServ-01 cli% showpd 34
ID CagePos SizeMB Chunk Free Spare ----Node_WWN----  State  APort  BPort LdA
34  0:9:0  953344  3724  225   280 2210000A33008420 failed 0:0:1- 1:0:1-   N
----------------------------------------------------------------------------
 1         953344  3724  225   280
CLD1-3PAR-InServ-01 cli%

CLD1-3PAR-InServ-01 cli% showcage -d cage0 Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side  0 cage0 0:0:1     0 1:0:1     0     12 24-30   08   08   DC3  n/a

-----------Cage detail info for cage0 ---------

Position: Cabinet00 Bay7 -----------Midplane Info------------  VendorId,ProductId     3PARdata,DC3          Serial_Num  OPS69907C015F9A            Node_WWN 20000050CC015F9A    TempSensor_State               OK    TempSensor_Value               29      OpsPanel_State               OK Audible_Alarm_State            Muted           ID_Switch                1          Cage_State               OK

Interface Board Info   LoopA   LoopB      Firmware_status Current Current          Product_Rev      08      08            IFC_State      OK      OK            ESH_State      OK      OK           Master_CPU     Yes      No             Loop_Map   valid   valid           Link_Speed   2Gbps   2Gbps          Port0_State      OK      OK          Port1_State  No_SFP  No_SFP          Port2_State  No_SFP  No_SFP          Port3_State  No_SFP  No_SFP

Power Supply Info State Fan State AC Model               ps0    OK  MedSpeed OK    --               ps1    OK  MedSpeed OK    --

--------------Drive Info--------------- ----LoopA----- ----LoopB----- Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState   0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK   1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK   3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK   4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK   5:0 2210000a3300c198   Normal      30 0xd6        OK 0xd6        OK   7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK   8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK   9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK  11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK  12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK  13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK  15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK CLD1-3PAR-InServ-01 cli%

[PCD]

Yes, the new disk is inserted.

[Torsten.]

Did you use guided maintenance or did you start manually? Was the status "succeeded"? A
cli% servicemag status
would be interesting.


The last posted output has a very bad format (not readable at all).

[PCD]

Sorry the showcage -d cage0 doesn't format correctly on my previous post.

Below is the better format of t he  showcage -d cage0  command.

CLD1-3PAR-InServ-01 cli% showcage -d cage0

Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side

 0 cage0 0:0:1     0 1:0:1     0     12 24-30   08   08   DC3  n/a

-----------Cage detail info for cage0 ---------

Position: Cabinet00 Bay7

-----------Midplane Info------------

 VendorId,ProductId     3PARdata,DC3

         Serial_Num  OPS69907C015F9A

           Node_WWN 20000050CC015F9A

   TempSensor_State               OK

   TempSensor_Value               29

     OpsPanel_State               OK

Audible_Alarm_State            Muted

          ID_Switch                1

         Cage_State               OK

Interface Board Info   LoopA   LoopB

     Firmware_status Current Current

         Product_Rev      08      08

           IFC_State      OK      OK

           ESH_State      OK      OK

          Master_CPU     Yes      No

            Loop_Map   valid   valid

          Link_Speed   2Gbps   2Gbps

         Port0_State      OK      OK

         Port1_State  No_SFP  No_SFP

         Port2_State  No_SFP  No_SFP

         Port3_State  No_SFP  No_SFP

Power Supply Info State Fan State AC Model

              ps0    OK  MedSpeed OK    --

              ps1    OK  MedSpeed OK    --

--------------Drive Info--------------- ----LoopA----- ----LoopB-----

Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState

  0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK

  1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK

  3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK

  4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK

  5:0 2210000a3300c198   Normal      30 0xd6        OK 0xd6        OK

  7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK

  8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK

  9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK

 11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK

 12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK

 13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK

 15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK

CLD1-3PAR-InServ-01 cli% showcage -d cage0

Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side

 0 cage0 0:0:1     0 1:0:1     0     12 24-29   08   08   DC3  n/a

-----------Cage detail info for cage0 ---------

Position: Cabinet00 Bay7

-----------Midplane Info------------

 VendorId,ProductId     3PARdata,DC3

         Serial_Num  OPS69907C015F9A

           Node_WWN 20000050CC015F9A

   TempSensor_State               OK

   TempSensor_Value               29

     OpsPanel_State               OK

Audible_Alarm_State            Muted

          ID_Switch                1

         Cage_State               OK

Interface Board Info   LoopA   LoopB

     Firmware_status Current Current

         Product_Rev      08      08

           IFC_State      OK      OK

           ESH_State      OK      OK

          Master_CPU     Yes      No

            Loop_Map   valid   valid

          Link_Speed   2Gbps   2Gbps

         Port0_State      OK      OK

         Port1_State  No_SFP  No_SFP

         Port2_State  No_SFP  No_SFP

         Port3_State  No_SFP  No_SFP

Power Supply Info State Fan State AC Model

              ps0    OK  MedSpeed OK    --

              ps1    OK  MedSpeed OK    --

--------------Drive Info--------------- ----LoopA----- ----LoopB-----

Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState

  0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK

  1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK

  3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK

  4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK

  5:0 2210000a3300c198   Normal      29 0xd6        OK 0xd6        OK

  7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK

  8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK

  9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK

 11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK

 12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK

 13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK

 15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK

CLD1-3PAR-InServ-01 cli%

[PCD]

I start manually. The disk is a brand new disk.

Here is the servicemag status -d info:

CLD1-3PAR-InServ-01 cli% servicemag status -d Cage 0, magazine 9: A servicemag start command failed on this magazine. The command completed at Wed Jul 17 06:36:14 2013. The output of the servicemag start was: Failed -- Unable to run servicemag command Start at the moment, a start/resume may already ongoing. servicemag start -pdid 34 -- Failed CLD1-3PAR-InServ-01 cli%