Storage Boards Cleanup
To make it easier to find information about HPE Storage products and solutions, we are doing spring cleaning. This includes consolidation of some older boards, and a simpler structure that more accurately reflects how people use HPE Storage.
3PAR StoreServ Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

Servicemag resume failed

PCD
Occasional Advisor

Servicemag resume failed

We replace a failed hard disk on our 3PAR Inserv E200 SAN.

We initiated the command "servicemag start -pdid 34 and the command was successful.

Next we physically replace the new hard drive.

Then we inititated the command "servicemag resume 0 9"  but this command failed and we got the following error:

 

 

CLD1-3PAR-InServ-01 cli% servicemag status -d
Cage 0, magazine 9:
A servicemag resume command failed on this magazine.
The command completed at Mon Jul 15 09:02:12 2013.
The output of the servicemag resume was:
servicemag resume 0 9
... mag 0 9 already onlooped
... firmware is current on pd WWN [2210000A33008420] Id [34]
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
... checking for valid disks...
... disks not valid yet..trying admit/onloop again
... onlooping mag 0 9
... checking for valid disks...
Failed --
disk WWN [2210000A33008420] Id [34] is not in valid state
servicemag resume 0 9 -- Failed

 

 

We try to search for this error but cannot find a solution for it. Please advise.

Also, we want to cancel this process  on cage 0 and magazine 9 and restart the disk replacement process again. And would like your advise if this command:

    "servicemag resume unmark -cg 0 -mg 9"  would do the job?

 

Thanks in advance for your comments.

16 REPLIES
Torsten.
Acclaimed Contributor

Re: Servicemag resume failed

Did you try to re-seat the disk?

 

Is it connected to both loops (showpd)?


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: servicemag resume failed

What do these commands show?

showversion

 

servicemag status -d

 

showpd 34

showpd -i 34

showpd -s 34

 

If ID 34 doesn't exist, then replace by: -p -cg 0 -mg 9

 

showcage -d cage0

 

>firmware is current on pd WWN [2210000A33008420] Id [34]

 

The OUI for this WWN is Emulex.

PCD
Occasional Advisor

Re: Servicemag resume failed

Hi,

 

Thank you for your quick respond.

I re-seated the disk but it still show the disk as failed (prior to reseat it said not ready)

Below is the showpd command after I re-seated the disk (ID 34 is the bad disk):

 

login as: 3paradm
Using keyboard-interactive authentication.
Password:
CLD1-3PAR-InServ-01 cli% showpd
ID CagePos   SizeMB  Chunk  Free Spare ----Node_WWN----  State  APort  BPort LdA
 0  0:0:0    380928   1488   377    93 20000024B601C8B9  valid 0:0:1* 1:0:1    Y
 1  0:3:0    380928   1488   375    93 20000024B601F561  valid 0:0:1  1:0:1*   Y
 2  0:4:0    380928   1488   375    93 20000024B601CEDA  valid 0:0:1  1:0:1*   Y
 3  0:7:0    380928   1488   376    93 20000024B601F569  valid 0:0:1* 1:0:1    Y
 4  0:8:0    380928   1488   377    93 20000024B601CE20  valid 0:0:1* 1:0:1    Y
 5 0:11:0    380928   1488   376    93 20000024B601F51D  valid 0:0:1  1:0:1*   Y
 6 0:12:0    380928   1488   376    93 20000024B601CA73  valid 0:0:1  1:0:1*   Y
 7 0:15:0    380928   1488   376    93 20000024B601F558  valid 0:0:1* 1:0:1    Y
 8  1:0:0    380928   1488   377    93 2000001862583D04  valid 0:0:2* 1:0:2    Y
 9  1:3:0    380928   1488   376    93 20000024B60138E1  valid 0:0:2  1:0:2*   Y
10  1:4:0    380928   1488   375    93 2000001862583B77  valid 0:0:2  1:0:2*   Y
11  1:7:0    380928   1488   376    93 20000024B6012FFE  valid 0:0:2* 1:0:2    Y
12  1:8:0    380928   1488   377    93 20000024B6014239  valid 0:0:2* 1:0:2    Y
13 1:11:0    380928   1488   376    93 20000024B6014204  valid 0:0:2  1:0:2*   Y
14 1:12:0    380928   1488   376    93 2000001862583CC4  valid 0:0:2  1:0:2*   Y
15 1:15:0    380928   1488   377    93 20000024B601401E  valid 0:0:2* 1:0:2    Y
16  2:0:0    380928   1488   377    93 20000024B60EF3E4  valid 0:3:1* 1:3:1    Y
17  2:3:0    380928   1488   376    93 20000024B601F539  valid 0:3:1  1:3:1*   Y
18  2:4:0    380928   1488   375    93 20000024B60EF3BB  valid 0:3:1  1:3:1*   Y
19  2:7:0    380928   1488   376    93 20000024B601F567  valid 0:3:1* 1:3:1    Y
20  2:8:0    380928   1488   377    93 20000024B60EF409  valid 0:3:1* 1:3:1    Y
21 2:11:0    380928   1488   376    93 20000024B601F551  valid 0:3:1  1:3:1*   Y
22 2:12:0    380928   1488   376    93 20000024B60EF351  valid 0:3:1  1:3:1*   Y
23 2:15:0    380928   1488   377    93 20000024B601F527  valid 0:3:1* 1:3:1    Y
24  3:0:0    380928   1488   378    93 20000024B601394B  valid 0:3:2* 1:3:2    Y
25  3:3:0    380928   1488   376    93 20000024B601CABA  valid 0:3:2  1:3:2*   Y
26  3:4:0    380928   1488   376    93 20000024B60134C2  valid 0:3:2  1:3:2*   Y
27  3:7:0    380928   1488   376    93 20000024B601CA28  valid 0:3:2* 1:3:2    Y
28  3:8:0    380928   1488   377    93 20000024B6013AA9  valid 0:3:2* 1:3:2    Y
29 3:11:0    380928   1488   376    93 20000024B601CA41  valid 0:3:2  1:3:2*   Y
30 3:12:0    380928   1488   376    93 20000024B60141DC  valid 0:3:2  1:3:2*   Y
31 3:15:0    380928   1488   377    93 20000024B601CE2A  valid 0:3:2* 1:3:2    Y
33 1:13:0    953344   3724     5   280 2210000A3300C3CA  valid 0:0:2* 1:0:2    Y
34  0:9:0    953344   3724   225   280 2210000A33008420 failed 0:0:1- 1:0:1-   N
35 0:13:0    953344   3724     5   280 2210000A330083A5  valid 0:0:1* 1:0:1    Y
36  1:1:0    953344   3724    14   280 2210000A33008F33  valid 0:0:2  1:0:2*   Y
37  1:5:0    953344   3724     5   280 2210000A3300837A  valid 0:0:2* 1:0:2    Y
38  1:9:0    953344   3724    14   280 2210000A33008449  valid 0:0:2  1:0:2*   Y
39  0:1:0    953344   3724    14   280 2210000A3300BCBD  valid 0:0:1  1:0:1*   Y
40  2:1:0    953344   3724   705   279 2210000A3300C3D6  valid 0:3:1  1:3:1*   Y
41  2:5:0    953344   3724  1112   279 2210000A3300C3B7  valid 0:3:1* 1:3:1    Y
42  2:9:0    953344   3724   706   279 2210000A3300C519  valid 0:3:1  1:3:1*   Y
43 2:13:0    953344   3724  1113   279 2210000A3300C3D5  valid 0:3:1* 1:3:1    Y
44  0:5:0    953344   3724     5   280 2210000A3300C198  valid 0:0:1* 1:0:1    Y
45  3:5:0    953344   3724  1112   279 2210000A3300C500  valid 0:3:2* 1:3:2    Y
46  3:9:0    953344   3724   705   279 2210000A3300C542  valid 0:3:2  1:3:2*   Y
47 3:13:0    953344   3724  1113   279 2210000A3300C4D0  valid 0:3:2* 1:3:2    Y
48  3:1:0    953344   3724   706   279 2210000A3300C7EC  valid 0:3:2  1:3:2*   Y
--------------------------------------------------------------------------------
48         27443200 107200 19599  7448
CLD1-3PAR-InServ-01 cli%

Torsten.
Acclaimed Contributor

Re: Servicemag resume failed

Was this disk used before?

I would expect a new disk ID at the same position 0:9:0 but there is none?

The "new" disk is inserted now?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
PCD
Occasional Advisor

Re: servicemag resume failed

Hi,

 

Thank you for your prompt respond.

Below is the command info of Showversion, servicemag status -d, showpd 34 and showcage -d cage0

I ran the command below after I re-seated the disk and try to do a servicemag start -pdid 34 :

 

showversion:

CLD1-3PAR-InServ-01 cli% showversion Release version 2.2.4 (MU4) Patches:  None

Component Name                   Version CLI Server                       2.2.4 (MU4) CLI Client                       2.2.4 (MU4) GUI Server                       2.2.4 (MU4) System Manager                   2.2.4 (MU4) Kernel                           2.2.4 (MU4) TPD Kernel Code                  2.2.4 (MU4) CLD1-3PAR-InServ-01 cli%

 

CLD1-3PAR-InServ-01 cli% servicemag status -d
Cage 0, magazine 9:
A servicemag start command failed on this magazine.
The command completed at Wed Jul 17 06:36:14 2013.
The output of the servicemag start was:
Failed --
Unable to run servicemag command Start at the moment, a start/resume may already ongoing.
servicemag start -pdid 34 -- Failed
CLD1-3PAR-InServ-01 cli%

 

CLD1-3PAR-InServ-01 cli% showpd 34
ID CagePos SizeMB Chunk Free Spare ----Node_WWN----  State  APort  BPort LdA
34  0:9:0  953344  3724  225   280 2210000A33008420 failed 0:0:1- 1:0:1-   N
----------------------------------------------------------------------------
 1         953344  3724  225   280
CLD1-3PAR-InServ-01 cli%

 

CLD1-3PAR-InServ-01 cli% showcage -d cage0 Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side  0 cage0 0:0:1     0 1:0:1     0     12 24-30   08   08   DC3  n/a

-----------Cage detail info for cage0 ---------

Position: Cabinet00 Bay7 -----------Midplane Info------------  VendorId,ProductId     3PARdata,DC3          Serial_Num  OPS69907C015F9A            Node_WWN 20000050CC015F9A    TempSensor_State               OK    TempSensor_Value               29      OpsPanel_State               OK Audible_Alarm_State            Muted           ID_Switch                1          Cage_State               OK

Interface Board Info   LoopA   LoopB      Firmware_status Current Current          Product_Rev      08      08            IFC_State      OK      OK            ESH_State      OK      OK           Master_CPU     Yes      No             Loop_Map   valid   valid           Link_Speed   2Gbps   2Gbps          Port0_State      OK      OK          Port1_State  No_SFP  No_SFP          Port2_State  No_SFP  No_SFP          Port3_State  No_SFP  No_SFP

Power Supply Info State Fan State AC Model               ps0    OK  MedSpeed OK    --               ps1    OK  MedSpeed OK    --

--------------Drive Info--------------- ----LoopA----- ----LoopB----- Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState   0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK   1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK   3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK   4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK   5:0 2210000a3300c198   Normal      30 0xd6        OK 0xd6        OK   7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK   8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK   9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK  11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK  12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK  13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK  15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK CLD1-3PAR-InServ-01 cli%

 

 

 

 

 

PCD
Occasional Advisor

Re: Servicemag resume failed

Yes, the new disk is inserted.

 

Torsten.
Acclaimed Contributor

Re: Servicemag resume failed

Did you use guided maintenance or did you start manually? Was the status "succeeded"? A
cli% servicemag status
would be interesting.


The last posted output has a very bad format (not readable at all).

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
PCD
Occasional Advisor

Re: servicemag resume failed

Sorry the showcage -d cage0 doesn't format correctly on my previous post.

Below is the better format of t he  showcage -d cage0  command.

 

CLD1-3PAR-InServ-01 cli% showcage -d cage0

Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side

 0 cage0 0:0:1     0 1:0:1     0     12 24-30   08   08   DC3  n/a

 

-----------Cage detail info for cage0 ---------

 

Position: Cabinet00 Bay7

-----------Midplane Info------------

 VendorId,ProductId     3PARdata,DC3

         Serial_Num  OPS69907C015F9A

           Node_WWN 20000050CC015F9A

   TempSensor_State               OK

   TempSensor_Value               29

     OpsPanel_State               OK

Audible_Alarm_State            Muted

          ID_Switch                1

         Cage_State               OK

 

Interface Board Info   LoopA   LoopB

     Firmware_status Current Current

         Product_Rev      08      08

           IFC_State      OK      OK

           ESH_State      OK      OK

          Master_CPU     Yes      No

            Loop_Map   valid   valid

          Link_Speed   2Gbps   2Gbps

         Port0_State      OK      OK

         Port1_State  No_SFP  No_SFP

         Port2_State  No_SFP  No_SFP

         Port3_State  No_SFP  No_SFP

 

Power Supply Info State Fan State AC Model

              ps0    OK  MedSpeed OK    --

              ps1    OK  MedSpeed OK    --

 

--------------Drive Info--------------- ----LoopA----- ----LoopB-----

Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState

  0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK

  1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK

  3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK

  4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK

  5:0 2210000a3300c198   Normal      30 0xd6        OK 0xd6        OK

  7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK

  8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK

  9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK

 11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK

 12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK

 13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK

 15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK

CLD1-3PAR-InServ-01 cli% showcage -d cage0

Id  Name LoopA Pos.A LoopB Pos.B Drives  Temp RevA RevB Model Side

 0 cage0 0:0:1     0 1:0:1     0     12 24-29   08   08   DC3  n/a

 

-----------Cage detail info for cage0 ---------

 

Position: Cabinet00 Bay7

-----------Midplane Info------------

 VendorId,ProductId     3PARdata,DC3

         Serial_Num  OPS69907C015F9A

           Node_WWN 20000050CC015F9A

   TempSensor_State               OK

   TempSensor_Value               29

     OpsPanel_State               OK

Audible_Alarm_State            Muted

          ID_Switch                1

         Cage_State               OK

 

Interface Board Info   LoopA   LoopB

     Firmware_status Current Current

         Product_Rev      08      08

           IFC_State      OK      OK

           ESH_State      OK      OK

          Master_CPU     Yes      No

            Loop_Map   valid   valid

          Link_Speed   2Gbps   2Gbps

         Port0_State      OK      OK

         Port1_State  No_SFP  No_SFP

         Port2_State  No_SFP  No_SFP

         Port3_State  No_SFP  No_SFP

 

Power Supply Info State Fan State AC Model

              ps0    OK  MedSpeed OK    --

              ps1    OK  MedSpeed OK    --

 

--------------Drive Info--------------- ----LoopA----- ----LoopB-----

Drive          NodeWWN    State Temp(C) ALPA LoopState ALPA LoopState

  0:0 20000024b601c8b9   Normal      24 0xe1        OK 0xe1        OK

  1:0 2210000a3300bcbd   Normal      28 0xe0        OK 0xe0        OK

  3:0 20000024b601f561   Normal      24 0xda        OK 0xda        OK

  4:0 20000024b601ceda   Normal      25 0xd9        OK 0xd9        OK

  5:0 2210000a3300c198   Normal      29 0xd6        OK 0xd6        OK

  7:0 20000024b601f569   Normal      25 0xd4        OK 0xd4        OK

  8:0 20000024b601ce20   Normal      26 0xd3        OK 0xd3        OK

  9:0 2210000a33008420 Degraded     N/A 0xd2        OK 0xd2        OK

 11:0 20000024b601f51d   Normal      26 0xce        OK 0xce        OK

 12:0 20000024b601ca73   Normal      25 0xcd        OK 0xcd        OK

 13:0 2210000a330083a5   Normal      28 0xcc        OK 0xcc        OK

 15:0 20000024b601f558   Normal      25 0xca        OK 0xca        OK

CLD1-3PAR-InServ-01 cli%

PCD
Occasional Advisor

Re: Servicemag resume failed

I start manually. The disk is a brand new disk.

Here is the servicemag status -d info:

 

CLD1-3PAR-InServ-01 cli% servicemag status -d Cage 0, magazine 9: A servicemag start command failed on this magazine. The command completed at Wed Jul 17 06:36:14 2013. The output of the servicemag start was: Failed -- Unable to run servicemag command Start at the moment, a start/resume may already ongoing. servicemag start -pdid 34 -- Failed CLD1-3PAR-InServ-01 cli%

PCD
Occasional Advisor

Re: Servicemag resume failed

Do you know a way to terminate a servicemag start  command?

 

 

Torsten.
Acclaimed Contributor

Re: Servicemag resume failed

It is degraded now, not failed.

 

How about

 

cli% showpd -c -p -cg 0


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
PCD
Occasional Advisor

Re: Servicemag resume failed

Here is the info:


CLD1-3PAR-InServ-01 cli% showpd -c -p -cg 0
         ---- Normal Chunklets ----- ---- Spare Chunklets ----
         -- Used -- ---- Unused ---- - Used - ---- Unused ----
ID Total    OK Fail Free Uninit Fail  OK Fail Free Uninit Fail
 0  1488  1018    0  377      0    0   0    0   93      0    0
 1  1488  1020    0  375      0    0   0    0   93      0    0
 2  1488  1020    0  375      0    0   0    0   93      0    0
 3  1488  1019    0  376      0    0   0    0   93      0    0
 4  1488  1018    0  377      0    0   0    0   93      0    0
 5  1488  1019    0  376      0    0   0    0   93      0    0
 6  1488  1019    0  376      0    0   0    0   93      0    0
 7  1488  1019    0  376      0    0   0    0   93      0    0
34  3724     0    0  225   3216    3   0    0  280      0    0
35  3724  3439    0    5      0    0 280    0    0      0    0
39  3724  3430    0   14      0    0  10    0  270      0    0
44  3724  3439    0    5      0    0 280    0    0      0    0
--------------------------------------------------------------
12 26800 18460    0 3257   3216    3 570    0 1294      0    0
CLD1-3PAR-InServ-01 cli%

Dennis Handly
Acclaimed Contributor

Re: servicemag resume failed

>we want to cancel this process  on cage 0 and magazine 9 and restart the disk replacement process again. And would like your advise if this command:

> servicemag resume unmark -cg 0 -mg 9,  would do the job?

 

You would probably need to talk to Support about that.

And as long as "servicemag resume" or showpd can't talk to the disk, there isn't much you can "restart".

Note: unmark and resume are mutually exclusive.

 

What does this show:  showpd -state 34

 

>34  0:9:0    953344   3724   225   280 2210000A33008420 failed 0:0:1- 1:0:1-   N

 

You may need to try to manually onloop the mag, since the "-" above says the ports are disabled:

controlmag onloop cage0 9

 

If that doesn't work, you may have to manually reset the two ports.

Note:  There are 12 disks on these two ports!  So you should probably wait between the two commands.

controlport rst 0:0:1

controlport rst 1:0:1

John Bowe
Occasional Contributor

Re: Servicemag resume failed

How soon after inserting the magazine did you try to bring it online?  There needs to be a few minutes before initiating the "resume" command for all the drives to spin up.  If you initiate the command too soon, this error will occur.

moinf
Visitor

Re: Servicemag resume failed

We had the same issue on 1 13 in our 3par HP_3PAR 7400

It was solved when we triggered: 

servicemag resume -partial 1 13
........
Estimated time for relocation completion based on 29 seconds per chunklet is: 1 days, 14 minutes and 21 seconds
The cumulative output so far is:
servicemag resume -partial 1 13
.........
 checking for valid disks...
... disks in mag : 1 13
... normal disks: WWN [5000CCA0321642D7] Id [145] diskpos [0]
... not normal disks: WWN [5000CCA02D0E96E7] Id [87]
... verifying spare space for disks 145 and 145
... playback chunklets from pd WWN [5000CCA0321642D7] Id [145]
... All chunklets played back / relocated.
... cleared logging mode for cage 1 mag 13
... relocating chunklets from spare space
Juliano_Flores
Occasional Advisor

Re: Servicemag resume failed

Hello moinf,

This exactly situation (3PAR 7400: disk replaced + "servicemag resume -partial...") didn't work for us: the HPE was needed to replace this disk for a new one.

Best Regards,
Juliano Flores