Replacing disks on 3PAR System (servicemag)

[3padm]

Hi, We have a Inserv F400 with 4 failed NL Disks which need replacing with new ones. Does anyone have the step-by-step instructions on how this is done? From my understanding this is not as straightforward like an EVA where disks are simply removed and added.

Could you help!

P.S. This thread has been moved from Storage > Training & Education to Storage Area Networks (SAN) (Enterprise).
-HP Forum Moderator

[Dennis Handly]

>We have a Inserv F400 with 4 failed NL Disks which need replacing with new ones.

What version of the HP 3PAR OS do you have?

>Does anyone have the step-by-step instructions on how this is done?

Have you looked at the CLI Admin manual?

The command is "servicemag start" on each magazine.  After replacing, you do "servicemag resume".

[3padm · Accepted solution]

Hi,

OS Version is 3.1.2. I though you would have to run 'admithw' to submit new hardware?

Do you know the start to end proceadure i.e.

remove broken disks (run abc)

add new disks ( this xyz)

Do healthcheck (...) ?

This would be much appriciated!

[Dennis Handly]

>I though you would have to run 'admithw' to submit new hardware?

That's for new, not for replacements.

>Do you know the start to end procedure

I don't remember if your F class has multiple disks per mag?  The output of showpd below would verify.

(I found pictures, it only has one disk per mag.)

If multiple, then you replace them all at the same time with the one servicemag.

If not, you can do multiple servicemag commands at the same time but would be safer to do one by one.

Multiple would put more of an I/O load on the system and lead to possible confusion during replacement.

1) First find your failed disks:

    a) showpd -s -failed -degraded

    b) showpd -i -failed -degraded

Please post the output of these two commands.

2) Taking the cage and mag info from the above commands:

     servicemag start cage# mag#

3) Checking status:

     servicemag status -d cage# mag#

4) Once this is done, you can replace the disk(s) in the magazine.   Use this to blink the LEDs:

     locatecage -t 255 <cage_name> <mag>

     (Note: This is a cage name, not number.  I.e. cage0.

    Warning: Make sure you replace the right magazine and disk!

    Make sure you familiarize yourself with the magazine and disk numbering and ordering direction.

5) Then do do the resume:

     servicemag resume cage# mag#

6) You can check the status as in 3).

When done, you could do 1) to make sure no more failed disks.

And you could do:

  checkhealth pd

You might want to check out the HPGT for F class:

https://h20566.www2.hpe.com/portal/site/hpsc/template.PAGE/public/psi/manualsResults/?sp4ts.oid=5047531&spf_p.tpst=psiContentResults&spf_p.prp_psiContentResults=wsrp-navigationalState%3Daction%253Dmanualslist%257Ccontentid%253DUser-Guide-%252528how-to-use%252529%257Clang%253Den&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

I see a locating a cage here:

https://h20566.www2.hpe.com/portal/site/hpsc/template.PAGE/public/psi/manualsResults/?sp4ts.oid=5047531&spf_p.tpst=psiContentResults&spf_p.prp_psiContentResults=wsrp-navigationalState%3Daction%253Dmanualslist%257Ccontentid%253DUser-Guide-%252528how-to-use%252529%257Clang%253Den&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Unfortunately the example forgot the time value after the -t.  :-(

Layout and drive order:

http://h20584.www2.hp.com/hpgt/guides/select?lang=en&cc=us&prodTypeId=12169&prodSeriesId=5047531&lang=en&cc=us

[3padm]

Hi thank you that looks great... hereis the output from the commands.

showpd -s -failed -degraded

Id CagePos Type -State- ------------------Detailed_State------------------
13 1:1:0   NL   failed  vacated,invalid_media,media_failed
22 1:13:0  NL   failed  vacated,invalid_media,smart_threshold_exceeded
28 2:5:0   NL   failed  vacated,invalid_media,smart_threshold_exceeded
40 3:5:0   NL   failed  vacated,invalid_media,multiple_chunklets_media_bad
--------------------------------------------------------------------------
 4 total

showpd -i -failed -degraded

Id CagePos State  ----Node_WWN---- --MFR-- ---Model---- -Serial- -FW_Rev-- Protocol MediaType
13 1:1:0   failed 2210000A330061F8 SEAGATE ST31000340NS 9QJ6S5BY XR38,1610 SATA     Magnetic
22 1:13:0  failed 2210000A33006790 SEAGATE ST31000340NS 9QJ6JH63 XR38,1610 SATA     Magnetic
28 2:5:0   failed 2210000A330066F4 SEAGATE ST31000340NS 9QJ6MKC4 XR38,1610 SATA     Magnetic
40 3:5:0   failed 2210000A330066AB SEAGATE ST31000340NS 9QJ6S683 XR38,1610 SATA     Magnetic
---------------------------------------------------------------------------------------------
 4 total

[Torsten.]

Probably the most simple method would be now to point your browser to the SP and use guided maintenance to replace the disks one by one (wait for the re-balance to finish before replacing the next disk - you may check with servicemag status command).

[3padm]

guided maintenance is not an option my friend as we dont seem to have the password :(

plus I always thgouht GM was only to swap disks on the SP rather than the disk in the shelvs ?

[Dennis Handly]

>13 1:1:0   NL   failed  vacated,invalid_media,media_failed
>22 1:13:0  NL   failed  vacated,invalid_media,smart_threshold_exceeded
>28 2:5:0   NL   failed  vacated,invalid_media,smart_threshold_exceeded
>40 3:5:0   NL   failed  vacated,invalid_media,multiple_chunklets_media_bad

Since these are already vacated, hopefully the first part of servicemag will go fast.

The cage and mag for each is:

servicemag start  1 1

servicemag start 1 13

servicemag start 2 5

servicemag start 3 5

For locatecage:

locatecage -t 255 cage1 1

locatecage -t 255 cage1 13

locatecage -t 255 cage2 5

locatecage -t 255 cage3 5

[3padm]

Just to be extra sure will this therefore be the steps for asingle disk?

1- servicemag start  1 1
2- locatecage -t 255 cage1 1
3- Remove faulty Disk
4- Add new Disk
5- servicemag resume 1 1
6- checkhealth pd

Then move onto disk 2?

Thanks again!

[Torsten.]

Sounds good. Before doing (3), wait until "servicemag status" shows "succeeded".

[Dennis Handly]

>wait until "servicemag status" shows "succeeded".

Right.  You can also add -d for details:  servicemag status -d

[3padm]

Hi, after placing the new disk in and running the resume command, the SP whent into checking the new disk and the wan basically said disk was not normal...

----------------------------------------------------------------------------------------

P-001 cli% servicemag status -d 3 5
The magazine was successfully brought offline by a servicemag start command.
The command completed Mon Aug 19 09:06:32 2013.
The output of the servicemag start was:
servicemag start 3 5
... servicing disks in mag: 3 5
...      normal disks:
...  not normal disks:  WWN [2210000A330066AB] Id [40]  diskpos [0]
... relocating chunklets to spare space...
... spinning down disk WWN [2210000A330066AB] Id [40]
... bypassing mag 3 5
... bypassed mag 3 5
servicemag start 3 5 -- Succeeded
3P-001 cli% showpd -c 3 5
                             -------- Normal Chunklets -------- ---- Spare Chunklets ----
                             - Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State  Total   OK Fail Free Uninit Unavail Fail  OK Fail Free Uninit Fail
 3 0:4:0   FC   normal  1115   76    0 1004      0       0    0   0    0   35      0    0
 5 4:2:0   NL   normal  3724 1368    0 2247      0       0    0 109    0    0      0    0
-----------------------------------------------------------------------------------------
 2 total                4839 1444    0 3251      0       0    0 109    0   35      0    0
3P-001 cli% servicemag status -d 3 5
The magazine was successfully brought offline by a servicemag start command.
The command completed Mon Aug 19 09:06:32 2013.
The output of the servicemag start was:
servicemag start 3 5
... servicing disks in mag: 3 5
...      normal disks:
...  not normal disks:  WWN [2210000A330066AB] Id [40]  diskpos [0]
... relocating chunklets to spare space...
... spinning down disk WWN [2210000A330066AB] Id [40]
... bypassing mag 3 5
... bypassed mag 3 5
servicemag start 3 5 -- Succeeded
3P-001 cli% servicemag status -d 3 5
The magazine was successfully brought offline by a servicemag start command.
The command completed Mon Aug 19 09:06:32 2013.
The output of the servicemag start was:
servicemag start 3 5
... servicing disks in mag: 3 5
...      normal disks:
...  not normal disks:  WWN [2210000A330066AB] Id [40]  diskpos [0]
... relocating chunklets to spare space...
... spinning down disk WWN [2210000A330066AB] Id [40]
... bypassing mag 3 5
... bypassed mag 3 5
servicemag start 3 5 -- Succeeded
3P-001 cli% servicemag status -d 3 5
The magazine was successfully brought offline by a servicemag start command.
The command completed Mon Aug 19 09:06:32 2013.
The output of the servicemag start was:
servicemag start 3 5
... servicing disks in mag: 3 5
...      normal disks:
...  not normal disks:  WWN [2210000A330066AB] Id [40]  diskpos [0]
... relocating chunklets to spare space...
... spinning down disk WWN [2210000A330066AB] Id [40]
... bypassing mag 3 5
... bypassed mag 3 5
servicemag start 3 5 -- Succeeded
3P-001 cli% servicemag resume 3 5
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 3 5
... mag 3 5 already onlooped
... firmware is current on pd WWN [2210000A330066AB] Id [40]
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 3 5
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 3 5
... checking for valid disks...
... checking for valid disks...
... disks not normal yet..trying admit/onloop again
... onlooping mag 3 5
... checking for valid disks...
Failed --
disk WWN [2210000A330066AB] Id [40] is not normal. Please use showpd -s to see details of disk state
servicemag resume 3 5 -- Failed
Command failed

------------------------------------------------------------------------------------------------------

are there additional commands to run to get the firmware upto date or what else is missing from my commands ??

Thanks!

[Torsten.]

showpd -c 3 5

is showing disks 3 and 5, but you want to see disk 40!

BUT ...

... spinning down disk WWN [2210000A330066AB] Id [40] --- the old disk

disk WWN [2210000A330066AB] Id [40] is not normal

This is still the same disk - no new disk listed.

Please do

% showpd -c -p -cg 3

% showpd -failed -degraded

% showpd -s 40

Did you replace the wrong disk?

[3padm]

40 is the disk ID and 3 5 is the posision. so this is the right command

[Torsten.]

The replaced disk will have another ID than the bad disk at this position.

[3padm]

This is the thing! Once I replace the disk the ID remains the same, do you know if I need to replace the caddy too? At the moment we are removing the old disk with alankey then placing the new disk into the original caddy. Is this the right approach?

[Torsten.]

Please do



% showpd -c -p -cg 3



% showpd -failed -degraded



% showpd -s 40

[3padm]

Here you go Mr....

3P-001 cli% showpd -c -p -cg 3
                              -------- Normal Chunklets --------- ---- Spare Chunklets ----
                              - Used -- -------- Unused --------- - Used - ---- Unused ----
 Id CagePos Type State  Total   OK Fail  Free Uninit Unavail Fail OK  Fail Free Uninit Fail
 36 3:0:0   FC   normal  1115   85    0   995      0       0    0  0     0   35      0    0
 37 3:1:0   NL   normal  3724 1352    0  2263      0       0    0  0     0  109      0    0
 38 3:3:0   FC   normal  1115   73    0  1007      0       0    0  0     0   35      0    0
 40 3:5:0   NL   failed  3724    0    0     0      0    3610    6  0     0    0      0  108
 41 3:7:0   FC   normal  1115   81    0   999      0       0    0  0     0   35      0    0
 42 3:8:0   FC   normal  1115   86    0   994      0       0    0  0     0   35      0    0
 44 3:11:0  FC   normal  1115   76    0  1004      0       0    0  0     0   35      0    0
 45 3:12:0  FC   normal  1115   75    0  1005      0       0    0  0     0   34      0    1
 46 3:13:0  NL   normal  3724 1626    0  1989      0       0    1  0     0  108      0    0
 47 3:15:0  FC   normal  1115   83    0   997      0       0    0  0     0   35      0    0
 60 3:2:0   FC   normal  1115   68    0  1012      0       0    0  0     0   35      0    0
 61 3:6:0   FC   normal  1115   64    0  1016      0       0    0  0     0   35      0    0
 62 3:10:0  FC   normal  1115   69    0  1011      0       0    0  0     0   35      0    0
 63 3:14:0  FC   normal  1115   64    0  1016      0       0    0  0     0   35      0    0
 80 3:9:0   NL   normal  3724 1355    0  2260      0       0    0  0     0  109      0    0
112 3:4:0   FC   normal  1115   76    0  1005      0       0    0  0     0   34      0    0
-------------------------------------------------------------------------------------------
 16 total               28276 5233    0 18573      0    3610    7  0     0  744      0  109
3P-001 cli% showpd -failed -degraded
                           --Size(MB)-- ----Ports----
Id CagePos Type RPM State    Total Free A      B      Cap(GB)
13 1:1:0   NL     7 failed  953344    0 2:0:2  3:0:2*    1000
22 1:13:0  NL     7 failed  953344    0 2:0:2* 3:0:2     1000
28 2:5:0   NL     7 failed  953344    0 2:2:1* 3:2:1     1000
40 3:5:0   NL     7 failed  953344    0 2:2:2* 3:2:2     1000
-------------------------------------------------------------
 4 total                   3813376    0
3P-001 cli% showpd -s 40
Id CagePos Type -State- ----------------------------------Detailed_State-----------------------------------
40 3:5:0   NL   failed  vacated,invalid_media,multiple_chunklets_media_bad,invalid,inquiry_failed,servicing
-----------------------------------------------------------------------------------------------------------
 1 total
3P-001 cli%

[3Par_Eng]

I think you have inserted the same disk again. You can confirm by issuing the following command:

showpd -i -p -cg 3 -mg 5

 If the WWN is 2210000A330066AB, then its the same old disk that you were trying to replace.

The new disk will have a new WWN and a new ID, but will be at the same position.

When you remove the disk with WWN 2210000A330066AB, it will go missing and the ID will be displayed untill servicemag resume completes.

[3padm]

Hi, appriciate your help so far...hopefully we can crack this issue!  Here is the output...

CLRB-STF400-001 cli% showpd -i -p -cg 5 -mg 5
Id CagePos State  ----Node_WWN---- --MFR-- -----Model----- ----Serial---- -FW_Rev-- Protocol MediaType
85 5:5:0   normal 2210000A33009BE7 Hitachi HUA722010ZLA330 JK11A8B9J4U83F A3GF,1610 SATA     Magnetic
------------------------------------------------------------------------------------------------------

I can comfirm that the new disk is infact new and was un wrapped this morning.

Thanks!

[3Par_Eng]

Please provide outtpu of:

showpd -i -p -cg 3 -mg 5

and not

showpd -i -p -cg 5 -mg 3

[Torsten.]

Look at this:

cli% servicemag resume 3 5
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 3 5
... mag 3 5 already onlooped
... firmware is current on pd WWN [2210000A330066AB] Id [40]
... checking for valid disks...
... checking for valid disks...

Usually you see the new disk here too, but not in this case.

Are you sure you replaced the correct disk position with a correct supported replacement drive?

If you do a

% locatecage cage3

all LEDs of cage 3 will blink orange for 1 minute.

Is this the cage where you replaced the disk?

Maybe the output of

% showpd

% showalert

% checkhealth -svc

could help.

[3padm]

sorry here you go

3p-001% showpd -i -p -cg 3 -mg 5
Id CagePos State  ----Node_WWN---- -MFR- ---Model---- -Serial- -FW_Rev- Protocol MediaType
40 3:5:0   failed 2210000A330066AB ATA   ST31000340NS 9QJ6S683 ,1610    SATA     Magnetic
------------------------------------------------------------------------------------------

[3padm]

I can see that the wwn is the same but I have a very strong feeling this is the caddy wwn and not the disk. As this was visible while I was loadin it.

Do I need to replace caddy too?

[Torsten.]

The caddy has no WWN, the label on it refers to the disk.

Please check the disk location - see my previous post.