Storage Boards Cleanup
To make it easier to find information about HPE Storage products and solutions, we are doing spring cleaning. This includes consolidation of some older boards, and a simpler structure that more accurately reflects how people use HPE Storage.
3PAR StoreServ Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

Evicting Disks from 3PAR Array

3padm
Advisor

Evicting Disks from 3PAR Array

Hi,

 

We have an F400 array and wish to complely evict 4 failed disks from the NL disk group. Are there a set of commands we can run to 'spin down' the disks and have them completly out of the array ?

 

Here is the output from 'showpd -c' on the failed disks.

 

3PAR-001 cli% showpd -c 13 22 28 40
                             ------- Normal Chunklets -------- ---- Spare Chunklets ----
                             - Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State  Total OK  Fail Free Uninit Unavail Fail OK  Fail Free Uninit Fail
13 1:1:0       NL   failed  3724  0     0    0      0    3318  297  0     0    0     0  109
22 1:13:0    NL   failed  3724  0     0    0   1460       0 2159  0     0    0     0  105
28 2:5:0       NL   failed  3724  0     0    0   1461       0 2156  0     0    0     0  107
40 3:5:0      NL   failed   3724  0     0    0      0    3610    6  0     0    0     0  108
----------------------------------------------------------------------------------------
 4 total               14896  0     0    0   2921    6928 4618  0     0    0 

 

Thanks

 

17 REPLIES
Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>We have an F400 array and wish to completely evict 4 failed disks from the NL disk group.

 

I was going to suggest you look at this topic for instructions on how to do servicemag but I see it's yours:

http://h30499.www3.hp.com/t5/Storage-Area-Networks-SAN/Adding-new-disks-to-3PAR-System/m-p/6164679

 

So what's the current status and how come you weren't able to replace the disks?

Have you talked to Support yet?

 

Basically the command to remove a PD is dismisspd.  But in practice it is not obvious what you need to do first.

1) You have to permanently move off the used chunklets

2) You have to remove the spares on the PD. (easy and quick)

3padm
Advisor

Re: Evicting Disks from 3PAR Array (dismisspd)

Hi,

 

Long story but It looks like the array looks at the caddys WWN rather than the disks so when we replace the disk - by putting it in the same caddy, the array does not care and still shows the caddys wwn thinking its the same old broken disk. We did have support on this array but it expired so we need to do it the long way. Have many other 3Pars but this one is only for Dev/Backups but still need to get it in a stable state.

 

So in the short term we thougth we'd evict the 4 failed disks and change the sparing algorithum from 'minimum' to 'default' but the array doesnt like to change this policy since it is already in an unstable state.

 

So what would be the commands to get say disk 3:5:0 out cleanly? 

 

Thanks

 

Torsten.
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>> Long story but It looks like the array looks at the caddys WWN rather than the disks



We discussed this in your other thread already!


There is NO "caddy WWN"!

I still guess you used unsupported disk as a replacement.

 

 

Such replacement may even fail if the system firmware is too old to recognize newer disk firmware.


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>but It looks like the array looks at the caddy's WWN

 

Did you ever provide the "showcage -d" output that I requested?  (While the new disk was inserted?)

 

>We did have support on this array but it expired so we need to do it the long way.

 

Hopefully you'll reconsider.  There may be something wrong with the loop to that magazine.

All the cases I put in an unsupported disk, showed the new WWN, serial, vendor and model.

 

> change the sparing algorithm from 'minimum' to 'default'

 

What command are you using to change it?

 >So what would be the commands to get say disk 3:5:0 out cleanly?

 

I'll have to check my notes for step 1).  2) is just removespare.
 

>There is NO "caddy WWN"!

 

Especially not in an F class.  For T and V, there is a bridge to enable SAS drives but I wasn't aware it had a WWN?

 

>Such replacement may even fail if the system software is too old

 

3.1.2 should be new enough.  But there is MU2.

Torsten.
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

Now I understand what you are trying to tell.

 

 

 

The disk chassis has a fibre channel interface, but all the SSD, SATA and SAS disks used in the array have not.

 

They have an adapter to "translate" to fibre channel and this has a WWN.

 

Consider to see this adapter as part of the disk!

 

You exchanged the disk only, you kept the adapter, right? This is your "caddy WWN".

 

From the system point of view there is still the same old disk, because there is still the same adapter with the same WWN.

 

 

You need an original spare part including caddy, adapter and disk.

 

I already sent you the part numbers.

 

Otherwise you may try a "dissmisspd", maybe this may help to remove the disk without replacement.

 

 

 

Another option could be to "unfail" the disk - you need to discuss this with 3par support.

 


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>They have an adapter to "translate" to fibre channel and this has a WWN.

 

Well for SAS in T and V, it's called a bridge.  :-)  And it's part of the magazine.

 

>You exchanged the disk only, you kept the adapter, right? This is your "caddy WWN".

 

Using the bridge, you see one of the 4 WWNs from the SAS disk.

 

>you may try a "dismisspd", maybe this may help to remove the disk without replacement.

 

You'll quickly find it's easier said than done.  :-)

Torsten.
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>> You'll quickly find it's easier said than done. :-)

 

For the system the data from the bad disk is still spared, not in a final position.

The bad disks are still bad, missing.

In other words, data still belongs to the bad disk.

The data needs to be moved permanently, not only spared.

I added the suggestion again to discuss this with 3par support.

But I think they will suggest to use an "original" spare part ... try to call them anyway.


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>So what would be the commands to get say disk 3:5:0 out cleanly?

>Id CagePos Type State  Total OK  Fail Free Uninit Unavail Fail OK  Fail Free Uninit Fail

>40 3:5:0        NL    failed    3724  0      0        0      0             3610    6      0     0        0        0     108

 

This is PD 40.

 

(Swapping my original ordering)

1) To remove the spares from PD 40:

    removespare 40:a

 

2) To move the chunklets permanently off of PD 40, you need to first find them:

   showpdch -mov -from 40

 

Please provide the output of the showpdch command so I can make sure my script first works.

(This may be a big list, so you should attach as a .txt file.)

 

Also, you can now try the "dismisspd 40" to see what errors you get.

Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>I added the suggestion again to discuss this with 3par support.

 

Exactly.

 

>But I think they will suggest to use an "original" spare part ... try to call them anyway.

 

One suggestion is to try to add that disk in a new slot to just see if the system recognizes it.

(But don't do anything else besides "showpd -i".)

 

This isolates any problems with servicemag or the old slot/loop.

Torsten.
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

I'm sure the root cause is the same WWN carried over with the "bridge" in the disk magazine.
The systems will "remember" this WWN as bad forever.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
3padm
Advisor

Re: Evicting Disks from 3PAR Array (dismisspd)

Hi,

 

Sorry havent replied sooner - working UK time. In short it looked like the issue was resolved as we managed to track down a new caddy. The array picked up the new disk in the new caddy, gave it a new ID and off it went relocatig chunclets....until this morning I ran service status and this is what I got grrrrr...

 

3PAR-001 cli% servicemag status -d 2 5
A servicemag resume command failed on this magazine.
The command completed at Thu Aug 29 00:26:20 2013.
The output of the servicemag resume was:
servicemag resume 2 5
... mag 2 5 already onlooped
... upgrading firmware on pd WWN [2210000A330077F7]...
... firmware is current on pd WWN [2210000A330066F4] Id [28]
... checking for valid disks...
... checking for valid disks...
...   disks in mag  : 2 5
...      normal disks:  WWN [2210000A330077F7] Id [ 4]  diskpos [0]
...  not normal disks:  WWN [2210000A330066F4] Id [28]
... verifying spare space for disks 28 and 4
... playback chunklets from pd WWN [2210000A330077F7] Id [ 4]
... All chunklets played back / relocated.
... cleared logging mode for cage 2 mag 5
... relocating chunklets from spare space
... chunklet 5:167 - move_error,move_failed, failed move
... chunklet 5:169 - move_error,move_failed, failed move
... chunklet 5:171 - move_error,move_failed, failed move
... chunklet 5:173 - move_error,move_failed, failed move
... chunklet 5:174 - move_error,move_failed, failed move
... chunklet 5:175 - move_error,move_failed, failed move
... chunklet 5:176 - move_error,move_failed, failed move
... chunklet 5:177 - move_error,move_failed, failed move

......................this goes on for a while....

 

For around 4 /6 hours it looked like it was ok and doing what it should. Within the gui everything looks ok but under the 'Device Protocol' columa all devcies are showing up s SATA apart from the new Disk this is showing up as FC. I have attached this as a screenshot.

 

 

 

Thanks

 

Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>I'm sure the root cause is the same WWN carried over with the "bridge" in the disk magazine.
>The systems will "remember" this WWN as bad forever.

 

And of course I kind of agree and disagree.

Ah, it appears the F class does have a bridge, SATA<->FC:

40 3:5:0   failed 2210000A330066AB SEAGATE ST31000340NS 9QJ6S683 XR38,1610 SATA     Magnetic

 

That ",1610" is the bridge FW version.

 

But if sysmgr can't talk to the disk (loop problems) IT will still remember the old WWN.

That's why a "showcage -d" is needed.  The cage may see the new WWN.  Or it may not see anything at all.

Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>as SATA apart from the new Disk this is showing up as FC.

 

I've seen this happen with SAS when it has problems with disk models.

 

>chunklet 5:167 - move_error,move_failed, failed move

 

You may want to look at event logs to see if more details.

3padm
Advisor

Re: Evicting Disks from 3PAR Array (dismisspd)

Hi,

 

The disk models appear to be the same 'ST31000340NS' for the existing and new disk. Going back to the original question; how to evict a disk. I followed your steps and attached is the output. Though I was a little cheeky and ran 'removespare ' before showpdch . Interestingly it complains about the same chuncklet '5:167' as you have below which is the first one in the list...!

 

Thanks

Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>I followed your steps and attached is the output.

 

You had listed PD 40 as bad but your steps have PD 4!

 

3Par-001 cli% removespare 4:a
Are you sure you want to remove spares?
select q=quit y=yes n=no: y
107 spares removed


3Par-001 cli% dismisspd 4
Error : Pd id 4 is referenced by chunklet 5:167
3Par-001 cli% showpdch -mov -from 4

 

Perhaps you were trying to work on PD 28 which moved to PD 4?

 I picked 40 since you hadn't done a servicemag on it.

 

>ran 'removespare' before showpdch.

 

The ordering isn't a problem because showpdch checks used before spare.  But you have removed the spares from the wrong disk.  Unfortunately createspare needs to be told exactly which chunklet to be made spare on PD 4.

 

To put them back, I would need the output of:

$ showpdch -a 4

 

But this may be moot if PD 4 is bad too.

 

>it complains about the same chunklet '5:167' as you have below which is the first one in the list!

 

Yes, that's the first one to be moved back.

(I guess these were the ones moved from PD 28 to PD 5.)

 

Unfortunately we don't know what this means:

.. chunklet 5:167 - move_error,move_failed, failed move

 

Did it fail copying from PD 5?  Or to your new disk, PD 4 is bad?

Anything move in the eventlogs or alerts?

3padm
Advisor

Re: Evicting Disks from 3PAR Array (dismisspd)

Hi,

 

Sorry I should have stated yes PD 4 is bad too. See screenshot. Kinda stuck on this one.

 

 

 

ill check logs shortly

 

thanks

Dennis Handly
Acclaimed Contributor

Re: Evicting Disks from 3PAR Array (dismisspd)

>PD 4 is bad too.

 

I think PD 4 is the new number when you did a servicemag on PD 28.  It has the same cage position.

 

>>Did it fail copying from PD 5?  Or to your new disk, PD 4 is bad?

 

What do these show:

showpd -s 4 5

showpd -i 4 5

showcage -d cage2