MSA HP Storage Works P2000 G3 - vdisk went offline during PSU replacement.

[shahrukhntdc]

Dear All.

My name is Shahrukh and I am a SCADA Engineer. We are using MSA HP Storage Works P2000 G3 in our network in redundant configuration with RAID-5. One of the MSA P2000, I will call it STB02 (Storage Bay 02), was facing some PSU issues for some time. Today I was replacing the faulty PSU with a new one but unfortunately after replacement of PSU, when I checked the STB02 on SMU, the event display said, vdisk went offline. In Vdisks tree, instead of vd01 (RAID5), there is only :datanode" written. Now in Front Panel Graphical View, There is only 1 disk visible (actually 4 physical Disks) with LEFTOVR tag and in status it says OFFL. I am totally new to this MSA thing, unfortunatley I restarted the Manage Controller of this STB02 from Tools of SMU and now its no more accessible on SMU from Web Browser and I cannot watch events or any other SMU display. STB02 is not coming back on Network, no ping reply. STB02 is accessible on CLI. All the 4 Hard drives of the array has Orange LED ON. There is no Warning "LED ON" on Right Ear Cap and on Left Ear Cap "1" is visible. The other P2000 (STB01) is working fine. Please help me.

Regards,

Shahrukh

[shahrukhntdc]

Here is the output of CLI command show vdisks:

HP StorageWorks MSA Storage P2000 G3 FC

System Name: ldsnstb02

System Location:Uninitialized Location

Version:L100R013

# show vdisks

Name  Size     Free   Own Pref   RAID   Disks Spr Chk  Status Jobs      Serial N

umber

  Drive Spin Down       Spin Down Delay

-------------------------------------------------------------------------------

vd01  1348.8GB 35.8GB A   A      RAID5  4     0   64k  OFFL

  00c0ffdb1b3c0000a38e464d00000000 Disabled              0

-------------------------------------------------------------------------------

Success: Command completed successfully.

As I told that I restarted the Manage Controller A of the said P2000, STB02, through SMU Tools, but now its not coming back on network, no ping reply, and I am not able to access STB02 thorugh SMU. The "OK" & "CACHE" LEDs on Back Panel are On in Green. Help please.Rear PanelFront Panel

Regards,

Shahrukh

[SUBHAJIT KHANBARMAN_1]

@shahrukhntdc 

If you are sure that there was no hardware error with any of the 4 drives before replacement of PSU then you can try below command and it may help,

Login to Controller A CLI,

trust enable 

trust vdisk vd01

If you are not sure about health of all 4 drives then you can capture all events and look for any medium or hardware errors,

show events

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

Dear Subhajit Sir,

Thank you very much for the prompt support. There were only PSU related errors in the Events List on SMU before I replaced the PSU. I am sharing events with you. Please have a look on it. Unfortunatley, when I restarted "Management Controller" of the MSA P2000 through SMU Tools, it did not return back and I am no longer able to see the SMU displays. I am fetching events through CLI. Sir why My MSA P2000 is not coming back on Network? No ping reply no SMU access, I am sharing events list with you then I will do accordingly as you suggest. Once again thank you very much.

Regards

https://drive.google.com/drive/folders/1gLR603QhtQBhyNJY9kYLCFihQ4iTOt4h

[shahrukhntdc]

Subhajit Sir,

From show disks command the output is:

show disks all
Location Serial Number Vendor Rev How Used Type Size
Rate*(Gb/s) SP Status
---------------------------------------------------------------------------
1.3 JVWM3BAL HITACHI A510 LEFTOVR SAS
450.0GB 6.0 A Up
---------------------------------------------------------------------------
Info: * Rates may vary. This is normal behavior.

Success: Command completed successfully.

Only Hard Disk in Slot 3 is visible. What does it mean? Are the Disk Drives in Slot 1, 2 & 4 have become faulty? When SMU was accessible, in front panel view only drive in slot 3 was visible with LEFTOVR tag. But there is no hardware failure event in the events list as per my observation.

[SUBHAJIT KHANBARMAN_1]

@shahrukhntdc 

It looks like drives went down due to power issue.

LEFTOVER state doesn't mean drives are failed. It means Controller not able to recognize them as part of vdisk. So we need to tell controller that drives are fine and controller can trust them to start recover vdisk data. In order to achieve that we run below commands,

Login to Controller A CLI,

trust enable 

trust vdisk vd01

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

Dear Subhajit Sir!!

Thank you so so much. Trust command has given me hope. I ran the Trust command and it executed successfully. I am attaching the output of CLI for your review. But now I am facing another problem. Have a look:

1. Trust worked well and completed successfully. vd01 is back but its still OFFL as there is only one Disk out of 4 i.e. 1-3 which came online and now blinking in Green. The rest 3 are still with Orange LED On. Please look at the snap shot attached in Goole Drive Link. What u suggest? should i draw out them and then reseat them? Should I do this while MSA P2000 is in ON position and Host connected? What should be the sequence of Disks to be drawn out & re-seated back?
2. Fortunately I just unplugged & re-plugged the LAN cable in Controller-A ethernet port and it just turned On and now I can access the P2000 on SMU. I am attaching the snapshots of SMU for your review.
3. How will my P2000 will come to its normal state? please have a look on the snap shots and CLI_Trust output and guide me.  Also tell me how can I directly attach RARs or TXT files with my post? Sorry for the goggle drive link please.

https://drive.google.com/drive/folders/1gLR603QhtQBhyNJY9kYLCFihQ4iTOt4h 

Regards,

Shahrukh

[SUBHAJIT KHANBARMAN_1 · Accepted solution]

@shahrukhntdc 

I can see only 1 drive getting detected by Controller out of 4 drives, that's why data recovery not happening.

In order to recover data, controller need to read metadata from each drive in order to understand the vdisk datastructure and start recoving data as per the RAID defined.

If you have valid support contract with HPE then please log a support call with HPE Support. This type of issue not possible to resolve in public forum.

If you don't have support coverage,  then on a best effort basis you shutdown the MSA or power down the MSA. Then physically re-seat all three drives which were not getting detected. Then power on MSA and check from SMU and CLI if all 4 drives getting detected or not. if so then need to check they are in which state. If more than 1 drive not available or failed then data recovery not possible. If at least 3 drives visible and in LEFTOVER state or 4 drives visible and in LEFTOVER state then check vdisk state. If it shows offline then you can try trust command again and check. Please note multiple trust command not good as it's re-writing vdisk metadata.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

Dear Subhajit Sir,

Before proceeding towards powering down the MSA P2000, I want to ask something. Our SCADA System is in running condition since year 2012, non stop, without any rest. I am afraid that CMOS battery (if there is one like other servers/ workstations) of my MSA P2000 STB02 will now be very weak and If I Power down the P2000, I am afraid that it will wash out its settings/ configurations specially RAID settings as I have already experienced this while restroing an HP Server DL-320 G6, I powered down the server and when it came back online, the RAID settings were gone. So is there any way that without washing out the Raid or other Bios settings, I can draw out and re-seat the disks? If I 1st replace the old CMOS battery with the new One, while MSA P2000 is in ON position, will it work? Please guide me.

Regards,

Shahrukh

[SUBHAJIT KHANBARMAN_1]

@shahrukhntdc 

I am assuming that you don't have valid support contract with HPE so you want to continue here.

Please note Vdisk information along with RAID details saved inside each drive metadata. So if your drives are fine then only your vdisk or actual data will be fine in terms of vdisk structure or actual data.

Coming to battery part, each controller containes Coin battery and this battery provides backup power for the real-time (date/time) clock only.

The steps that I have suggested on a best effort basis only and there is no guarantee that data recovery will happen or not. It purely depends on health of the drive actually.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

Dear Subhajit SIr!

I did the same as you guided, Powered down the MSA, drawn out only those disks which were previously not detected, re-seated them exactly in the same place as they were, powered on the MSA, checked the SMU and found that now all 4 disks are detected, please look at the snapshots. Hurray!!! Thank you so much, but now CLI of controller A is not accessible. I tried to connect with Terminal Emulator but when I plug the CLI USB cable in my Laptop it says, USB device not recognized. Its the same laptop and same cable which I was using previously till todays morning. What happened? Do i need to Retasrt Manage Controller from SMU? Please note that when I powered down and powered Up the MSA, the USB cable remained plugged-in in MSA & Laptop, Did it cause the problem? Can I run Trust command from SMU also? please guide and also look at the SMU snaps after power Down/Up. Sorry for the google drive link.

https://drive.google.com/drive/folders/1gLR603QhtQBhyNJY9kYLCFihQ4iTOt4h 

Regards

Shahrukh

[SUBHAJIT KHANBARMAN_1]

@shahrukhntdc 

Good to see all drives detected now.

If SMU is accessible then CLI should also be accessible.

Try to ping Controller A IP address and see the response. If working then try to SSH to Controller A and run trust command as you did before.

If not then power down MSA again and remove all 4 drives few inch so that Controller should not detect them. Then power up MSA again. Then open CLI. Then insert one drive at a time online and check from CLI if it's detected or not. One by one you insert all 4 drives and check from CLI if they are getting detected or not and you still have CLI access or not. If for any drive you lost CLI access then you need to re-peat the steps again and next time don't insert that drive fully. You need to try and recover the vdisk without that drive.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

My Dear Subhjit Sir! You are my Hero!

I SSH the MSA and ran trust and now all 4 disks are blinking green and vdisk is in FTOL status. Do I have to do anything else for data recovery or it will do this at its own? everything looks green now. Will it come to its normal state automatically? As u can see there is no Controller B in my MSA STB02, but it saying it as Fault. Is it normal? Please have a llok at snapshots.

https://drive.google.com/drive/folders/1gLR603QhtQBhyNJY9kYLCFihQ4iTOt4h 

Regards

Shahrukh

[SUBHAJIT KHANBARMAN_1]

@shahrukhntdc 

Your vdisk in FTOL means all fine now in terms of data. There is nothing more to do for data recovery perspective. You check if volumes are accessible from hosts and you should have access to data.

Congrats for your hard effort to achieve this.

Now you convert your system as single controller system to fix the Controller error.

You can check the redundancy mode of the system by issuing the following CLI command.

# show redundancy-mode

To set the system as a single controller system use the following CLI command.

# set advanced-settings single-controller

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************

[shahrukhntdc]

Dear Subhajit Sir!

I am greatly thankful to you for your wonderful support although I reported lots and lots and lots of problems but you replied me with patience and guided me, Hats off to you. As I mentioned earlier that this MSA P2000 is a completely new Thing for me. In the past 8 years I have faced only this PSU failure problem and tried to replace the faulty with new from spares but I have learnt alot from you. This problem was indeed a blessing in disguise. Thank you Subhajit SIr! May you live long with good health and complete peace of mind. Salute to HP & HP Forum.

Regards,

Shahrukh