sysmgr down, undefinite wait for node 1 to boot up

[mihailos]

Dear all

Im having trouble with starting our 3par (7200c) after power loss.

It has 2 nodes, one of whitch seems to start (node 0), but no blinking first led, while the other has blinking blue led and does not seem to start (no lights no network switch etc)

Most CLI commands i‚ve tried produce "System manager is still initializing - try this command later." (Its been over 10 hours)

showsysmgr returns "Table of contents (TOC) quorum found for generation 203227, but waiting for nodes 1 to boot up. If the nodes are offline, use "setsysmgr tocgen 203227""

setsysmgr tocgen 203227 just returns me to the same position a while later.

I am also unable to perform shutdown of sys or node and recieve message "Could not read domain infromation", of whitch i found little.

Is there a way to skip waiting for node 1 to boot and start up with 1 node, so i can atleast backup, or any other solution to this problem? I can't seem to find anyone else with this problem or a soultion.

Thanks in advance,
all best,
Mihailo

[Vikas Mishra]

Hello mihailos,

Based on your issue description, it seems you have experience a node 1 failure. Blue LED is often indicative of a failed node. You can attempt reseating the failed node and if it boots up, you are in luck. Else replacing the node should sure fix the problem. About booting array on single node, its likely you have some powerfail data left behind on the nodes, preventing sysmgr from initializing. Since sysmgr tocgen couldn't bring it up on the last generation of TOC, you are probably going to need junking this powerfail data (user data). Now, Since this could potentially cause some data loss, I would sincerely hope you call support. They can root into the array and force node 0 to boot up, ignoring the possibly small leftover data.

My apologies for not writng the command here based on impact it can have on forum visitors, not understanding the extent of damage it can cause.

Sincerely

Vikas Mishra

[mihailos]

Thank you for the response.

I've tried to reseat it but no luck. Even tried switching nodes places but the result is the same (Same node not booting). Was this a bad idea?

The HP support was called and will visit anyday now.

Anyhow, i'm curious, is this failover data located on the nodes? Could i access it trough the ssh or do i need to console connect to the working node?
I'd also appreciate the command in the private message, in case i need it in the future (as i was not able to locate it).

Thanks again,
mihailos

[Vikas Mishra]

If you don't see a pvt msg from me, ping me in private please.

[mihailos]

Thanks bunch,

I've recieived the command and unfortunately none of the sysmgr setsysmgr showsysmgr commands work including the one provided. (sysmgr initialazing... waiting for node 1 to boot up... error) Thats trough system management console at least, with our highest privilege user.

Do they have to be run on the working node's console? Or maybe it is an idication of a bigger failiure?

[Vikas Mishra]

You can try the commands on the 3par CLI console with 3paradm user priviledge.

Reiterating, Please reach out to HPE phone or chat support. I'm positive this can be brought up partially.

[sushilachuthan]

Hello

Could u please share the commands to bring the storage up with single node

i do have the same issue.  I am done proper shutdown the cluster, unfortunatily, one node is not

booting.

sushil

[sbhat09]

Hello @sushilachuthan,

I don't think there is a separate command for that.

Here is all the documents for 3PAR - https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c03793101

You can refer 'Setting Up a Factory-Integrated Storage System' and 'Verifying Setup and Powering ON the Storage System'

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.