P4300G2 not yet fully initialized, doesn't talk to CMC

[Ryan Nelson_3]

Had a P4300G2 that was happy and running in a group until today. I told it to start a manager, and it disconnected from the CMC. I eventually restarted ithe NSM via iLo, but now it won't talk to the CMC anymore, and at the console says "cannot log in to the console because the storage system has not yet fully initialized. Wait a minute  and try again." 

I've verified network connectivity, I can ping it and SSH to it. 

I waited 4 hours, then restarted it again .... same 'not initialized message'

I attempted to use cliq to issue a hydra service restart command, but running any command says it cannot reach 127.0.0.1:5989

Anybody have ideas? TIA

[Ryan Nelson_3 · Accepted solution]

For future folks searching. I solved the issue by ejecting the node from the cluster and re-imaging. 

[Mukesh2]

Hi,

Apologies for the delayed response.

I see that you were able to resolve the issue by reimaging, however a power drain of the node may have helped.

Steps to perform power drain :

·         Shutdown the node in queston

·         Remove the power cables and then all the drives halfway out ( all drives).

· Please ensure that drives are not taken out of their respective slots as it may cause data mismatch and corruption. They should still in their slot and just removed from the backplane connector.

·         Press and hold the power button for 30 to 45 seconds.

·         Replace the cables and restart the node.

·         Once it boots up check if there is any POST error.

·         Go ahead and shutdown the node once again and reseat the drives back firmly.

·         Restart the node and check if it resolved the issue.

·         Wait for the volumes to resync.

IMPORTANT NOTE : IF ITS A ONE NODE CLUSTER OR ANY CLUSTER WITH ANY NRO VOLUME, YOU NEED A DOWNTIME FOR THIS ACTIVITY AS THE VOLUME WILL BE UNAVAILABLE DURING THIS TIME.

Regards,
Mukesh

[miki777]

I have the exact same problem. Very strange issue. Will try power drain, and the re-imaging if it won't help.

Thanks for the info !

[Ryan Nelson_3]

Wont help? It very clearly solved the issue on my end. I believe it was the LH software erroneously reporting an error and not something in the hardware. For my issue, the reason is the error was not reported in by the SA controller at boot, nor in HPSSA. 

[miki777]

Re-imaging will probably help, as in your case ( good to know ). But will try the power drain first. This is the error -

[Ryan Nelson_3]

You know what? I confused this with another issue that I also re-imaged for.

I apologize, been a long day. Not the same thing. Carry on. 

[Jeremy-easyIT]

Did the powerdrain issue resolve it for you?

We've just had the same thing happen.

[George Vardikos]

Hello,

We have the same error message "Cannot log in to the console because the storage system has not yet fully initialized" too. This is on a newly formatted storevirtual 4330, so we tried reformatting it again without any success.

The installation completes normally and then we just receive that error after rebooting, typing "start" and then "Login".

[regi_p]

Hi

I had the same issue on my P4300 G2 nodes. Contacted HPE support. They have to clear some log files via SSH/iLO2 

(Normally it gets cleared automatically but in odd cases they have to do this & My system was running for last 7 years)

Power drain will help & Re-Imaging is a last thing you want to try i think  

Cheers

/R