3PAR 7200 does not Boot up

[555-denoh]

Hello Everyone.

This is 555-denoh.

An end user shut down a 3PAR 7200 in response to a power outage in their building.
The end user then powered on the 3PAR 7200. However, the 3PAR 7200 did not boot.

1. Insplore Log

TPD undergoing upgrade. Safe mode enabled

communications lock acquired

Creating InSplore data archive of cluster system1...

Safe mode enabled. Not running TOC related capability.

Gathering global CLI data...

   Starting showalert -d ......  Done

   Starting showbattery ..  Failed

   Starting showcage .  Failed

   Starting showcage -d ..  Failed

   Starting showcage -e ..  Failed

   Starting showcage -sfp -ddm .  Failed

   Starting showcage -sfp -d ..  Failed

   Starting showcpg ..  Failed

   Starting showcpg -r .  Failed

   Starting showcpg -sag ..  Failed

   Starting showcpg -sdg ..  Failed

   Starting showdate ..  Done

   Starting showeeprom ......  Done

   Starting showeventlog -d -debug -oneline ...  Done

   Starting showeventlog -d -fprefix events_nd ........  Done

   Starting showeventlog -d -fprefix events_al ..  Done

   Starting showfirmwaredb ..  Failed

showdate

Node Date

0    2026-01-13 16:34:45 JST (Asia/Tokyo)

showsys

Error: System manager is still initializing - try this command later.

Use 'showsysmgr' to see the current system state.

showversion -a -b

2. checkhealth Log

Checking alert
Checking cabling
Could not check cabling: Error: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking cage
Could not check cage: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking dar
Checking date
Checking ld
Checking license
Checking network
Could not check network: Error: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking node
Could not check node: Error: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking pd
Could not check pd: Error: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking port
Could not check port: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking rc
Checking snmp
Checking task
Could not check task: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking vlun
Could not check vlun: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking vv
Could not check vv: Error: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Component --------------------------------------------------Description--------------------------------------------------- Qty
Alert New alerts 3
Could cabling: 1
Could cabling 1
Could cage: 1
Could cage 1
LD System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state. 1
Could network: 1
Could network 1
Could node: 1
Could node 1
Could pd: 1
Could pd 1
Could port: 1
Could port 1
Could task: 1
Could task 1
Could vlun: 1
Could vlun 1
Could vv: 1
Could vv

3. Node LED status
NODE0
    Status LED Green ON
     UID LED OFF
     Health LED OFF
NODE1
    Status LED OFF
    UID LED Blue Flashing
    Health LED Amber flashing

4.showsysmgr Command result
showsysmgr
TOC quorum found for TOC 405840, but waiting for nodes 1 to boot up. If the nodes are offline, use 'setsysmgr tocgen 405840'.
TOCs found:
Valid TOC:   Name(system1) Sysid(31407) Gen(405840) Modified(2026-01-09 15:51:18 JST) Disk count(20/20)
20 disks were available to be examined for TOCs.

Next, to boot Node0 in single mode,
the end user attempted to access the MFG serial port and run "setsysmgr tocgen 405840", but was unable to access the MFG port and setsysmgr was not executed.

As a next step, the end user will attempt to replace Node1, scheduled for January 20th, Japan time.

So I'm wondering if replacing Node1 is the right action.
Since they are able to run Insplore and Checkhealth, I think they should SSH into the 3PAR7200 as 3paradm and run setsysmgr.
I think it would be best to return Node0 to the "sysmgr is running" state and then replace Node1.

Any comments would be much appreciated.

Thanks best regards.

[veeyarvi]

Hi 555-denoh,

I am not sure whether you were able to come out of this situation.
If a two-node array booted up with a single node, user need to confirm the TOC using the setsysmgr command so that the array can continue booting. In this scenario, replacing the failed node is not a recommended action since it cannot join the cluster (in fact the cluster is not yet formed). Need to wait for the array to boot up for a successful node rescue to the new node.

Another option is to use the old node disk (from the failed node) in the new node and set the parameters (like node ID and system ID) in the eeprom from serial port,

Regads,

Veeyaarvi

[555-denoh · Accepted solution]

Veeyaarvi san.
This is 555-Denoh.
Thank you for your support.

The situation has been resolved.
1. Check the status
cli% > showsysmgr
TOC quorum found for TOC 405840, but waiting for node 1 to boot up.
If the nodes are offline, use 'setsysmgr tocgen 405840'.
TOCs found:
Valid TOC: Name(system1) Sysid(31407) Gen(405840) Modified(2026-01-09 15:51:18 JST) Disk count(20/20)
20 disks were available to be examined for TOCs.

2. The maintenance company engineer performed the following:
InServ cli% setsysmgr force_iderecovery
Are you sure you want to run setsysmgr force_iderecovery?
select q=quit y=yes n=no: y
InServ cli% showsysmgr
System is up and running from 2026-01-13 08:37:28 JST

I thought they would execute "setsysmgr tocgen 405840," but they immediately execited "setsysmgr force_iderecovery."

3. Replace Node 1. Verify that the cluster booted up successfully.

Thank you for the following steps. I'll keep them as a record of my knowledge.

Another option is to use the old node disk (from the failed node) in the new node and set the parameters (such as node ID and system ID) in the eeprom from the serial port.

Thanks ,Regads,

[Thaufique_Mod]

Hello @555-denoh,

Great job documenting the fix! Your detailed steps are invaluable and will make it easier for peers to resolve this. Thank you.