HPE 3PAR Table of contents (TOC) quorum not reached

[LuisMadera]

After a power fail our HPE 3PAR stop working.

We only see this: 

showsysmgr -d

Table of contents (TOC) quorum not reached.

Reason for being unable to automatically start: Waiting for 60% of TOC disks to be present before starting (currently 46 of an estimated 94). Run "setsysmgr tocgen" to start despite this.

184 disks were available to be examined for TOCs. InServ cli% Waiting for 60% of TOC disks to be present before starting (current ly 46 of an estimated 94). Run "setsysmgr tocgen" to start despite this.

Did you see something similar that can help me fixing it?

Thanks

[veeyarvi]

Hi  LuisMadera

Could you confirm physically (by looking in to the LED status) all hardware is fine? Kindly check for any Node missing currently and also check whether all cages + PDs are normal.

You may also try a complete power cycle of the array Document - HP 3PAR StoreServ 7450 Storage - Powering On/Off the Storage System | HPE Support

Regards,
Veeyaarvi

[LuisMadera]

And this messages:

showsysmgr -d Table of contents (TOC) quorum not reached. Reason for being unable to automatically start: Waiting for 60% of TOC disks to be present before starting (currently 46 of an estimated 94). Run "setsysmgr tocgen" to start despite this. 184 disks were available to be examined for TOCs. InServ cli% Waiting for 60% of TOC disks to be present before starting (current ly 46 of an estimated 94). Run "setsysmgr tocgen" to start despite this.
Checking alert
Checking ao
Could not check ao: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking cabling
Could not check cabling: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking cage
Could not check cage: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking cert
Checking dar
Checking date
Checking file
Checking fs
Checking host
Could not check host: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking ld
Checking license
Checking network
Could not check network: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking node
Could not check node: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking pd
Could not check pd: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking pdch
Could not check pdch: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking port
Could not check port: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking qos
Checking rc
Checking snmp
Checking task
Could not check task: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Checking vlun
Could not check vlun: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking vv
Could not check vv: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Checking sp
Component --------------------------------------------------------Summary Description--------------------------------------------------------- Qty
Alert New alerts 14
Component Could not check ao 1
Component Could not check cabling 1
Component Could not check cage 1
cert Certificates that have expired 1
File Filesystems which are over 90% full 1
File Nodes with Dump or HBA core files 1
File sr_mnt not mounted 1
File Admin Volume is not mounted 1
fs Cannot check File Services 1
Component Could not check host 1
LD The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later. 1
Component Could not check network 1
Component Could not check node 1
Component Could not check pd 1
Component Could not check pdch 1
Component Could not check port 1
QoS Unable to check QoS 1
Component Could not check task 1
Component Could not check vlun 1
Component Could not check vv 1
--------------------------------------------------------------------------------------------------------------------------------------------------
21 total 34

Component ------Identifier------- --------------------------------------------------------------------------------Detailed Description---------------------------------------------------------------------------------
Alert sw_pr:0 The PR is currently getting data from the internal drive on node 1 not the admin volume. Previously recorded alerts will not be visible until the PR transitions to the admin volume.
Alert hw_cage:0,hw_cage_ifc:1 Cage 0, Interface Card 1 Failed (Interface Card Warning {0x403})
Alert sw_sysmgr The PR is not available on the admin volume. The system was unable to save status data for 1 tasks.
Alert sw_port:1:2:2 Port 1:2:2 Degraded (Target Qlength Above Threshold {0x5})
Alert hw_cage:5 Cage 5 Degraded (Loop Offline {0x10})
Alert hw_cage:5,hw_cage_ifc:0 Cage 5, Interface Card 0 Failed (Interface Card Not_Present {0x405})
Alert hw_cage:5,hw_cage_ifc:1 Cage 5, Interface Card 1 Failed (Interface Card Unrecoverable {0x404})
Alert hw_node:3,hw_ps:0 Node 3, Power Supply 0 Failed (AC Failed {0x2})
Alert hw_node:3,hw_ps:1 Node 3, Power Supply 1 Failed (AC Failed {0x2})
Alert hw_cage:5,hw_cage_ps:0 Cage 5, Power Supply 0 Failed (Power Supply Failed {0x402})
Alert hw_cage:5,hw_cage_ps:1 Cage 5, Power Supply 1 Failed (Power Supply Failed {0x402})
Alert hw_node:1 Node 1 No power detected on Cluster Link0 to Node 3 from Node 1
Alert sw_pr:0 The PR is currently getting data from the internal drive on node 1 not the admin volume. Previously recorded alerts will not be visible until the PR transitions to the admin volume.
Alert sw_sysmgr Table of contents (TOC) quorum not reached.
Component -- Could not check ao: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check cabling: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check cage: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
cert HP_3PAR 7400-1635815 Certificate for Service:unified-server* expired on Sep 2 20:21:33 2020 GMT
File node:1 Filesystem /dev/sda7 mounted on "/common" is over 90% full
File node:1 Dump or HBA core files found
File -- sr_mnt not mounted
File node:master Admin Volume is not mounted
fs -- Get cluster id - The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check host: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
LD -- The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check network: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check node: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check pd: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check pdch: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check port: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
QoS -- Unable to check QoS - The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check task: System manager is still initializing - try this command later.
Use 'showsysmgr' to see the current system state.
Component -- Could not check vlun: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
Component -- Could not check vv: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
34 total
InServ cli%

[LuisMadera]

3PAR cli% shownode -d
-------------------------------------------Nodes--------------------------------------------
Control Data Cache
Node --Name--- -State- Master InCluster -Service_LED- ---LED--- Mem(MB) Mem(MB) Available(%)
0         1635815-0   OK       Yes         Yes             Off           GreenBlnk       8192        8192           100
1 1635815-1             OK       Yes         Yes             Off           GreenBlnk       8192        8192           100
2 1635815-2            OK       Yes         Yes             Off           GreenBlnk       8192        8192           100
3 1635815-3         Failed      No          No        Unknown     Unknown           0              0               0

[veeyarvi]

Hi LuisMadera,

Sorry, but did you check the hardware as mentioned in my previous reply?

Regards,
Veeyaarvi

[LuisMadera]

Yes,

Here you can see the status:

Power Supply, disks and SP leds in green color.

Cage 5 led in yellow.

Node 0, Node 1 and node 3 leds in green blinking 

Node 2 led in amber and blue

[veeyarvi]

Hi LulsMadera,

One of the nodes is failed. As per the command output, it is the Node 3. But the LED indication suggests Node 2 - could you confirm the same. Node 3 should be in the top slot.

Anyway, a complete power cycle should bring the array back to Normal. And plan to get the failed node replaced as soon as possible since one more node failure/reboot will again cause an outage.

If you have HPE support contract, please contact the support. It is also still possible to get paid support if the array is out of contract.

Regards,

Veeyaarvi

[LuisMadera]

try to do a power cycle I got this:

getSysData: docli -C -D -I10.70.92.55 -V3.2.2.612 -S1635815 SERIALNO=1635815 indicates failure Error: The cache server is waiting for the system manager to finish reporting the system configuration and status. Please try again later.
SP0001635815

[veeyarvi]

Hi LuisMadera,

So that needs a detailed analysis. I suggest you to contact the HPE support.

Regards,

Veeyaarvi