3par 8400 Node 0 Degraded, All Physical Disk Degraded

[agang]

HI,ALL
    My 3PAR 8400 storage is malfunctioning, I checked and found that Node 1 is failed, but why is Node 0 and disk is Degraded? What to do about it? Thank you all for your suggestions!

Time : 2022-11-03 18:04:44 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:44 CST
Severity : Degraded
Type : Component state change
Message : Node 0 Degraded (Link Error {0x2})

Time : 2022-11-03 18:04:44 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:44 CST
Severity : Minor
Type : FPGA Status
Message : Removed Node: 1

Time : 2022-11-03 18:04:44 CST
Severity : Minor
Type : FPGA Status
Message : HW-Power Subsystem Summary Failure: NSR0=41 SLP_S5_L=0 NSR0=41

Time : 2022-11-03 18:04:45 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:45 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:45 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:46 CST
Severity : Minor
Type : FPGA Status
Message : HW-Power Subsystem Summary Failure: NSR0=41 SLP_S5_L=0 NSR0=41

Time : 2022-11-03 18:04:47 CST
Severity : Degraded
Type : Eagle link error
Message : Fatal link error between member nodes 0 to 1 posted by node 0 Link 3 config reg [0x00000003] [EN_LTSM RT_COMPLEX] Link 3 status reg [0x40000004] [SND_TO DATA_TX_WD].

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:0:0, Physical Disk 0 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:1:0, Physical Disk 1 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:2:0, Physical Disk 2 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:3:0, Physical Disk 3 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:4:0, Physical Disk 4 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:5:0, Physical Disk 5 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:6:0, Physical Disk 6 Degraded (Missing B Port {0x8a})

Time : 2022-11-03 18:04:48 CST
Severity : Degraded
Type : Component state change
Message : Magazine 0:7:0, Physical Disk 7 Degraded (Missing B Port {0x8a})

[sbhat09]

Hello @agang,

Degraded is the state that indicates that the hardware or software component can't function at it's full capacity. It is not a failure but needs attention.
These alerts are expected during node failure scenario. Cages/Drive lost the redundant path via node1. And node0 can't see it's pair node active for failover. So, they are showing degraded error.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[agang]

Srinivas Bhat，

  Thank you for your reply! But the problem is, I find that the access to storage is very slow now. In the worst case, the delay exceeds 100ms. Is this normal in the 3APR 8000 storage system?

[sbhat09]

Hello @agang,

Yes. During node failure scenario, the system is expected to be slower. But how slow is subjective. It depends on the amount of iops the hosts are doing, how much read and write requests are in queue.

In comparison with a fully functional system, half of the host paths are not accessible, half of the cache and CPUs are not available, also only half of the SAS cables are available to execute read/writes on disks. So, during the peak period, due to the longer queue length, the system may be a lot slower.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[sbhat09]

Hello @agang,

It is not advisable to continue with single node. I recommend you to replace the failed node ASAP. 

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.