all disk degraded

[xcientist]

"After a power outage, everything is working but all disks show as 'degraded'. How can I fix this and what should I do? I am using an HP3PAR 7200c model."

7200c cli% showpd
                             ----Size(MB)----- ----Ports----
Id CagePos Type RPM State       Total     Free A      B      Capacity(GB)
 0 0:0:0   FC    10 degraded   559104   169984 1:0:1* -----           600
 1 0:1:0   FC    10 degraded   559104   190464 1:0:1* -----           600
 2 0:2:0   FC    10 degraded   559104   180224 1:0:1* -----           600
 3 0:3:0   FC    10 degraded   559104   528384 1:0:1* -----           600
 4 0:4:0   FC    10 degraded   559104    12288 1:0:1* -----           600
 5 0:5:0   FC    10 degraded   559104   528384 1:0:1* -----           600
 6 0:6:0   FC    10 degraded   559104   177152 1:0:1* -----           600
 7 0:7:0   FC    10 degraded   559104   188416 1:0:1* -----           600
 8 0:8:0   FC    10 degraded   559104   186368 1:0:1* -----           600
 9 0:9:0   FC    10 degraded   559104   153600 1:0:1* -----           600
10 0:10:0  FC    10 degraded   559104   186368 1:0:1* -----           600
11 0:11:0  FC    10 degraded   559104   529408 1:0:1* -----           600
12 0:12:0  FC    10 degraded   559104   184320 1:0:1* -----           600
13 0:13:0  FC    10 degraded   559104   156672 1:0:1* -----           600
14 0:14:0  FC    10 degraded   559104   185344 1:0:1* -----           600
15 0:15:0  FC    10 degraded   559104   189440 1:0:1* -----           600
16 0:16:0  FC    10 degraded   559104   187392 1:0:1* -----           600
17 0:17:0  FC    10 degraded   559104   158720 1:0:1* -----           600
18 0:18:0  FC    10 degraded   559104   180224 1:0:1* -----           600
19 0:19:0  FC    10 degraded   559104   529408 1:0:1* -----           600
20 0:20:0  SSD  150 degraded   456704   342016 1:0:1* -----           480
21 0:21:0  SSD  150 degraded   456704   342016 1:0:1* -----           480
22 0:22:0  SSD  150 degraded   456704   343040 1:0:1* -----           480
23 0:23:0  SSD  150 degraded   456704   343040 1:0:1* -----           480
24 1:0:0   FC    10 degraded   559104   178176 1:0:2* -----           600
25 1:1:0   FC    10 degraded   559104   174080 1:0:2* -----           600
26 1:2:0   FC    10 degraded   559104   175104 1:0:2* -----           600
28 1:4:0   FC    10 degraded   559104   177152 1:0:2* -----           600
29 1:5:0   FC    10 degraded   559104   175104 1:0:2* -----           600
30 1:6:0   FC    10 degraded   559104   176128 1:0:2* -----           600
31 1:7:0   FC    10 degraded   559104   174080 1:0:2* -----           600
32 1:8:0   FC    10 degraded   559104   177152 1:0:2* -----           600
33 1:9:0   FC    10 degraded   559104   176128 1:0:2* -----           600
34 1:10:0  FC    10 degraded   559104   175104 1:0:2* -----           600
35 1:11:0  FC    10 degraded   559104   179200 1:0:2* -----           600
36 1:12:0  FC    10 degraded   559104   177152 1:0:2* -----           600
37 1:13:0  FC    10 degraded   559104   179200 1:0:2* -----           600
38 1:14:0  FC    10 degraded   559104   178176 1:0:2* -----           600
39 1:15:0  FC    10 degraded   559104   176128 1:0:2* -----           600
40 1:16:0  FC    10 degraded   559104   179200 1:0:2* -----           600
41 1:17:0  FC    10 degraded   559104   171008 1:0:2* -----           600
42 1:18:0  FC    10 degraded   559104   176128 1:0:2* -----           600
43 1:19:0  FC    10 degraded   559104   169984 1:0:2* -----           600
44 1:20:0  FC    10 degraded   559104   178176 1:0:2* -----           600
45 1:21:0  FC    10 degraded   559104   179200 1:0:2* -----           600
46 1:22:0  FC    10 degraded   559104   182272 1:0:2* -----           600
47 1:23:0  FC    10 degraded   559104   179200 1:0:2* -----           600
48 1:3:0   FC    10 degraded   559104   185344 1:0:2* -----           600
-------------------------------------------------------------------------
48 total                     26427392 10421248

[xcientist]

7200c cli% checkhealth -svc -detail
Checking alert
Checking ao
Checking cabling
Checking cage
Checking cert
Checking dar
Checking date
Checking file
Checking fs
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking pdch
Checking port
Checking qos
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Checking sp
Component ------------------------Summary Description------------------------- Qty
Alert     New alerts                                                             2
Cabling   Missing node for cabling check                                         1
Cage      Cages missing B loop                                                   2
Cage      Cages not on current firmware                                          2
File      sr_mnt not mounted                                                     1
host      Hosts not seen by multiple nodes                                       2
LD        LDs in write through mode                                             19
LD        LDs without backup                                                    22
LD        Failed or degraded LDs                                                 1
LD        Preserved data storage space does not equal total node's Data memory   1
License   Licenses which have expired                                            1
Network   Too few working admin network connections                              1
Node      Number of node environmental factors out of tolerance                  4
Node      Nodes that are not online                                              1
PD        Unbalanced PD types in Cages                                           3
PD        Magazines with failed servicemag operations                            4
PD        Magazines with active servicemag operations in progress                1
PD        PDs that are degraded                                                 48
QoS       Unable to check QoS                                                    1
vlun      Hosts not connected to a port                                          2
----------------------------------------------------------------------------------
       20 total                                                                119

Component ------Identifier------ ---------------------------------Detailed Description---------------------------------
Alert     sw_pd:11,sw_ch:530     Could not find a suitable spare chunklet for 11:530.
Alert     sw_pd:5,sw_ch          Physical Disk 5 is degraded due to repeated failures relocating chunklets:  pd5 ch538.
Cabling   node0                  No cabling data for node0. Check status of node0
Cage      cage:0                 Missing B loop
Cage      cage:0                 Firmware is not current
Cage      cage:1                 Missing B loop
Cage      cage:1                 Firmware is not current
File      --                     sr_mnt not mounted
host      hpDL360_ust            Host is not seen by multiple nodes, only seen from node 1
host      hpDL360_alt            Host is not seen by multiple nodes, only seen from node 1
LD        ld:admin.usr.0         LD is in write-through mode
LD        ld:admin.usr.0         LD does not have a backup
LD        ld:admin.usr.1         LD is in write-through mode
LD        ld:admin.usr.1         LD does not have a backup
LD        ld:.srdata.usr.0       LD is in write-through mode
LD        ld:.srdata.usr.0       LD does not have a backup
LD        ld:.srdata.usr.1       LD is in write-through mode
LD        ld:.srdata.usr.1       LD does not have a backup
LD        ld:log0.0              Detailed State:  Not Started
LD        ld:pdsld0.0            LD does not have a backup
LD        ld:pdsld0.1            LD does not have a backup
LD        ld:pdsld0.2            LD does not have a backup
LD        ld:HYPERV_VM.usr.0     LD is in write-through mode
LD        ld:HYPERV_VM.usr.0     LD does not have a backup
LD        ld:HYPERV_VM.usr.1     LD is in write-through mode
LD        ld:HYPERV_VM.usr.1     LD does not have a backup
LD        ld:HYPERV_VM.usr.2     LD is in write-through mode
LD        ld:HYPERV_VM.usr.2     LD does not have a backup
LD        ld:HYPERV_VM.usr.3     LD is in write-through mode
LD        ld:HYPERV_VM.usr.3     LD does not have a backup
LD        ld:HYPERV_VM.usr.4     LD is in write-through mode
LD        ld:HYPERV_VM.usr.4     LD does not have a backup
LD        ld:HYPERV_VM.usr.5     LD is in write-through mode
LD        ld:HYPERV_VM.usr.5     LD does not have a backup
LD        ld:tp-1-sa-1.0         LD is in write-through mode
LD        ld:tp-1-sa-1.0         LD does not have a backup
LD        ld:tp-1-sa-1.1         LD is in write-through mode
LD        ld:tp-1-sa-1.1         LD does not have a backup
LD        ld:tp-1-sd-0.0         LD is in write-through mode
LD        ld:tp-1-sd-0.0         LD does not have a backup
LD        ld:tp-1-sd-0.1         LD is in write-through mode
LD        ld:tp-1-sd-0.1         LD does not have a backup
LD        ld:HYPERV_QUORUM.usr.0 LD is in write-through mode
LD        ld:HYPERV_QUORUM.usr.0 LD does not have a backup
LD        ld:HYPERV_VM.usr.6     LD is in write-through mode
LD        ld:HYPERV_VM.usr.6     LD does not have a backup
LD        ld:HYPERV_VM.usr.7     LD is in write-through mode
LD        ld:HYPERV_VM.usr.7     LD does not have a backup
LD        ld:HYPERV_VM.usr.8     LD is in write-through mode
LD        ld:HYPERV_VM.usr.8     LD does not have a backup
LD        ld:HYPERV_VM.usr.9     LD is in write-through mode
LD        ld:HYPERV_VM.usr.9     LD does not have a backup
LD        --                     Preserved data storage space is less than total node's Data memory
License   Peer Motion            License has expired
Network   --                     Nodes  have no admin network link detected
Node      node:0                 Environmental factor SBB Canister 0 CPU2 is Not Present
Node      node:0                 Environmental factor SBB Canister 0 is Not Present
Node      node:0                 Environmental factor SBB Canister 0 memory is Not Present
Node      node:0                 Environmental factor SBB Canister 0 near CPU is Not Present
Node      node:0                 Node is not online
PD        Cage:0                 PDs FC/10K/600GB unbalanced. Primary path: 20 on Node:1, 0 on Node:0
PD        Cage:0                 PDs SSD/150K/400GB|480GB unbalanced. Primary path: 4 on Node:1, 0 on Node:0
PD        cage:0,mag:0           Magazine has a failed servicemag operation
PD        cage:0,mag:3           Magazine has an active servicemag operation in progress
PD        cage:0,mag:5           Magazine has a failed servicemag operation
PD        cage:0,mag:11          Magazine has a failed servicemag operation
PD        cage:0,mag:19          Magazine has a failed servicemag operation
PD        Cage:1                 PDs FC/10K/600GB unbalanced. Primary path: 24 on Node:1, 0 on Node:0
PD        disk:0                 Degraded States: missing_B_port,servicing
PD        disk:1                 Degraded States: missing_B_port
PD        disk:2                 Degraded States: missing_B_port
PD        disk:3                 Degraded States: missing_B_port,servicing
PD        disk:4                 Degraded States: missing_B_port
PD        disk:5                 Degraded States: missing_B_port
PD        disk:6                 Degraded States: missing_B_port
PD        disk:7                 Degraded States: missing_B_port
PD        disk:8                 Degraded States: missing_B_port
PD        disk:9                 Degraded States: missing_B_port
PD        disk:10                Degraded States: missing_B_port
PD        disk:11                Degraded States: missing_B_port
PD        disk:12                Degraded States: missing_B_port
PD        disk:13                Degraded States: missing_B_port
PD        disk:14                Degraded States: missing_B_port
PD        disk:15                Degraded States: missing_B_port
PD        disk:16                Degraded States: missing_B_port
PD        disk:17                Degraded States: missing_B_port
PD        disk:18                Degraded States: missing_B_port
PD        disk:19                Degraded States: missing_B_port
PD        disk:20                Degraded States: missing_B_port
PD        disk:21                Degraded States: missing_B_port
PD        disk:22                Degraded States: missing_B_port
PD        disk:23                Degraded States: missing_B_port
PD        disk:24                Degraded States: missing_B_port
PD        disk:25                Degraded States: missing_B_port
PD        disk:26                Degraded States: missing_B_port
PD        disk:28                Degraded States: missing_B_port
PD        disk:29                Degraded States: missing_B_port
PD        disk:30                Degraded States: missing_B_port
PD        disk:31                Degraded States: missing_B_port
PD        disk:32                Degraded States: missing_B_port
PD        disk:33                Degraded States: missing_B_port
PD        disk:34                Degraded States: missing_B_port
PD        disk:35                Degraded States: missing_B_port
PD        disk:36                Degraded States: missing_B_port
PD        disk:37                Degraded States: missing_B_port
PD        disk:38                Degraded States: missing_B_port
PD        disk:39                Degraded States: missing_B_port
PD        disk:40                Degraded States: missing_B_port
PD        disk:41                Degraded States: missing_B_port
PD        disk:42                Degraded States: missing_B_port
PD        disk:43                Degraded States: missing_B_port
PD        disk:44                Degraded States: missing_B_port
PD        disk:45                Degraded States: missing_B_port
PD        disk:46                Degraded States: missing_B_port
PD        disk:47                Degraded States: missing_B_port
PD        disk:48                Degraded States: missing_B_port
QoS       --                     Unable to check QoS - node0: No route to host
vlun      host:hpDL360_ust       Host wwn:21000024FF5390CB is not connected to a port
vlun      host:hpDL360_alt       Host wwn:21000024FF2E09D1 is not connected to a port
-----------------------------------------------------------------------------------------------------------------------
      119 total
7200c cli%

[Cali]

One Node (Controller) is Down.

check with shownode

Cali

[xcientist]

7200c cli% shownode
                                                               Control    Data        Cache
Node --Name--- -State- Master InCluster -Service_LED ---LED--- Mem(MB) Mem(MB) Available(%)
   1 1689247-1 OK      Yes    Yes       Off          GreenBlnk   16384    4096          100
7200c cli%

This looks normal, there is no problem

[Cali]

Sorry, no, Node 0 is not here. 

Should look like this:

Node --Name--- -State- Master InCluster -Service_LED ---LED--- Mem(MB) Mem(MB) Available(%)
0 1618123-0 OK Yes Yes Off GreenBlnk 8192 4096 100
1 1618123-1 OK No Yes Off GreenBlnk 8192 4096 100

Most of the other Errors are follow up on the missing Node.

So next is, to check the Controller LED and the State by using the Serial Cable.

Failed to see controller 1 HP 3PAR 7400 - Hewlett Packard Enterprise Community (hpe.com)

Cali

[xcientist]

Thank you very much for your response. When I checked, I couldn't see any lights on the screen, and when I looked at the back of the device, there were no lights on either. I removed the unit and will be reattaching it, but I couldn't pull out the metal rod, it seems to be stuck. How can I check if it will turn on by itself when I plug it back in?

[xcientist]

Today, I performed a "plug and unplug" operation and the blue and orange lights are on. What does this mean? When I did "shownode", I still couldn't view it. I am attaching a video.  https://s12.gifyu.com/images/gif34010f0133eafcef.gif 

What other methods should I try? The device is old and there is no support contract. Could the SSD disk inside node0 be dead? If I replace it, will it boot up?

[AnilKT]

Hi xcientist,

The Amber/Orange color LED flashing on the controller node indicates that there may be a physical issue with the node.

Please contact HPE tech Support, they will help to fix this issue.

Hope this helps.!

Regards
Anil