3par 7400 Drive cages degraded

[yassir2]

hi guys,

looking your support and advice for the degraded faulty issue and advice i replaced storeserv 7400 on exsisiting drives cages and i notice that cages degraded as below picture.

[support_s]

System recommended content:

1. HPE 3PAR StoreServ 7000 Storage Service and Upgrade Guide: Customer Edition (3PAR OS 3.3.1 and 3.2.2)

2. HP 3PAR StoreServ 7000/7450 Storage Cabling Configuration Guide A: 2 Node Systems with Small 2.5-inch Drive Enclosures (QR482-96452, August 2013)

If the above information is helpful, then please click on "Thumbs Up/Kudo" icon.

Thank you for being a HPE community member.

[sbhat09]

Hello @yassir2,

Looks like the firmware of the cage and the 3PAR OS ( on the 7400 system) are not compatible versions.

1. Please let me know the sequence of activities you have done before the cages went degraded.

2. Please provide the cli command outputs for below:
showversion -a -b
admithw -checkonly

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[yassir2]

Hi Sbht09, 

thanks much for reply and kindly find the below APYR:

3par7400 cli% showversion -a -b
Release version 3.1.3.334 (MU3)
Patches: None

Component Name Version
CLI Server 3.1.3.334 (MU3)
CLI Client 3.1.3.334
System Manager 3.1.3.334 (MU3)
Kernel 3.1.3.334 (MU3)
TPD Kernel Code 3.1.3.334 (MU3)
CIM Server 3.1.3.334 (MU3)
WSAPI Server 3.1.3.334 (MU3)
Console Menu 3.1.3.334 (MU3)
Event Manager 3.1.3.334 (MU3)
Internal Test Tools 3.1.3.334 (MU3)
LD Check Tools 3.1.3.334 (MU3)
Network Controller 3.1.3.334 (MU3)
Node Disk Scrubber 3.1.3.334 (MU3)
PD Scrubber 3.1.3.334 (MU3)
Per-Node Server 3.1.3.334 (MU3)
Persistent Repository 3.1.3.334 (MU3)
Powerfail Tools 3.1.3.334 (MU3)
Preserved Data Tools 3.1.3.334 (MU3)
Process Monitor 3.1.3.334 (MU3)
Rolling Upgrade Tools 3.1.3.334 (MU3)
Software Updater 3.1.3.334 (MU3)
TOC Server 3.1.3.334 (MU3)
VV Check Tools 3.1.3.334 (MU3)
SNMP Agent 1.7.0
SSH 5.5p1-6+squeeze1
Firmware Database 20151123 (MU3)
Drive Firmware 20151123 (MU3)
UEFI BIOS 14.05.23
MCU Firmware 4.23
Cage Firmware (DC1) 4.44
Cage Firmware (DC2) 2.64
Cage Firmware (DC3) 08
Cage Firmware (DC4) 2.64
Cage Firmware (DCS1) 3211
Cage Firmware (DCS2) 3211
Cage Firmware (DCN1) 3211
QLogic QLA4052C HBA Firmware 03.00.01.77
QLogic QLE8242 CNA Firmware 04.11.237
Emulex LP11002 HBA Firmware 02.82.x10
Emulex LPe12002 HBA Firmware 02.01.x14
Emulex LPe12004 HBA Firmware 02.01.x14
3PAR FC044X HBA Firmware 200A8
LSI 9201-16e HBA Firmware 17.11.00
LSI 9205-8e HBA Firmware 17.11.00
3par7400 cli% admithw -checkonly
Checking for drive table upgrade packages
Checking nodes...

Checking volumes...

Checking system LDs...

After an upgrade from 2.2.4 or earlier, if sufficient space is present,
admithw will automatically remove and recreate the preserved data LDs
with a larger set size to improve their availability.

Checking ports...

Checking state of disks...
The following disks are NOT in an acceptable state:
Id CagePos Type -State- -------------------------Detailed_State-------------------------
24 1:8:0 FC failed vacated,missing,invalid_media,smart_threshold_exceeded,servicing
68 4:4:0 FC failed vacated,missing,invalid_media,smart_threshold_exceeded,servicing
----------------------------------------------------------------------------------------
2 total
Volume creation may fail due to the following issues:

Component --------Description-------- Qty
PD Cages with unbalanced disks 2

Component -Identifier- ----------------------------Description----------------------------
PD Cage:1 PDs FC/10K/900GB unbalanced. Primary path: 8 on Node:0, 7 on Node:1
PD Cage:4 PDs FC/10K/900GB unbalanced. Primary path: 8 on Node:0, 7 on Node:1

Checking cabling...

Checking cage firmware...
6 cages require firmware upgrades.
This will require approximately 24 minutes.

Checking system health...
Checking alert
Checking cabling
Checking cage
Checking dar
Checking date
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking port
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Component ---------------Description--------------- Qty
Alert New alerts 14
Cage Cages not on current firmware 6
Network Too few working admin network connections 1
PD Cages with unbalanced disks 2
PD Magazines that are being serviced 2
PD PDs that are failed 2

admithw has completed.
3par7400 cli%

[sbhat09]

Hello @yassir2,

Please check this asymmetry of connection and fix it:

Component -Identifier- ----------------------------Description----------------------------
PD Cage:1 PDs FC/10K/900GB unbalanced. Primary path: 8 on Node:0, 7 on Node:1
PD Cage:4 PDs FC/10K/900GB unbalanced. Primary path: 8 on Node:0, 7 on Node:1

2. Then run 'admithw' command to update the cage firmware to match the 3PAR OS.

Let me know if you face anymore issues while doing it.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[yassir2]

hi Sbhat,

i follow your advice to upgrade the cages but still have an error, please see the below:

3par7400 cli% admithw

Checking for drive table upgrade packages

Checking nodes...

Checking volumes...

Checking system LDs...

Checking ports...

Checking state of disks...

Checking cabling...

Checking cage firmware...

6 cages require firmware upgrades.

This will require approximately 24 minutes.

Enter y to proceed with this action, or n to skip it.

y

2021-11-11 15:15:11 EET

Upgrading cage cage0 cpuA from rev 3211 to revision in file /opt/tpd/fw/cage/ebod/hp_eos_canister_local_combined_v3.2.0.17.gff.

Upgrading cage cage0 cpuB from rev 3211 to revision in file /opt/tpd/fw/cage/ebod/hp_eos_canister_local_combined_v3.2.0.17.gff.

Error:    Opcode         = SCCMD_DOCDB

   Node           = 253

   Tpd error code = TE_INVALID          -- Invalid input parameter

   Tpd error info = PCM VPD firmware download may have failed. Unable to verify success of download. Cage: 0, PS State: Normal, Version: 2D9457FC Filename: /opt/tpd/fw/cage/ebod/0984125_he764W_pcm_vpd_v5_41bef99c_slot2.gff, State Retries: 0, Verification Retries: 3, Download Retries: 3

[sbhat09]

Hello @yassir2,

I think you replaced the cages from another 3PAR which was running on a higher version of 3PAR OS. Now Your 3PAR 7400 is on a very old version of OS and those cages are already on a new version. 3PAR OS and cage firmware are not matching. With admithw command, cages can't update to the system's compatible version as they are already on the new version. To confirm this, please check what was the OS version of the system those cages previously belonged.

The solution is to upgrade your 3PAR 7400 to a supported version of OS. 3.2.2MU6 or later is recommended.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[Torsten.]

3par7400 cli% showversion -a -b
Release version 3.1.3.334 (MU3)

This is obsolete for almost 10 years now ...

[yassir2]

hi Sbhat,

just wanna to confirm if i can go directly to 3.2.2MU6 or should i move gradually , how can i check valid OS from current one because i tried to upload 3.2.1 which is unaccepted as per below pic., btw am going to download the version as u suggested.

thanks for prompet response.

[sbhat09]

Hello @yassir2,

v3.1.3 is very old. So, directly upgrading to v3.2.2 is not possible.

I will check if direct upgrade is possible through offline method. And for online upgrade, what are the intermediate versions it has to go through.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[yassir2]

hi Sbhat,

actually online update not available at this service processor version I think its has an old version as well, any way i tried to search which OS valid from where we r to unfied our system. 

[sbhat09]

Hello @yassir2 

I checked the details. Direct upgrade from version 3.1.3.334 (MU3) to 3.2.2MU6 is possible. First you need to upgrade the service processor to v4.4. Then upgrade 3PAR OS. As it is older version, the upgrade will be through CLI. It is quite complicated and critical. So, I recommend you get it done from HPE.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.

[yassir2]

thanks Sbhat, let me try as per your advice and let you know

[sbhat09 · Accepted solution]

Hi @yassir2,

Here is the email contact to schedule 3PAR OS upgrade from HPE engineers - 'gdc-sps-staff@hpe.com'.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of HPE.