HP 8400 Upgrade to 3.3.2.135-GA failure recovery did not work cleanly

[MonkeyBed]

Background:  We have a 3par 8400 StoreServ system. In the middle of an online upgrade from 3.3.2.135-GA one of the hosts on the SAN lost a single path connection causing the firmware upgrade to halt and eventually timeout fail.  Further more, the resume/abort/cancel prompt on the service processor did not work as expected as the web GUI window showed "The HPE 3PAR OS update from {0} to {1} was stopped.  No data available in table" and did not function.  Regardless, we were forced to resume the upgrade from the CLI legacy approach (using upgrade***) and we were able to resume the upgrade to OS-3.3.2.135-GA, completing the reboots and finishing the upgrade.  Further more, we were able to revert back successfully to 3.3.1.648 and go through staging and upgrading directly to OS-3.3.2.159-MU1.  However, we still have an unusual defect left over on our StoreServ from 3.3.2.135-GA that we can not get rid of....

Problem:  On our service processor it still says that update "3.3.2.135-GA" is in status "Ready" on the attached StoreServ as it's in the internal staging directory.  We've deleted the package from the service processor and it makes no difference to the managed StoreServ.  Anything we do to try to request that the 3.3.2.135-GA update is unstaged from the staged directory whether from the service console or from the CLI it results in errors about an errored/missing property file associated to that package.  From the service processor you receive the following task error:

Jun 30, 2022 2:42:31 AM EDT Created
Jun 30, 2022 2:42:31 AM EDT Start user operation
Jun 30, 2022 2:42:31 AM EDT Removing staged package OS-3.3.2.135-GA from StoreServ
Jun 30, 2022 2:42:32 AM EDT Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.
Verify the package is valid and was transferred correctly..

From the CLI attempting an unstage you receive:

Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.
Verify the package is valid and was transferred correctly..

In the log file it has:

2022-06-30 02:43:59 EDT The specified package does not contain a property file.
2022-06-30 02:43:59 EDT Verify the package is valid and was transferred correctly.
2022-06-30 02:43:59 EDT 2022-06-30 02:43:59 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.
Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.
Verify the package is valid and was transferred correctly.

How do we force the elimination of this staged kit package and resolve this error?

Thanks!

Jeremy

[MonkeyBed]

One last thing, the upgrade kit package does not show when listing staged upgrade packages.  However, if you policy check it does show it is there:

>policycheck output
Available space per node:
0: 44050.5MB 1: 44332.0MB 2: 44115.4MB 3: 41922.3MB

Kit_type Current Maximum Reduced
OS 1 2 0
Patch 0 5 0
Ukit 0 2 0
Tkit 0 5 0

>checksignature output

OS-3.3.2.135-GA
Checking upgrade kit for a valid signature...
The specified kit (OS-3.3.2.135-GA) does not contain a valid tpdpackage.
Verify the package is valid and was transferred correctly.

[sbhat09]

Hello @MonkeyBed,

Have you tried rebooting the service processor after removing the package?

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.

[MonkeyBed]

We rebooted the service processor and attempted to "X" this update from the StoreServ and same task error.

Jun 30, 2022 11:31:00 AM EDT Created
Jun 30, 2022 11:31:00 AM EDT Start user operation
Jun 30, 2022 11:31:00 AM EDT Removing staged package OS-3.3.2.135-GA from StoreServ
Jun 30, 2022 11:31:02 AM EDT Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.
Verify the package is valid and was transferred correctly..

[sbhat09]

Hello @MonkeyBed,

To investigate the issue further, it required log analysis and a live troubleshooting session. I think you will have to report this to HPE support or the HPE 3PAR deployment services team.

Also, you can consider to download all the logs and then redeploy the VSP.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.

[MonkeyBed]

Hello Srinivas,

    As per the policyoutput and signature check listed above the issue to us appears as the leftover KIT deployed to each of the nodes' staging directory that from what I understand is a combination between the package file and the property file.  I've seen it as well throughout the communication files between the SP working with managing the configuration of the nodes. Is there a command to directly access the staging directory on each node?  Is it accessible via an undocumented command (as -unstage can't get past this property file issue)?  Maybe is there a command to clear a node's staging directory, as the policycheck shows that is the only thing remaining on each node?  Is the staging directory accessible by utilitizing the management serial connection or maybe via SFTP using one of my accounts?  In the past there was an "spdood" login to get to a real node shell, can I allow the spdood account to get to a real SSH prompt to get to the load/staging directory on each node?

Thanks!

-Jeremy

Here is an communication example via the upgrade logs to OS-3.3.2.159-MU1 showing the -getlog output with the TCL/TPD interpreter not able to do anything with the package file due to the removed or errored property file:

2022-06-29 21:54:04.860-0400 INFO c.t.i.d.u.a.TPDArrayUpgradeSocketReal - Send TCL command: {upgradesys_getlog}
2022-06-29 21:54:04.908-0400 INFO c.t.i.d.u.a.TPDCliConnectorUpgrades - completed getlog
2022-06-29 21:54:04.909-0400 INFO c.t.i.d.u.m.TPDArrayUpgradeMonitor - 0 {{{2022-06-29 21:33:56 EDT Update to OS-3.3.2.159-MU1 started by user 3parservice} {2022-06-29 21:33:58 EDT The specified package does not contain a property file.} {2022-06-29 21:33:58 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:33:58 EDT 2022-06-29 21:33:58 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:35:11 EDT The specified package does not contain a property file.} {2022-06-29 21:35:11 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:35:11 EDT 2022-06-29 21:35:11 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:35:37 EDT The specified package does not contain a property file.} {2022-06-29 21:35:37 EDT The specified package does not contain a property file.} {2022-06-29 21:35:37 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:35:37 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:35:37 EDT 2022-06-29 21:35:37 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:35:37 EDT 2022-06-29 21:35:37 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:36:21 EDT The specified package does not contain a property file.} {2022-06-29 21:36:21 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:36:21 EDT 2022-06-29 21:36:21 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:36:21 EDT Required space: 8194.4MB, Available space, by node:} {2022-06-29 21:36:21 EDT 0: 41737.2MB, 1: 41808.6MB, 2: 41595.2MB, 3: 42149.9MB} {2022-06-29 21:36:21 EDT Storage freespace check Passed.} {2022-06-29 21:36:37 EDT Package validation started by user 3parservice for package OS-3.3.2.159-MU1.} {2022-06-29 21:37:04 EDT Package validate for user 3parservice returned a good signature response for package OS-3.3.2.159-MU1.} {2022-06-29 21:37:04 EDT Package validation for user 3parservice, successfully completed for package OS-3.3.2.159-MU1.} {2022-06-29 21:37:04 EDT Extracting...} {2022-06-29 21:37:14 EDT The specified package does not contain a property file.} {2022-06-29 21:37:14 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:37:14 EDT 2022-06-29 21:37:14 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:37:14 EDT Required space: 0MB, Available space, by node:} {2022-06-29 21:37:14 EDT 0: 41737.2MB, 1: 41808.5MB, 2: 41595.3MB, 3: 37814.7MB} {2022-06-29 21:37:14 EDT Storage freespace check Passed.} {2022-06-29 21:37:22 EDT Applying new package...} {2022-06-29 21:37:22 EDT Calculating checksums...} {2022-06-29 21:37:35 EDT Performing pre-update checks...} {2022-06-29 21:37:48 EDT Distributing package to all nodes...} {2022-06-29 21:38:02 EDT Running preupdate scripts...} {2022-06-29 21:38:03 EDT Installing base filesystem.} {2022-06-29 21:38:17 EDT Installing to partition: 2.} {2022-06-29 21:38:21 EDT Installing software...} {2022-06-29 21:40:15 EDT Shutting down sysmgr.} {2022-06-29 21:40:15 EDT Shutting down sysmgr.} {2022-06-29 21:40:15 EDT System manager Table of Contents (TOC) conversion requested from current version 193 to version 195.} {2022-06-29 21:40:15 EDT Requesting system shutdown. Table of Contents (TOC) Version set to 195} {2022-06-29 21:40:18 EDT Finished shutting down sysmgr.} {2022-06-29 21:40:19 EDT Internal system disk of node 0 updated successfully.} {2022-06-29 21:40:19 EDT Internal system disk of node 1 updated successfully.} {2022-06-29 21:40:19 EDT Internal system disk of node 2 updated successfully.} {2022-06-29 21:40:19 EDT Internal system disk of node 3 updated successfully.} {2022-06-29 21:40:20 EDT Update complete.} {2022-06-29 21:40:20 EDT Update successful.} {2022-06-29 21:53:34 EDT The specified package does not contain a property file.} {2022-06-29 21:53:34 EDT Verify the package is valid and was transferred correctly.} {2022-06-29 21:53:34 EDT 2022-06-29 21:53:34 EDT;TPDSTRUCTMSG;CSUT;ERR_PROPERTYREADFAIL;OS-3.3.2.135-GA;The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly. Failed to read the property file for kit OS-3.3.2.135-GA, The specified package does not contain a property file.} {Verify the package is valid and was transferred correctly..} {2022-06-29 21:53:46 EDT Update finished.} {}} 0} 

[sbhat09]

Hello @MonkeyBed,

spdood user is not available anymore after v5.0.x. Only available users are admin (for SP administrator), hpepartner (for HPE authorised server providers) and hpesupport (for HPE support personnel). The root access is available only for hpesupport user. So, I think contacting HPE support or 3PAR deployment services is the best option. If not, you can try to redeploy the VSP.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.