Resetting HPE Smart Storage Administration Configuration without Data Loss and Reenabling the Cache

[Danny_Need_Help]

Hello,

Short version: The option to reenable the controller cache module is throwing an error: The operation failed. An unknown error occurred during execution.

https://imgur.com/a/ay72QcL These are links to the screenshots.

The firmware is version 7.00

I am running SSA 3.40.3

I am now wondering if I shut down the server, remove all the disks. Boot into SSA from Intelligent Provisioning and clear the configuration, then shut down the server and put the drives back and boot it back up will the data be intact or lost since the configuration data is supposed to be stored on the drives as well. 

I know running the clear configuration with the disks installed will result in data loss but I have no way to test this without risking significant data loss but our server cannot perform without the write cache being available and we need this working in the next few hours!

Long Version:

The battery in my DL 380 Gen9 went bad and while waiting on another battery, we decided to risk it and enable write-back on the storage controller cache. The whole unit was on a UPS and we haven't had any crashes so we thought while there was some risk it was minimal.

Long story short, the replacement battery came in. We go to shut down the server and after the disks were idle and it said shutting down it froze and had a driver power state failure bluescreen that it wouldn't recover from on Server 2019. it threw an interrupt so we were forced to reset the system.

Before that we actually installed the new battery by removing the fans for 30 seconds while it was still running. All was well and we got it installed but on reboot it said that the cache was lost. It booted successfully and said the cache was Okay so we changed the disable cache on battery not present/charged on and it immediately permanently disabled the cache even though it says the new battery is fine.

The option to re-enable the cache fails and I have tried in boot environment as well as in Windows to re-enable it. I have tried both with and without the battery installed and it just keeps throwing the same error. Any advice?!

[Torsten.]

You really replaced the battery while the server was running?
This may damage the hardware.

[Danny_Need_Help]

Yes, I did while it was running but I don't see why this would cause an issue. Regardless of whether this could or couldn't. It was detecting the battery fine AND still is. Also the cache was still working fine after the fact but warned that the data was lost due to system being reset because I had to reset it after the driver power state failure on shutdown. The driver power state shutdown issue has happened a few times since coming to Server 2019 so I don't think this has anything to do with me installing the battery.

It was only when I renabled Disable Write Cache when battery depleted or not present that it permanently disabled the cache and did so immediately.

I updated to 4.15.6.0 for Smart Storage Administrator and tried Recover Cache Module: 

The operation failed. An unknown error occurred during execution.

The Controller Cache Details:

Cache Status Permanently Disabled Cache Status Details Cache disabled; backup to flash memory failed Cache

Disable Reason Permanent disable condition. The posted write cache has been disabled because the backup operation failed in a flash-backed write cache module.

[Torsten.]

Replacing the battery is an offline operation (power off).

However, maybe the flash memory inside the cache module is damaged, hence you need to replace the cache modul (depending on the controller model this may mean the complete controller).

[Danny_Need_Help]

The behavior I am experiencing is described here:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04947297&docLocale=en_US

The only thing is the firmware was already on 7.00 before this occurred so this issue should have been mitigated. I feel like this is more of a firmware glitch.

I know clearing the configuration with the disks installed will result in the disk contents being lost.

If I shut down the server, remove the disks and boot into SSA from Intelligent Provisioning and wipe the configuration and it fixes the cache issue. If I shut it back down and put back in the disks. Will it restore the drive configuration to the controller because I know this is the behavior if I just REPLACE the controller. Do you have any insight on this? 

P.S. Thank you for letting me know the battery was supposed to be an offline operation... I do it on laptops all the time so didn't even think it would be an issue.

[Torsten.]

According to the advisory:

RESOLUTION

To resolve this issue after it has occurred, replace the HPE Smart Array controller if the Smart Array Comptroller cache is embedded on the Smart Array controller board (not removable).