HPE Simplivity 380 Gen9 upgrade to 3.7.8 fails after firmware upgrade...

[TomiKarlsson]

HAs anyone had this problem? After upgraded Arbiter, web client plugin and first OVC to 3.7.8 of node in a 3 node cluster, after OVC reboot everythings fine, Federation show ok. Next rebooted hos and upgraded with firmware 790-000107-LSI-j.iso and SVTSP-2019_0315.04.iso, the OVC fails in startup showing that Simplivity HAL daemon and Simplivity OmniCube services FAIL to start??? Any ideas someone? Support Case is opened.

From $SVTLOG:

simplivity@mgmt@omnicube-ip2-111:~$ tail -f $SVTLOG
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.manager] StorageManager.cpp:213 Initializing Counters
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.manager] StorageManager.cpp:230 Initializing Counters done.
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [svtfs.app] thriftlogbridge.cpp:71 Thrift error: TSocket::open() connect() <Host: Port: 0>No such file or directory
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [hal.halobjectprovider] halobjectprovider.cpp:218 Thrift Transport Exception Unable to connect to HAL server:connect() failed: No such file or directory 1
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [hal.halobjectprovider] halobjectprovider.cpp:224 Tried 0 time(s) to connect to hal server and gather information, but was unsuccessful (-3)
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.hwdevicefactory] hwdevicefactory.cpp:76 Controller Type detected: 255
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:355 Message: Could not find any devices (from function _getBasicDeviceInfo)
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:311 Failed to get all devices for instance 0
2019-05-14T11:46:36.139Z FATAL 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:163 Message: Build devices failed! (from function start)
2019-05-14T11:46:36.139Z FATAL 0x7f25e3feb880 [:] [datapath.datapath] datapath.cpp:278 Error -1 starting Storage Manager

From /var/log/svt-upgrade.log:

simplivity@mgmt@omnicube-ip2-111:~$ tail -f /var/log/svt-upgrade.log
2019/05/14 11:46:33 INFO> HalStateDb.pm:158 HalStateDb::_loadDb - Reading HalStateDb file /var/svtfs/svt-hal/0/state/svt-hal-state-db.yaml
2019/05/14 11:46:33 INFO> SSD.pm:673 SSD::_getSmartAttributeValue - Field [173] found for smart data: [Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 42] [wear value: 100]
2019/05/14 11:46:33 INFO> StorCLI.pm:732 Controller::Factory::LSI::StorCLI::buildPhysicalDrives - Found 5 physical drive(s)
2019/05/14 11:46:33 INFO> HP.pm:111 System::HP::overrideCustomerVisibleModelString - System Model: N/A; Customer Visible Model: N/A
2019/05/14 11:46:33 INFO> Base.pm:170 System::Base::_setCustomerVisibleModelStrings - Model strings [customerVisibleModel: N/A] [model: N/A]
2019/05/14 11:46:34 INFO> Hardware.pm:238 InstallTypes::Hardware::waitForNiosActive - TIA card is active
2019/05/14 11:46:34 INFO> Hardware.pm:203 InstallTypes::Hardware::waitForEEPROM - TIA card EEPROM read succeeded (0s)
2019/05/14 11:46:34 INFO> CheckVersions.pm:351 PreFlight::CheckVersions::componentsToUpgrade - No components need upgrading
2019/05/14 11:46:34 INFO> oaPreflight.pl:118 OA::Preflight::printNoop - Checks to make sure versions are correct for upgrade:.....noop
2019/05/14 11:46:34 INFO> Common.pm:157 Util::Common::Exit - /var/tmp/build/os/upgrade/install//oaPreflight.pl exiting, requested exit code is 3

[gustenar]

It appears to be a problem with a storage component. Could be a bad disk, or RAID controller issue. I would first make sure these components are in good status before trying to bring the filesystem back online again. 

[TomiKarlsson · Accepted solution]

Hi! Problem solved at OVC level with great help from Kevin C at support! First suggested solution was to power off the host for at least 5 minutes. But on the second cluster node also OmniStack upgrade went ok, but after firmware upgrade again in OVC;. Since svtfs wasn´t up and running Kevin found that in the directory  /var/svtfs/0/ was a file "nostart".

Solution: Delete the "nostart" file and then stop svtfs and start svtfs and immediately OVC was back in the Federation!

There was also another issue with firmware; at boot the HPE Simplivity 380 Gen9 got stuck at "Starting drivers"

Solution for me was to upgrade the ILO4 with latest version "ilo4_262.bin" manually. After that server rebooted nicely.

I hope this will be of help if anyone else comes up with this problem!

Cheers and thanks a lot to Kevin C for his support!