HPE SimpliVity

Re: HPE Simplivity 380 Gen9 upgrade to 3.7.8 fails after firmware upgrade...

Go to solution

HPE Simplivity 380 Gen9 upgrade to 3.7.8 fails after firmware upgrade...

HAs anyone had this problem? After upgraded Arbiter, web client plugin and first OVC to 3.7.8 of node in a 3 node cluster, after OVC reboot everythings fine, Federation show ok. Next rebooted hos and upgraded with firmware 790-000107-LSI-j.iso and SVTSP-2019_0315.04.iso, the OVC fails in startup showing that Simplivity HAL daemon and Simplivity OmniCube services FAIL to start??? Any ideas someone? Support Case is opened.


simplivity@mgmt@omnicube-ip2-111:~$ tail -f $SVTLOG
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.manager] StorageManager.cpp:213 Initializing Counters
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.manager] StorageManager.cpp:230 Initializing Counters done.
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [svtfs.app] thriftlogbridge.cpp:71 Thrift error: TSocket::open() connect() <Host: Port: 0>No such file or directory
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [hal.halobjectprovider] halobjectprovider.cpp:218 Thrift Transport Exception Unable to connect to HAL server:connect() failed: No such file or directory 1
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [hal.halobjectprovider] halobjectprovider.cpp:224 Tried 0 time(s) to connect to hal server and gather information, but was unsuccessful (-3)
2019-05-14T11:46:36.139Z INFO 0x7f25e3feb880 [:] [storage.hwdevicefactory] hwdevicefactory.cpp:76 Controller Type detected: 255
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:355 Message: Could not find any devices (from function _getBasicDeviceInfo)
2019-05-14T11:46:36.139Z ERROR 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:311 Failed to get all devices for instance 0
2019-05-14T11:46:36.139Z FATAL 0x7f25e3feb880 [:] [storage.devicemanager] devicemanager.cpp:163 Message: Build devices failed! (from function start)
2019-05-14T11:46:36.139Z FATAL 0x7f25e3feb880 [:] [datapath.datapath] datapath.cpp:278 Error -1 starting Storage Manager


From /var/log/svt-upgrade.log:

simplivity@mgmt@omnicube-ip2-111:~$ tail -f /var/log/svt-upgrade.log
2019/05/14 11:46:33 INFO> HalStateDb.pm:158 HalStateDb::_loadDb - Reading HalStateDb file /var/svtfs/svt-hal/0/state/svt-hal-state-db.yaml
2019/05/14 11:46:33 INFO> SSD.pm:673 SSD::_getSmartAttributeValue - Field [173] found for smart data: [Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 42] [wear value: 100]
2019/05/14 11:46:33 INFO> StorCLI.pm:732 Controller::Factory::LSI::StorCLI::buildPhysicalDrives - Found 5 physical drive(s)
2019/05/14 11:46:33 INFO> HP.pm:111 System::HP::overrideCustomerVisibleModelString - System Model: N/A; Customer Visible Model: N/A
2019/05/14 11:46:33 INFO> Base.pm:170 System::Base::_setCustomerVisibleModelStrings - Model strings [customerVisibleModel: N/A] [model: N/A]
2019/05/14 11:46:34 INFO> Hardware.pm:238 InstallTypes::Hardware::waitForNiosActive - TIA card is active
2019/05/14 11:46:34 INFO> Hardware.pm:203 InstallTypes::Hardware::waitForEEPROM - TIA card EEPROM read succeeded (0s)
2019/05/14 11:46:34 INFO> CheckVersions.pm:351 PreFlight::CheckVersions::componentsToUpgrade - No components need upgrading
2019/05/14 11:46:34 INFO> oaPreflight.pl:118 OA::Preflight::printNoop - Checks to make sure versions are correct for upgrade:.....noop
2019/05/14 11:46:34 INFO> Common.pm:157 Util::Common::Exit - /var/tmp/build/os/upgrade/install//oaPreflight.pl exiting, requested exit code is 3


Re: HPE Simplivity 380 Gen9 upgrade to 3.7.8 fails after firmware upgrade...

It appears to be a problem with a storage component. Could be a bad disk, or RAID controller issue. I would first make sure these components are in good status before trying to bring the filesystem back online again. 




I am an HPE employee
Accept or Kudo

Re: HPE Simplivity 380 Gen9 upgrade to 3.7.8 fails after firmware upgrade...

Hi! Problem solved at OVC level with great help from Kevin C at support! First suggested solution was to power off the host for at least 5 minutes. But on the second cluster node also OmniStack upgrade went ok, but after firmware upgrade again in OVC;. Since svtfs wasn´t up and running Kevin found that in the directory  /var/svtfs/0/ was a file "nostart".

Solution: Delete the "nostart" file and then stop svtfs and start svtfs and immediately OVC was back in the Federation!

There was also another issue with firmware; at boot the HPE Simplivity 380 Gen9 got stuck at "Starting drivers"

Solution for me was to upgrade the ILO4 with latest version "ilo4_262.bin" manually. After that server rebooted nicely.

I hope this will be of help if anyone else comes up with this problem!

Cheers and thanks a lot to Kevin C for his support!