Using Smart Update Firmware DVD 9.0 with multiple servers

[chuckk281]

Some good discussion on doing firmware upgrades on BladeSystem.

This was some info from Mark after he had used the new HPSUM DVD9.0.

**************************************************************************************************************

Hello,

I used the Smart Update Firmware DVD V9.0 today to update multiple servers and experienced something that may be helpful to others. It did not flash the UUID LED until about 15-20 minutes into the update. It did complete successfully, but the User's Guide indicates it will flash the UUID while performing the update.

Here is what I did in case you want more details. I was using a DVD drive connected to the USB port of the primary OA.

Since this was the first time I had used V9.0 to update multiple servers, I decided to try just a single server first (Bay 1). When it did not appear to be working after about 10 minutes I reset the server and connected the DVD drive to the USB port from the SUV connection on the front of the server. I ran the update from there (Interactively) and it went fine. I then decided to put the DVD drive back on the OA, and selected 4 servers to update. They were in bays 13-16. I kicked it off again and had the same results, except I decided to wait longer and see if they would eventually update.

I used the "Rack Firmware" link and the "Device Bays" link to monitor any changes that would take place with ILO or ROM firmware, as well monitoring if the DVD connection was lost. Much to my surprise, after about 15 minutes I noticed the UUID blink on the server in Bay 16, and then noticed that the ILO firmware reflected 1.81. It was at 1.78 before I started. Finally a sign something was working. Within a few minutes all four servers showed they had been updated. I then decided to run the update again on the servers in bays 7-12. I experienced the same results. The time was slightly longer, but only by a minute or so. I suspect that may have been because I was doing 6 instead of 4. I then did the servers in bay 2-5 and had the same experiences.

Before doing any of this, I reviewed the firmware rev levels of all components within the servers. All servers are configured the same and have FC cards in Mezz slot 1, and Ethernet cards in Mezz 2. The servers are all BL460c G6. Each needed firmware updates for the embedded NICS, ROM, ILO, FC and Ethernet Mezz cards, along with the Smart Array controllers.

I was concerned that even though I saw updates to the ROM and the ILO, did everything else get updated? As a test I connected the DVD to three of the servers via the SUV port and booted selecting Interactive Mode. In reviewing these three servers ALL components were up-to-date.

Not sure if I missed something along the way that would have caused the UUID to flash throughout the update, but I followed the instructions in the User's Guide on page 21. I do have several more chassis that need to be done, so if anyone has done this and had the UUID flash throughout the update, I am curious what I may have done wrong.

  You are correct.  The Firmware DVD first runs a check to determine which updates need to be applied, copies them into a RAM disk and then executes the update from there.  This process takes a bit, but the UID light is only lit when actual updates are being performed. 

With the old FDT, the entire contents of the FDT was copied to a RAM drive so if an update to the iLO occurred, we would not lose connection to the virtual media.  With the much larger size and support the new Firmware DVD provides, we could no longer blindly copy 900MB of data through the iLO virtual media connection because of speed and network bandwidth concerns.  To achieve this same functionality, HPSUM is first executed in report mode to determine the updates that should be applied to the server.  These updates are copied to the RAM drive and executed in the correct order. 

In this first release, we made a determination to only blink the UID light when the actual updates occur.  This would be akin to the "Please don't reboot the server" message that is displayed when updates are applied by other applications.  If this is a problem, we can always change it to occur much earlier in the process.

Also, I saw a mention on the PDL earlier this week of how many servers can be updated simultaneously.  We added logic starting with FDT 1.70 that carried forward to the Firmware DVD 9.00 to reboot the server if an iLO connection wasn't serviced.  With this logic, every server in an enclosure can be updated together and if the iLO virtual media errors out, we'll handle the reboot to try again.  In our lab, we've been able to update 16 blades in about 20 minutes even with one of the servers rebooting because it was unable to launch the ISO.

**********************************************************************************************************************

Michael gave some info on the HPSUM utility:

*******************************************************************************************************************

You are correct.  The Firmware DVD first runs a check to determine which updates need to be applied, copies them into a RAM disk and then executes the update from there.  This process takes a bit, but the UID light is only lit when actual updates are being performed. 

With the old FDT, the entire contents of the FDT was copied to a RAM drive so if an update to the iLO occurred, we would not lose connection to the virtual media.  With the much larger size and support the new Firmware DVD provides, we could no longer blindly copy 900MB of data through the iLO virtual media connection because of speed and network bandwidth concerns.  To achieve this same functionality, HPSUM is first executed in report mode to determine the updates that should be applied to the server.  These updates are copied to the RAM drive and executed in the correct order. 

In this release, we made a determination to only blink the UID light when the actual updates occur.  This would be akin to the "Please don't reboot the server" message that is displayed when updates are applied by other applications. 

Also, I saw a mention on the PDL earlier this week of how many servers can be updated simultaneously.  We added logic starting with FDT 1.70 that carried forward to the Firmware DVD 9.00 to reboot the server if an iLO connection wasn't serviced.  With this logic, every server in an enclosure can be updated together and if the iLO virtual media errors out, we'll handle the reboot to try again.  In our lab, we've been able to update 16 blades in about 20 minutes even with one of the servers rebooting because it was unable to launch the ISO.

*****************************************************************************************************************

Here's to successful upgrade procedures! Let us know how we can make it better!