DL380 G6 SAS Backplane Firmware update problem

[ERLB]

Hi,

I was in the course of updating a DL380 G6 firmwares for all components currently in it. I had downloaded the latest HP Firmware Maintenance CD which is v9 and proceeded to download the other firmware versions that were newer that the CD's release date.

I created a bootable USB HP Smart Update using HP's USB Utility and included the additional firmware file per the instructions. I then entered setup and selected all available components for update including the 24k20 PIC which is the SAS backplane which had a problem in the end.

All components were successfully updated except the 24k20 PIC which displayed an error saying that it failed to update. I rebooted the server and the four left side locations where hard disks can be installed are now blinking blue, the other four on the right are normal. The disks attached to these no longer display activity indicator or any kind of activity except the blinking blue LED. Note that everything still functioned perfectly normal, OS booted normally and all disk I/O seemed OK. IML logged no errors. I turned off the server and moved the disks to the right side. The left side still was blinking blue though now no disks were attached to it and the right side was functioning normally with activity LEDs active.

I rebooted from the USB and both 24k2 PIC were still offered for update, one showed its version is 1.18(actually 1.18B, latest) and the other N/A. I tried flashing the firmware again, this time it finished with a success message but the same behavior still occurred. Note that all other components succeeded and are now at the latest available version: iLO2 v2.0, P410i v3.30, etc.

Any idea what is wrong or what is happening? I tried searching but unable to find anything related to this sort of behavior.

[itelio GmbH]

We have same Problem on 2 DL380 G6.
We have the blue LEDs Blinking on both Servers after Firmware Update CD 9.0. After this we tried a Firmware Update Again with USB Version of CD 9.0.

We copied th 1.18B Version to the Stick went to interactive Firmware Update (Selected also non Bundle Firmwares) and push the Firmware Upgrade again. This needs many Minutes on 99% because the %-Time is only for the Left 4 DiskPlaces. After Upgrading to 1.18B and shutdown, plug off Powercords the Leds are normal again.

[ERLB]

Unfortunately, I had tried that before and I tried it again now after you confirmed that it worked for you, but still the behavior persisted.

[Craig Sistek]

We also experienced the same issue after using the HP Smart Update Firmware DVD. I first ran the DVD in automatic mode. After running the update I had four disks with flashing blue lights and the fifth disc was green. We reran the update in interactive mode, shutdown the system and unplugged the power cords. I did not have any bundles available. Then I plugged the system back in and powered it up. The blue lights went away. The system RAID configuration stayed intact throughout the entire process. We have a mirror and a RAID5 configured.

[Simon Gordon]

Hello,
Thanks for the advice. We also experienced the problem after running the Firmware Update DVD in automatic mode. We solved the problem by re-running the firmware update DVD in interactive mode, selecting the 24K20 PIC update and re-applying this. This fails, so I exit the firmware update and power down the server and remove the power cables. On restart, the blue lights stop flashing and the problem is resolved.
I have now successfully created a RAID 5 Array and loaded the operating system.
I have erased this configuration and repeated the process without errors including running the Firmware Update DVD.

[gregersenj]

Hi Guys.

The Blue Light an identifiaction light.
It will lit to identify a drive og the drives within an array, and can be turned on by ACU or through the SMH. Possibly also in other ways.

So I guess, that the FW upgrade could turn it on. even though I haver never noticed myself.

If so, and the FW upgrade fail, I guess that it could stay on.

Do take note. The disk FW is upgraded by the smart array controller, after the reboot !!.

So mayby, it just didn't finish?

BR
/jag

[ERLB]

I did try to reflash numerous times and did what most of you mention, turning off the server and unplugging the power but nothing worked and the lights kept flashing.

Further checks revealed that the backplane went nuts and was no longer detecting the correct placement of the HDDs as well. The right side remained functional and correctly detected the disks from top to bottom as 5, 6, 7 & 8. The left side on the other hand detected them from top to bottom as 7, 6, 5, & 4 (if I remember correctly) which is totally wrong.

In the end, HP replaced the backplane.

[gregersenj]

yanking the power after a FW update, is always risky.
You would have to be absolutly sure, it completed.
If uprade hasn't completet you will corrupt it.

BR
/jag

[Stigge]

I noticed the issue when flashing FW on backplanes on MSA60 attached to a SA controller. My workaround when FW upgrades fails and is to select installation options then tick the allow rewrites. After that I managed to upgrade several backplanes.
/Stig

[gregersenj]

Hi Guys

I have just upgraded 9 DL380 G6's, 2 days ago.

Used FW DVD 9.10 (B)
All newer FW was copied in to \hp\swpackages.
So latest and greatest on sep 13th 2010.

Got blue flashing on the drives.
Upgrade reportet 1192 hours left.

Just leave. It will complete within 20 - 30 min.
Don't !! powercycle / reboot !! You might corrupt fw.

I had drives in 2 "boxes"
It first flashed box 1, then box 2.


Just like you I was impatient.
I removed the USB flash when it was hanging.
And startet on the next server.
So I had to run the upgrade twice to complete.

BR
/jag

[ERLB]

I never unplugged it when it was still running, the first time, it took about 20 minutes to finish and it reported a failure at the end. The next time, the process took about the same wait time, it was reported as success in the end but the flashing persisted.

[gregersenj]

In my case it stopped flashing.
The PIC was completet.
But it failed to continue with the P410, the ILo? (Not sure) and The BIOS.
So it had to reboot, from HPSUM.

It should have continued (I think), enven thoug I had removed the USB stick, since the fw is loaded into memeory.

I did update ILo to 2.01 and P410i to 3.50.

BR
/jag

[gregersenj]

Correction !!

I used FW DVD 9.10 (C).

Sorry

BR
/jag

[ERLB]

That version was still not released at the time when I was performing the upgrade, I resorted to the USB method.

Besides the failing firmware update for one of the PICs, all the other (iLO, P410, ...) succeeded, even the second PIC succeeded, all from the firs try.

[gregersenj]

Are you aware, that you can copy newer FW into the FW CD / DVD?

BR
/jag

[ERLB]

I did that with the USB that I created. I created it from the newest DVD that was available at the time. Individual firmwares were already updated past what was present on the DVD so I added them to the flash disk and performed the update using the newest version available.

[gregersenj]

So did I.

And now I'm really messing up.
Just re-chekked my image.

I did use 9.10 (B).

Sorry again.

BR
/jag

[Patty Nelke]

Same issues on 3 DL380 G6 servers.
First system the drives looked fine from the factory but I like to update first thing. Used firmware 9.10a but with all the supplemental updates I could find at the time (about 2 weeks ago). First update said it completed successfully, but then the drives showed the 4 blue lights flashing continually on the left bank, and no lights onthe right. Going into the array setup the system showed drive 5, 6, 7 and 8... and again, 5, 6, 7 and 8. A build would not go onto the system as it couldn't find the drive, though I could create logical drives. I booted back to firmware and found that the array controller hadn't updated, so pushed it back to 3.5 and rebooted. All was well.

2nd system all the firmware except the 24k20 PIC showed as current, even the array controller. Re-pushed the array to 3.5 again and pushed the backplane. Same problem. Downloaded 9.10c and applied it. Issue went away.

Haven't tackled the 3rd system yet, trying to narrow down what specific components are at play.

[gregersenj]

Try 9.10(c)

But do let it hang, with the flashing blue LED's.
It will continue after appr. 30 min.
And tell you that some FW has failed.
Then reboot and re-run fw upgrade.

This way you won't risk it to fail during falsh of a component.

BR
/jag

[apb]

I know this is an old thread, but for any future internet searchers, it looks like this firmware that was causing issues in 2010 is still causing issues in 2013:

http://community.spiceworks.com/topic/268756-urgent-smart-array-orphaned-drives

HP has not fixed the firmware updater even though their firmware writers have known of the issue for at least a year and their firmware team doesn't have configurations with 16 drive bays for testing.

Sad.  Workaround found by Marvin Miller is to only have one backplane at a time plugged in and, if possible, don't use a SAS expander for the firmware update.  I only found out about this AFTER I started searching for "hp backplane sas firmware how long" via Google.  Probably also a good idea to unplug your drives before updating, too, lest you suffer data loss.

Luckily I am running into this issue on a newly-bought DL380 G6 so even if it goes south I won't suffer any data loss.

[apb]

Looks like HP may have finally fixed this issue:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3884083&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_7065b9df61644429bc348b295a%257CswEnvOID%253D54%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vigne...

[Oscar A. Perez]

If you are upgrading the SAS backplane on a G7, be sure your iLO3 is version 1.57 or later.

[Boadi_4]

---

@ERLB wrote:
I never unplugged it when it was still running, the first time, it took about 20 minutes to finish and it reported a failure at the end. The next time, the process took about the same wait time, it was reported as success in the end but the flashing persisted.

---

after i updated the firmware some hard drives show green and others show amber color. hp dl380 G6. pls any help