SIM firmware push hanging DL380 G5

[Bruce Rose]

We have been updating server BIOS and opetion firmware using SIM for a few months.
In most case all is well .. the pushes work and after the server is rebooted all is
where it should be.

In one case we have had issues. On pushing to 70 DL380 G5 servers SmartArray P400,
BIOS and ILO2 we had 14 servers go into a hung state (they did not crash, but were
inaccessible and a power cycle had to be done. We noticed that the three files were
downloaded from the repository and a START was sent to the first file, ion this case
the P400. That is when the system became unresponsive and after 30 minutes had to be
power cycled.

This past weekend, we again tried to push to the remaining DL380 G5 servers (10 total
this time). We pushed the BIOS, S/A P400 and ILO2. Five of the ten went to this hung
state. We saw the files downloaded from the repository, a START sent to the first file,
in this case the BIOS and it succeeded, a START sent to the second, the P400 and here
again the servers became unresponsive. After the servers were power cycled, indeed the
BIOS was updated, but nothing more.

So in each case the servers, not all but a half in the second case, had issues when the
P400 firmware (4.12 sent .. 2.08 current) was sent the START to install. I have as yet
found anything pertaining to this issue and am hoping someone may have seen this as well
an perhaps have a cause.

Thanks

[KarloChacon]

hi



what I see you current is 2.08 and you want to update to 4.12 I see there are 2 4.12 version 4.12 and 4.12B I will assume it is the first one

let me ask you have you tried pushing 5.20 version instead of 4.12 just to check if you get the same issue? or you have any policy in which you can not use the latest version?


all versions here

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1157687&swItem=MTX-e2ab069150904db6bcdf3f1449&prodNameId=1157689&swEnvOID=1005&swLang=8&taskId=135&mode=5

this is I've seen some servers locking up when updating this p400 Fw controller and I would say that maybe 4.12 has this issue updating from that 2.08 version

try and let us know

bye

[KarloChacon]

hi again

also you say server updated System ROM but nothing else

what happens when you try to push again the updates for that server(the one that became unresponsive and only System ROM was updated) I mean p400 update is it successful? or it fails again?

I want to know if having System ROm update p400 Fw fails again or not

bye

[Graeme Bray]

We had the exact same issues with our servers. The workaround is to go 2.08->2.10->4.06 then to 4.12 or 5.20. I know its several reboots but it stops the server hangs.

Also, if you are running the server on W2K3 SP2 with a heavy load, you should install an updated Storport Driver after updating the SAS Controller to the latest version as well.

We use KB946448 and it stops multiple issues. Please let me know if you have any questions.

[Bruce Rose]

The servers are critical servers that we can't get to very often. We had one crash yesterday and while it was down they were able to update the p400i firmware (on line). We are going to 4.12(B) and the possible issue of having to go to an iterim version has come up. I hadn't seen any writeups on a possible issue so we tried going to the latest (what was the latest when we started this). Finally the KB for the storport. I can only pass this along to see if it can be done.
I will be working this with HP support as well and will post any fix that comes out of it.

Thanks again

[Graeme Bray]

I imagine that you have monthly reboots. If so, try to get approval to do an update from 2.08 to 2.10. Then go from 2.10 to 4.06. then finish off the cycle. I have not tried to update from 2.08 to 5.20 (however I probably wouldn't try it).

Please let me know if you have any issues.

I contacted HP and they wouldn't pass it on to their backline support to fix and there was nothing on their end. I just have my personal experience that killed several field Database servers that caused production outages during the scheduled push times. On our critical Cluster servers, we did the sequential upgrade and it worked fine.

I understand the issues getting management approval to do routine maintenance (THEY HATE IT!!) but thats what the technical people are for!

[Bruce Rose]

We have a total of 4 more DL380 G5 servers to get the P400 update sent to. We have done online firmware updates with no issue and we have gone right to 4.12 with them.

We have DL580 G4 and DL580 G5 servers to do, but of the 30 + servers there are only 6 still at the 2.08 so we may end up doing these manually with the online method. Hopefully going from the newer levels will not be an issue.

[RaMpaNTe]

Well, I would have to ask you for the driver versions currently installed, as you know this Online FW version uses the driver as a "bridge" if you wanna use simple words. So if the p400 driver is old (chances are they are old) so I would not be surprised if I see the servers hanging when trying to push a new FW trough and old driver.

So my recommendation at this point would be to push over SIM the most recent p400 driver and then once you have rebooted you can push the p400 FW.

Thanks,

RaMpaNTe

[Bruce Rose]

Yesterday evening we tried to run the online update to the server BIOS and the P400 with the same results. The BIOS succeeded and asked for the reboot, the P400 started but the server hung. This completely eliminates SIM.

We have been trying to get 2.08 updated to 4.12, but yesterday we were going to 5.20. The server BIOS was at 2008.01.09 this time while the last time it was at 2006.12.28.

We are looking to find other options including offline using the 8.2 firmware cd.
If there are other things we should look at and/or recommendations please let me know.

Thanks

Bruce

[Bruce Rose]

I will check out the drivers ...

We had compared the drivers between the boxes that worked (most worked just fine) and the ones that had the hang issue (14 the first SIM push .. 6 the second SIM push - 6 of 14 original .. 2 last night - not pushed - done local). The drivers were the same.