topic Query: iLO Amplifier Firmware update fails due to timeouts in Server Management - Remote Server Management

iLO Amplifier Firmware update fails due to timeouts

BjornM — Sun, 08 May 2022 05:20:53 GMT

Hi!

We're running into some issues where many of our firmware updates fails with the following error message:

"Non Interactive Update job failed because Pending State timed out. Launch IRC to check the status."

These are DL360 Gen 9 servers spread all around the globe, while the Amplifier appliance is installed in europe.

Updates to servers in europe completes without any issues, and some servers on the eastern borders of US also works, but servers elsewhere on the globe has issues and fails with the above error.

I've also tried using an external web-server to host the Service-Pack for Proliant .iso image, but i run into the same issue there.

So to me it seems like there is some kind of timeout that happens when the distance from the Amplifier to the ILO is too long.

Also, when the upgrade starts, on servers that works, it usually goes pretty quickly to 65% completed on the "Server update job", while when it fails, it gets stuck on 60% for a couple of hours before it fails with the error-message above.

Anyone have any good ideas ?

Query: iLO Amplifier Firmware update fails due to timeouts

support_s — Thu, 28 Apr 2022 10:09:12 GMT

System recommended content:

1. HPE Integrated Lights Out (iLO 4) - Troubleshooting Firmware Update Issues

2. HPE Integrated Lights-Out 3 (iLO 3) - Updating Firmware

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Thu, 28 Apr 2022 10:47:31 GMT

We do have all those in place., and the firmware upgrade works perfectly fine on all servers located in Europe.

But once the distance between the iLO Amplifier Appliance and the iLO/Server itself becomes "too long" it stops working.

We've also tried using an External Webserver to host the .iso file of the SPP, on each site locally, to avoid it having to read/use the amplifier-imported baseline, but this has not solved the issue either.

Re: iLO Amplifier Firmware update fails due to timeouts

Richard75 — Fri, 29 Apr 2022 12:53:14 GMT

What if you try to do a two steps- first stage only

do you use the full ISO, or did you create a smaller one, specific for the hardware you need.

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Fri, 29 Apr 2022 13:08:53 GMT

Hi!

I've tried stage-only, and it's on that part it fails.

And it's the full .iso for "gen 9" server's i've used.

Re: iLO Amplifier Firmware update fails due to timeouts

Richard75 — Fri, 29 Apr 2022 13:22:53 GMT

i guess it is just the distance as you already set.

because of the servers you already did you know which components need an update. just load the those seperate files

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Mon, 02 May 2022 07:28:02 GMT

But it is also a lot more work to update 50-60 servers manually instead of just staging and deploying

So would definately prefer it if HPE located and fixed the issue,.

Re: iLO Amplifier Firmware update fails due to timeouts

ManBha — Tue, 03 May 2022 04:24:55 GMT

Hello,

You can also try creating a custom image adding just the components which need to be updated.

https://support.hpe.com/hpesc/public/docDisplay?docId=a00026608en_us

https://spp.hpe.com/custom/loginrequired?redirect=%2Fcustom%2F

Thanks.

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Tue, 03 May 2022 07:00:05 GMT

I have tried using an external webserver located at the "remote location", but that has not helped either.

Have not tried using custom SPP's, so guess i'll have to try that next if i can figure out how to make one for Proliant DL360 Gen 9 servers.

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Wed, 04 May 2022 07:41:08 GMT

@ManBha

How long does it usually take the Build server to create a custom iso ?

It's gone more than 24 hours now, and it's still not ready for download, so just wanted to check how long this normally takes.

Re: iLO Amplifier Firmware update fails due to timeouts

ManBha — Thu, 05 May 2022 03:04:21 GMT

Hello,

This should take about 30-60mins to complete.

Please do log a support case to assist you on this.

Thanks.

Re: iLO Amplifier Firmware update fails due to timeouts

BjornM — Tue, 10 May 2022 10:21:01 GMT

@ManBha

Unfortunately the Custom SPP site seems broken currently. Been 4-5 days now and no bundle has been created. Also created a support-case, and they confirmed they had the same issue. So i believe a new "site-support" case was created.

Hopefully it will be fixed soon.

Re: iLO Amplifier Firmware update fails due to timeouts

pirx — Mon, 04 Jul 2022 12:53:55 GMT

ILOAmplifier is just a completely broken piece of SW. I have some cases open for ~1 year now. FW updates are failing even in local datacenter regularly. According to support (after 1 year) it's our network that is havin problems. Stragen that ILOAmplifier is the only product that is having problems and that is smells like a timeout issue in the software depending on the time certain steps in updates need to finiesh.

As ILOAmplifier is not a strategic product and will likely die soon, I don't put any more effort into creating bug reports.