Starting drivers. Please wait, this may ... take HOURS!!!

[kira12]

Hi,

thanks for this Information, i had this issue too and i had use also some nc550 in my Servers. By the way the nc550 are not specified for Gen9, only untill Gen7. Do you use the actually Firmware for the nc550? For Gen 9 is the nc557.

best regards ré

[MajorTom]

Hi,

thanks for this info, I know, these are "old" cards, but I still have lots of them - and they are supported from within ESXi 7&8. With some of them I have an issue with the firmware, that's correct, will upgrade them within the next maintenance window.

BR Tom

[kira12]

Hello Tom,

we did not talk about "old" Cards, i use them too if i have some of them. They are cheap and work a well. But just in Case something are not working, we have no support from hpe. Now i exchange some old Cards to newer one. They als have less Powerconsumption as the old.

best regards ré

[ianluk]

Today I got the same problem, when I plugged in the P840 Raid Controller on the DL380 Gen 9 server. which hold on "Starting drivers. Please wait..." screen.

I updated the BIOS and iLO firmware versions, but the problem is still happening.

The P840 raid controller contains add-on cache memory. When the add-on card is removed, the server boots up properly. As the result, I cleaned the add-on card connector and installed it back on the raid controller. The problem was fixed.

[bryan_it]

I have this problem with 2 servers, both are DL380 Gen9. They were working fine until I upgraded the firmware.

I started the upgrade process on the first server, and tried a lot of different options to get it to boot, but everytime it would hang / stall / stuck / stick.

The second server I upgraded ILO and BIOS, no problem.
I updated HP Intelligent Provisioning to a newer version, no problem.
Then I updated firmware on the LOM Broadcom 1G 4Port 331i (5719A) and the Mellanox QSFP FlexiLOM, now the issue has started. The firmware was updated using the Intelligent Provisioner (v2.87.5)

I followed all the suggestions in this forum, and lots of different possible options for BIOS, ILO, removing all possible hardware. Nothing seems to work on either server.

I have to conclude that it's a firmware problem with the Lan-on-motherboard (LOM) Broadcom 1G 4Port 331i (5719A) as this is when the problem started.  Perhaps the firmware upgrade partially failed, or maybe you need to flash the firmware twice to get it to stick.

I don't think the Broadcom LOM can be upgraded/downgraded via the ILO, can anyone confirm ? My guess would be if I could downgrade this, then it might resolve.

HP DL380 Gen9
BIOS 2.96 (2022) (The most recent version they let you download without an account)
ILO 2.82 Feb 06 2023
Intelligent Provisioner (v2.87.5)

[ianluk]

Please refer other post from https://community.hpe.com/t5/proliant-servers-ml-dl-sl/hp-proliant-dl380-g5-nic-s-not-found-after-firmware-update/m-p/6418056/highlight/true

Posted by Letze01 at ‎03-19-2014 08:53 AM

Boot FreeDOS from ilo

**When the system maintenance switch position 6 is set to the On position, the system is prepared to erase all system
configuration settings from CMOS.

[kevimax]

Hi Bryan,

Welcome to the club !

Exactly the same problem for me here:

https://community.hpe.com/t5/proliant-servers-ml-dl-sl/proliant-dl380-gen-9-starting-drivers-please-wait/td-p/7196792/page/2

You can downgrade easily with a bin file from an earlier version of the firmware that I can provide you, using the "system utilities" ==> "embedded applications" at boot.

The problem here is that we can't get through the POST to get there!

I succeeded on one of my 3 servers, I still don't know why I can't do it on the other 2, the configurations being identical! version of firmware and bios too..

translated by google from french

[Forcix]

Has anyone found a solution to this problem?

Just ran into today on two Gen9's.

Wish I hadn't tried upgrading anything.

[kira12]

Hi,

as i had the Problem i swap my P440ar tho a H240ar and the Server booted. now i can update all Firmware Files, most Important was the Bios. This i actually 3.30. After that the Server works again with my P440ar.

best regards ré

[Forcix]

Thank you for that suggestion.

I was hoping that simply removing the P440ar controller and cold-restarting, unplug power would have worked.

It hasn't.

I'm starting to think that @bryan_it may be right, that the problem is with Broadcom 1G 4Port 331i (5719A) LOM.

I hope I don't have to get another RAID\HBA controller so as to confirm your solution.

[Forcix]

Finally, SUCCESS!

The suspicion that @bryan_it had was correct!

In my case, the problem was not with the P440ar but it does have to be phsyically removed so that the POST does eventually get past "Starting drivers. Please wait ..."

By "eventually" I mean it can take hours (I don't know exactly how many because I went to sleep).

My objective was to get the system to boot from virtual media that mounted the "Service Pack for ProLiant (SPP) Version Gen9.1" (P52574_001_spp-Gen9.1-Gen9SPPGen91.2022_0822.4.iso).

The steps I followed in order to get past "Starting drivers. Please wait ..." are from another thread in this forum:
https://community.hpe.com/t5/proliant-servers-ml-dl-sl/proliant-dl380-gen-9-starting-drivers-please-wait/td-p/7196792

In particular, on page 2 of that thread I found two comments by "YKC" and "dom-b" that were of interest and that were ultimately the way to get past "Starting drivers. Please wait ..."

Before starting the quoted procedures below, I

1) Disconnected the HDDs from the backplane and disconnected the backplane's power connector from the motherboard.

2) Removed the PCIe riser, then removed the P440ar controller.

3) Flipped maintenance switch 7 to force a legacy BIOS boot.

Then followed the steps below that involved numerous "Resets" and "Cold Boots".

"YKC":
---- SNIP ----
Re: Proliant DL380 Gen 9 - Starting Drivers Please Wait
ok, no problem

I tried many method,

1). power off the server , enter iLo to downgrade FW to 2.4

2). power on the server , it will hold on starting driver....

3). press cold boot in iLo , then change One-Time Boot Status to intelligent provisioning

4). power on the server , when performing early system initialization , press reset in iLo

5). server reboot and POST , when hold on starting driver.... change the power settings to Static High Performance Mode

And then it suddenly pass the "starting driver...." and continue

hope these can help you!
---- SNIP ----

"domb-b":
---- SNIP ----
Re: Proliant DL380 Gen 9 - Starting Drivers Please Wait
Had the same issue on my DL380 G9, very frustrating to say the least and poor HP has not taken down the current LOM firmware from online firmware updates.

After MANY hours of messing around and taking inspiration from this forum, here was my approach, which I have now repeated twice:

1). power off the server , I left firmware on 3.08. Remove the P440AR. Flip maintenance switch 7 to force a legacy BIOS boot.

2). power on the server , using the ILO reset the server during "Early Processor Init"

3). server reboot and POST , when hold on starting driver.... change the power settings to Static High Performance Mode

And then it suddenly pass the "starting driver...." and continue

4) boot the SPP SPP2021050.2021_0504.129 via the ILO and downgrade the LOM firmware

Should now be all resolved and can you flip back to UEFI mode and install the P440AR.
---- SNIP ----

After many resets and cold starts I left it "Starting drivers. Please wait ..." and went to sleep

On a side note, I recall that I did not have the SPP .ISO mounted in Virtual Media before I went to sleep.

But I do recall setting the One-Time Boot Option to Intelligent Provisioning and left it at  "

When I woke up hours later, I saw the console at Intelligent Provisioning.

Once booted from the SPP .ISO and into Smart Update Manager (SUM), the component that was selected by default was "HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64 (firmware-nic-broadcom-2.29.0-1.1.x86_64):

It was from there where I did a Firmware Update so as to deploy the "HPE Broadcom NX1 Online Firmware Upgrade".

Then, just to be sure, I restarted the system and while it POST'ed, I mounted the "P52574_001_spp-Gen9.1-Gen9SPPGen91.2022_0822.4.iso".

It went past "Starting drivers. Please wait ..." as expected and not because I mounted the ISO.

Then used SUM to reinstall that firmware again and restarted.

Finally, I

1) Powered it down during POST.

2) Removed power from the power supplies.

3) Re-inlstalled the P440ar. Re-installed the PCIe riser.

4) Flipped maintenance switch 7 to OFF. Placed the cover. Plugged in power. Momentarily depressed the power button to power off.

5) Using the iLO, switched the "Boot Mode", that is under Virtual Media's Boot Order, back to Unified Extensible Firmware Interface (UEFI).

6) Powered system back on and booted normally.

Hope this helps.

[douf123]

THe exact same thing happend to me with this firmware... got the latest HP IP iso from their website... used it to do firmware upgrade.. it only upgraded the lom. No errors.

Now stuck on loading drivers... was qutie fast before i tried to update.

[kira12]

Hi,

i only had this issue with older BIOS, now i use 3.08 and did not have the issue, With 3.30 i did not try it,

best regards ré

[giuseppelm64]

i have same problem, but and with some p840 card stuck on please wait and with an other p840 card pass this check and is show as generic storage device. how i can fix it? i already updated with spp all firmware

[kira12]

Hello,

what BIOS do you use? 3.08 or earlier?

best regards ré

[giuseppelm64]

i used,  v3.08, v3.30 and v2.96

[kira12]

Hello,

you try the procedure from Forcix on the Page before?

best regards ré

[giuseppelm64]

yes but now afer some restart and with ilo 2.80 worked and i flashed lom firmware. but now the strange problem is that raid controller p840 is shown as generic storage controller, anyone have this trouble?

[kira12]

Hallo,

do you have the actual Firmware on the P840?

best regards ré

[giuseppelm64]

should be 7.0 but i am not sure for this, but do you know why is show as generic storage controller?

[kevimax]

Hello everyone,

Here is my conclusion regarding this problem. I was able to verify it on several servers:

The only thing that works is patience!

By dint of reset and cold reset, at some point the server passes the post.

Prerequisites :

Disconnect the RAID controllers and any extensions.

If there is a LOM extension, disconnect it too!! You will have to start the procedure again to flash the ext LOM firmware after having done the on-board LOM!

The manipulation works indiscriminately with bios versions 2.XX or 3.XX
Same for iLO.
Changing the type of power supply (balanced, max...) is just an illusion.

I prepared a USB key with the BIN file extracted from the previous version of Linux firmware.

From iLO, do resets, until (sometimes after several hours), POST passes.
You should not reset too quickly during POST, it takes up to 30 seconds to see progress when it works

Then use the EFI Embedded application for LOM flash.

Good luck...waiting for an HPE patch.....

[giuseppelm64]

in some server with P840 raid card still stuck on starting drivers also with old lom firmware, and do not start SO, the only think that i can do is change controller. do you have any fixes for this?

[Andrzej_Ang]

I've been struggling with this problem for days - ripped off everything I could from the server (no RAID, no PCI cards, only 1 memory module), tried to do "Reset" and "Cold boot" as it was described earlier - nothing helped.

Changing SW2 switch positions 6 to ON (clear nvram),  7 (legacy boot) to ON - no change.

Came up with the idea that maybe there is a keystroke that can cancel this "Please wait", so I started to press F9 many times even before it appeared on the screen - believe it or not, but the screen passed this horrible message, I loaded ISO image with Service Pack for ProLiant, downgraded NIC firmware from 20.25.42 to 20.22.41, and since then the problem have not happened again! Did many restarts, power offs, cold boots - sever works!

So try the F9 trick, good luck!

Andrzej

[MCSL]

Thank you Andrzej !!!

---

@Andrzej_Ang wrote:

I've been struggling with this problem for days - ripped off everything I could from the server (no RAID, no PCI cards, only 1 memory module), tried to do "Reset" and "Cold boot" as it was described earlier - nothing helped.

Changing SW2 switch positions 6 to ON (clear nvram),  7 (legacy boot) to ON - no change.

Came up with the idea that maybe there is a keystroke that can cancel this "Please wait", so I started to press F9 many times even before it appeared on the screen - believe it or not, but the screen passed this horrible message, I loaded ISO image with Service Pack for ProLiant, downgraded NIC firmware from 20.25.42 to 20.22.41, and since then the problem have not happened again! Did many restarts, power offs, cold boots - sever works!

So try the F9 trick, good luck!

Andrzej

---

This approach worked perfectly for us, note that you have a chance to change the selected F9 option once the themal calibration message is shown after the dreaded starting drivers, for us once we passed this for the first time it's been fine ever since.

[Andrzej_Ang]

I'm really glad it worked for you as well, all the best!

Andrzej