Buffer failure during boot. Proliant G9/VMware 6.7

[nferraro]

Good morning everyone.  We had an incident over the weekend. Some type of glitch in the building power. Most of the UPS's were fine, one of them wasn't (It's dead).  One of our DL360 G9 servers was connected to it and is no longer functional. VMware shifted everything to the other servers, so we're OK, but this server is getting a buffer failure when I try to boot it.  I've powered down and disconnected everything, pulled the internal SD boot card and reconnected all.  I am still getting this failure:

Loading /b.b00

Loading /jumpstrt.gz

Loading /useropts.gz

Loading /features.gz

Loading /k.b00

Loading /chardevs.b00

Loading /user.b00

Error 6 (Buffer too small) while decompressing data

  Input (541081), output(1437919732)

Gzip_extract failed for /user.b00 (size 541111): Buffer too small

Error 6 (Buffer too small) while loading module: /user.b00

Compressed MD5: dae04443fe4fd8d247dc727d0c9f83

Decompressed MD5: (about a thousand 0’s)

Fatal error: 6 (Buffer too small)

Internal diagnostics say everything is fine, and it's obviously seeing the card. I'm ASSUMING the VMware files got corrupted, but before I pull the files from the card and get hold of VMware, does anyone have any suggestions? BTW, this is a standard HP 8GB SD card and, as far as I know, VMware 6.7 sends all of it's log files to the local SCSI drive that is now mandatory. 

Thanks,

Nick Ferraro

[SB6]

Hello @nferraro 

This issue appers to be corruption in boot bank of ESXi host. 

Gzip_extract failed for /user.b00 (size 541111): Buffer too small

Error 6 (Buffer too small) while loading module: /user.b00

Compressed MD5: dae04443fe4fd8d247dc727d0c9f83

Decompressed MD5: (about a thousand 0’s)

Fatal error: 6 (Buffer too small)

To isolate possible recovery, if System ROM and iLo FW outdated, it would be advised to upgrade same first to latest

Further once verify in iLo > Diagnostics tab if any errors or alerts.

Further I see you had tried to remove and reseat SD card already, if possible you may try below steps again.

Power Down Server, Unplug both powercords, wait for 2-3 minutes and remove SD card and then wait for another 10-15 minutes, re-insert SD card and re-connect power cords and ower up the Host and see if this addresses issue.

If you still see issue, advised would be perform NVRAM clear using System Maintenance Switch on board. It would be advised to do same in co-ordination with HPE Support. You may log support case for further troubleshooting actions on this issue.

Regards,

SB6

I am an HPE Employee ! 

[Med-Hamdi]

Hello nferraro,

Based on the description and the details that you have shared "Fatal error: 6 (Buffer too small)". This is probably a VMware files got corrupted  and there is no much you can do.

You My first check the SD card Status from Ilo GUI if everything is ok or not.
Also you can Try to test with different SD card.

Else proceed with re-installation of the system.

Thanks

[nferraro]

Thanks for your reply.  iLo has been fully updated for a while (I did see the iLO 2.4 boot issue mentioned online). When I shut it down, I mean I FULLY shut it down, disconnected everything and drained it.    

I hadn't considered that the NVRAM might have been corrupted - I assumed (again) that as long as it went through POST without an issue and the iLO showed no errors, it was fine. I'll open a ticket and check on that.   Thanks again.

[nferraro]

Thanks for your reply.  iLO says card status is good. Just to verify, I removed the card, fired up the iLO and it then reported a connection error to the card. I figured it was corrupted or damaged, but I was hesitant to wipe it until I had some more data - a simple power failure has cascaded into a real PITA.  Thanks again.

[sudhirsingh]

Hi,

Do you see any "Unsupported configuration" or corrput meta-data for SD card slot 1 or 2 in IML events,

Login to ILO console and check Integrated Management logs event.

In my experience , have seen this issue with one of the SD card and and it failed to sync and eventuallty os failed to boot,

replacing suspected SD card has solved.

Hope this helps!

Regards,

Sudhir