ILO - Slot 255 is not responsive.

[bwood]

Hi, thanks a lot for the information and I'm glad to hear your issues are resolved. if you don't mind, may I ask what version of SPS you were running and what the mtbf for the 800ms delay was prior to updating SPS? thanks.

[ferrieux]

Hello,

Prior to upgrading, the SPS version was   4.4.4.300.0 and the MTBF was close to 15mn.

[bwood]

Hello, would it be possible for you to provide us with an AHS report? Hopefully one that covers the span of time prior to updating SPS and following the update?

If so, i can provide you with an HPE dropbox link via email if that works for you.

Thanks a lot.

[ferrieux]

Sure, please send the dropbox link.

[bwood]

Would you provide email address so that i can share the link?

[ferrieux]

If you're an HPE employee, I assume you have access to the e-mail address I registered my account with.

[bwood]

Hi,  I was only able to retrieve the email address that was associated with the customer reference number which you provided 

CASE-00114158.

We sent an email to that address on 12/6 but we never received a reply. Is that the same address you are referring to?  I'd be happy to send that email again.

Thanks again for you help.

[ferrieux]

Yes, please send it again. That's the proper channel anyway.

[bwood]

Ok. Email is being sent. Thanks so much.

[pirx]

@bwood 

I've updated a bunch of servers a few weeks ago (DL380gen10+) and one of them is now permanently reseting its ILO.  I'm not even able to logn via GUI most of the time. Via ilorest I was able to reset ILO once, all other commands (MCTP disable/Enable, fw update, collect logs) are failing due to communication issues (I'm running ilorest on the ESXi, so no network involved).

iLORest > serverlogs --selectlog=AHS --directorypath=/tmp/sdev2838.log
ERROR : Error 11 occurred while trying to open a channel to iLO

I created case 5383242083 for this. I have a bit of fear that the case will slowly progress as I'm not able to provide logs, this is most of the time then getting compicated...

In my remote syslog I see this:

Jul 5 12:06:23 kdev2838 iLO5 The iLO health monitoring status of the device / adapter located in Slot 255 is not responsive. ACTION: Do one or more of the following:#0121.Disable the MCTP and wait for 2 minutes. Enable the MCTP in iLO#0122.Perform the iLO reset.#0123.Perform the server reboot. If the issue still persists contact support
Jul 5 12:08:50 kdev2838 iLO5 Watchdog activated on smif_daemon, pc 01781158, sp 001f4498 R0=3b R1=99 R2=0 R3=0 R4=19a000 R5=541c8 R6=0 R7=f8134 R8=147e04 R9=0 R10=0 FP=0 IP=16f, SP=1f4498 LR=541c8
Jul 5 12:14:20 kdev2838 iLO5 iLO reset by watchdog.
--
Jul 5 12:21:52 kdev2838 iLO5 The iLO health monitoring status of the device / adapter located in Slot 255 is not responsive. ACTION: Do one or more of the following:#0121.Disable the MCTP and wait for 2 minutes. Enable the MCTP in iLO#0122.Perform the iLO reset.#0123.Perform the server reboot. If the issue still persists contact support
Jul 5 12:25:25 kdev2838 iLO5 Watchdog activated on smif_daemon, pc 01781158, sp 001f4498 R0=3b R1=99 R2=0 R3=0 R4=19a000 R5=541c8 R6=0 R7=f8134 R8=147e04 R9=0 R10=0 FP=0 IP=160, SP=1f4498 LR=541c8
Jul 5 12:30:50 kdev2838 iLO5 iLO reset by watchdog.
--
Jul 5 12:37:27 kdev2838 iLO5 The iLO health monitoring status of the device / adapter located in Slot 255 is not responsive. ACTION: Do one or more of the following:#0121.Disable the MCTP and wait for 2 minutes. Enable the MCTP in iLO#0122.Perform the iLO reset.#0123.Perform the server reboot. If the issue still persists contact support
Jul 5 12:42:16 kdev2838 iLO5 Watchdog activated on smif_daemon, pc 01781158, sp 001f4498 R0=3b R1=99 R2=0 R3=0 R4=19a000 R5=541c8 R6=0 R7=f8134 R8=147e04 R9=0 R10=0 FP=0 IP=160, SP=1f4498 LR=541c8
Jul 5 12:47:47 kdev2838 iLO5 iLO reset by watchdog.

[bwood]

Hi, have you tried removing a/c power and re applying to recover? If this is not an option, you can try to reset SPS via ipmitool depending on availablity of ipmitool  I know it's available and works for ESXi7,  Not sure about 8.  Assuming ipmitool is an option for you, please let me know and i will provide instructions.  Once recovered, you should be able to update to the latest versions of BIOS and iLO and collect AHS if you still have issues.

[pirx]

@bwood  Thanks for the quick reply. HPE support was also quick and straight forward this time, replacement motherboard is on its way. To remove power completely I've to send someone from external on-site service to the server. If it's already clear that chances are high that a part has to be replaced, I rather skip this step.

[bwood]

I don't think, with the info provided, that a board replacement is necessary.  Since a board swap requires power to be removed. I would suggest what i previously suggested prior to board swap.