iLO4, iLO5, iLO6 incompatibility with Thinkpads, network disconnects every few seconds

[vkcgm]

Hello, I have a problem with recently purchased second hand server. Obviously it has no warranty so I am just looking for suggestions. This is my very first HP server so I am not sure if it is normal or if my particular server has a broken iLO chip. Judging by some googling it's most likely a problem with iLO.

The video redirection is almost unusable, because every full screen refresh takes several seconds, e.g. I could choose which kernel to boot in the Grub screen, input a password and see the user's home screen on the VGA monitor already, while in iLO4 video redirection the screen would still show the Grub's kernel list and then it will take about a minute to show the boot process. The partial screen refresh (e.g. just moving a mouse cursor or typing text) is somewhat OK.

I've found out the cause of this problem: the download speed from iLO is 8-20 kilobytes per second! (tested with wget https://10.x.x.x/html/intgapp4_231.jar and then confirmed with wget https://10.x.x.x/html/intgapp4_232.jar after I've updated to the latest firmware).

But the upload speed is normal - I've uploaded a 13 MB firmware file in just few seconds, while monitoring nload in the terminal, and nload showed average speed = 6 Mbps and max speed = 8 Mbps. Also if I connect an ISO file as a Virtual Media then the upload speed is several megabytes per second again, the problem is only with download speed from the iLO.

For the clarity of an experiment I have tested two different LAN cables and the download speed is 8-20 kBps with both of them.

I did a factory reset of the iLO but it did not help.

One person suggested that it might be a problem with NAND memory: https://support.hpe.com/hpesc/public/docDisplay?docId=a00048622en_us

- but the iLO 4 GUI Diagnostics page does not show a "Degraded" status for "Embedded Flash/SD-Card"; and everything on Overview and Diagnostics pages is green and healthy.

Nevertheless I have formatted the flash using the hponcfg utility and xml file from the URL above. Unfortunately nothing changed - the download speed is still ~10 kilobytes per second, thus the video redirection is slow as hell.

I have found few posts with the similar issue (slow Video Redirection) but it does not match my one - one person reported that it was a problem with the browser, another reported that "noapic" or "nolapic" boot parameter helped with video speed in the installed OS, but my iLO is slow even in BIOS - before booting an OS.

Is there anything else I could try or it is definitely a problem with iLO chip? If the latter - is it possible to replace iLO chip alone or I will have to replace the whole motherboard?

P.S.  Note that the Linux manual at the URL above is incorrect - "hponcfg -I file.xml" (large "i") parameter does not exist in the latest version of hponcfg, only "hponcfg -i file.xml" does (small "i"), but it does not work anyway. Judging by the strace it creates an empty file "tmpXmlInput 1.xml" and reads for input from STDIN, but even if you write anything it to STDIN it does not work anyway.

You have to run the hponcfg utility like this: "hponcfg -i file.xmp < file.xml" then it will run succesfully and will format the NAND memory.

[Parvez_Admin]

Hello @vkcgm 

As there is no response to the query yet, I would recommend to directly contact technical support and log a support call for quicker resolution. Please refer the links below for support ticket options:

https://support.hpe.com/help/en/Content/supportAndOtherResources.html

https://www.hpe.com/psnow/doc/A00039121ENW

[vkcgm]

The server is second hand and its warranty expired 1 year ago (28-Dec-2020), so I doubt I would get any assistance from HP official support.

The server in question is DL320e Gen8 v2. Could you tell which is the approximate price of the motherboard? If it's more than few hundred bucks then I'll just dump this one and will buy another second hand server.

Back to the topic, it is definitely a hardware problem.

I have switched iLO to the shared port (NIC1) for the test, and the problem became even worse - the network connection disconnects every few seconds, especially when the network packets are larger than a simple ping. For example, if I download the abovementioned .jar file then the network will disconnect 100%, same as if I try to open the web console in the browser. If I SSH to iLO then it would work well until I request something long - e.g. simple commands like "version" or "show map1" work well, but "show map1/log1/"  will make the network go down.

However a simple ping will work indefinitely until I try to download something else (like opening iLO4 in the web browser) - then the network will disconnect immediately.

My computer's dmesg looks like this:

[3112290.402333] e1000e 0000:00:1f.6 eth1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[3112305.813688] e1000e 0000:00:1f.6 eth1: NIC Link is Down
[3112312.246420] e1000e 0000:00:1f.6 eth1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[3112331.877754] e1000e 0000:00:1f.6 eth1: NIC Link is Down
[3112338.682486] e1000e 0000:00:1f.6 eth1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[3112422.053901] e1000e 0000:00:1f.6 eth1: NIC Link is Down
[3112428.882593] e1000e 0000:00:1f.6 eth1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[3112434.050260] e1000e 0000:00:1f.6 eth1: NIC Link is Down
[3112440.478510] e1000e 0000:00:1f.6 eth1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

iLO4 Event Log looks like this:

ID Last Update Initial Update Count Description
253 12/10/2021 21:42 12/10/2021 21:42 1 SSH logout: Administrator - 10.127.0.3(DNS name not found).
252 12/10/2021 21:39 12/10/2021 21:39 1 SSH login: Administrator - 10.127.0.3(DNS name not found).
251 12/10/2021 21:42 12/10/2021 21:38 5 iLO network link down.
250 12/10/2021 21:46 12/10/2021 21:35 6 iLO network link up at 100 Mbps.

(note 5 disconnects / 6 connects in a row)

However when I switched the iLO network interface back to its dedicated port then the constant disconnecting has stopped. But iLO is still incredibly slow - the download speed of the abovementioned .jar file is 10KB/s on average.

[vkcgm]

The kind people in the internets told me that it's most likely a dead NAND chip.

Is it possible to replace the NAND chip alone to fix that issue? If yes, which one?

I've found these memory chips close to iLO chip:

- Samsung K4B2G16460-BCK0 (sits just below the iLO chip, could be the NAND in question)
- e4 A 25Q2813B40 7B575 VS PHL 447, has paper sticker "A12SDR2 iLO4: 2.02_p13" (backup iLO firmware?)
- SK Hynix H26M31003GMR e-NAND 509A M12WU787QS (another storage, for what?)
- Winbond 25Q64FVSIG 1511 (BIOS storage?)

I think that it will be easier to dump the chip FW & solder new chip & flash the backup FW than to sell the current server and to buy a new one. And this operation will cost just about the same.

[vkcgm]

To whom it may concern: I've bought another server with iLO4 (now gen9) and its iLO is dead too, with the very same symptoms: slow file download speed, constant disconnections (even from SSH if sending/receiving a large packet), etc.

The iLO firmware version was 2.53 from 3 May 2017 - never updated since the purchase. Updating to the latest version, factory resetting, and force formatting (as described in https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00048622en_us ) did not help.

BTW that document still shows incorrect command for Linux - there should be

./hponcfg -i anything.xml < Force_Format.xml

UPDATE: I feel embarassed and ashamed but I have to report this for the future self.

The gen9 server and its iLO4 are totally fine, and I think the gen8 server's iLO4 works well too. I've discovered it when decided to connect to iLO from another server rather than from my laptop. So the problem appeared to be some incompatibility between my laptop (Thinkpad X260) and HP servers. Or possibly a broken LAN port?

The most evident symptom is - the LAN connection between the laptop and iLO4 auto-negotiages to 100mbps Half Duplex instead of a Full Duplex. But if I connect iLO4 to another server then the LAN connection auto-negotiates to 100mbps Full Duplex.

And then the Remote Console speed is good too - the abovementioned .jar file downloads at 700 kbps, and the average Video Redirection speed is 100-200 kbps (gets up to 2 mbps) which is quite usable.

I thought it was an iLO problem because I've never seen anything like this before despite I've managed hundreds of physical and thousands of virtual servers over the years, including the extremely old servers where you have to use Windows XP with Java 6 and enable SSLv3 with DES encryption to make the Java applet work. But those servers mostly were Dell, Supermicro, Fujitsu and Lenovo, I've never used HP before. And I have never met the same problem with any server - only with these two HPs - so I believe it is more likely some kind of incompatibility instead of a broken LAN port.

Of course I've tried different LAN cables of different brands, because this was my very first thought, but this problem appeared with every single cable. And I could use other servers' IPMIs without any problems with the very same cables.

Well, today I've found yet another black sorcery in the IT world.

[gfmoore]

Late to the party having just bought a second hand microserver gen8. ILO 4 is obviously a dead duck. The only thing it is useful for is doing a reset of the server. How come previous in warranty gen8 owners didn't report issues? Or did they. Not sure I'd want to spend thousands on a new server if this is how HPE deal with this situation. Good research btw and possibly helpful  

[ProgentCT]

I just ran into this too.  What I've found is that the .net IPMI console is waaaay faster.  If you still have a WS2012R2 or an OS that supports the .net plugin, use it.  I tried copying data from an ISO 3 times wtih the HTML5 plugin and it bombed every time.  Each time failed after over 3-4 hours. I even set the console disconnect to infinite.  I am on ILO 2.8.2.

I still use HP servers because they've always been the enterprise standard.  It's undertandable that modern software will slowly fail to support older hardware.  Keeping around older tools is tough to do in this era of ransomware.

[vkcgm]

The gen9 server and its iLO4 are totally fine, and I think the gen8 server's iLO4 works well too. I've discovered it when decided to connect to iLO from another server rather than from my laptop. So the problem appeared to be some incompatibility between my laptop (Thinkpad X260) and HP servers. Or possibly a broken LAN port?

No, they are not fine.
The problem persists with newer servers (iLO5 and iLO6, up to gen11) as well as with newer Thinkpads. The main symptom is: link auto-negotiates to 100 Mbps instead of 1 Gbps.

[1511083.996668] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511086.065584] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511088.140583] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511088.497039] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511090.556663] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511090.837123] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511092.896844] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511093.545166] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511095.617659] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511096.377146] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511098.440827] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511098.721180] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511100.796790] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511101.069155] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511103.128663] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511103.409121] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511105.484980] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511105.757202] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511107.817216] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511108.349220] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511110.408718] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511110.837194] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511112.917422] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511113.189225] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511115.266037] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511115.917116] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511118.000988] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511118.769253] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511120.828696] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511121.881145] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511123.936984] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511124.609196] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511126.668621] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511126.949119] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511129.008662] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511129.649128] e1000e 0000:00:1f.6 eth2: NIC Link is Down
[1511131.724739] e1000e 0000:00:1f.6 eth2: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[1511131.989192] e1000e 0000:00:1f.6 eth2: NIC Link is Down

This is definitely a flaw in HP servers (ethernet chips?) as I do not experience the same issue with any other brand servers I manage.

[Marco Correnti Techno S]

Uhmm .... it seems that you are using a direct connection from your PC to the ILO, is it correct?

In this case I suggest you to use a switch and configure all the ports, the ILO, the PC and the two switch ports, to 100FD and see what happens.

Do not use autonegotiation. Anyway 100Mb/s could be ok if FD.

If you really need 1Gb/s  be sure that ALL ethernet interface, ILO, PC and the ones in the switch manage this speed.

It's unlikely that you have the same problem with 3/4 different servers and even of different generations with differentc PCs.

[vkcgm · Accepted solution]

> In this case I suggest you to use a switch and configure all the ports, the ILO, the PC and the two switch ports, to 100FD and see what happens.

That's what I have to use, and it does work. A dumb 100Mbps switch operating at 5V powered via USB adapter.

> Do not use autonegotiation.

Setting speed manually to 100Mbps or 1Gbps does not help, the network still disconnects every few seconds.

> If you really need 1Gb/s be sure that ALL ethernet interface, ILO, PC and the ones in the switch manage this speed.

I don't really need 1Gbps, I need stable connection. And it is stable with every single brand I've ever met except HPE.

> It's unlikely that you have the same problem with 3/4 different servers and even of different generations with differentc PCs.

However it actually is: different model Thinkpads and different model HP servers. No problem with Dell, Lenovo, Supermicro, whatever; constant disconnects with HP.

[Marco Correnti Techno S]

Is the hardware always used in all attempts the switch?
It seems like a great candidate to replace, have you tried?

[vkcgm]

Sorry, I don't understand your question. I've experienced this problem with the old gen8 server mentioned in the first post, then it was the same with gen9 server (and the same laptop), and now I have the very same problem with a very new gen11 server and a different laptop. But if I connect the laptop to the server using _any_ intermediate device, be it a managed 1Gbps switch or a dumb small 8-port 100Mbps switch then the problem disappears. And it is absolutely sure not a bad cable as I've tried like 5 different ones.

No, I'm not going to replace the newest server and/or buy a different laptop, I believe this will not help because the issue described is either a firmware incompatibility between Thinkpads and HPs, or some software issue - could be something with the network settings on my laptop(s)?

Nevertheless I'd like to emphasize again that there are no any issues with other brand servers.

[Marco Correnti Techno S]

I misunderstood what you wrote, but be aware of the following :

The iLO Shared Network Port connection can operate up to a maximum speed of 100 Mbps.

Source https://support.hpe.com/hpesc/public/docDisplay?docId=a00105236en_us&page=GUID-D7147C7F-2016-0901-06D0-0000000005B1.html

Until Gen7 (I don't have any Gen8) the maximum speed of the dedicated ILO is 100Mb/s

Gen9 and later are 1000/100/10 Mb/s. (and maybe even Gen8)