Port speed negotiates to 100 Mb when turned off

[wrons]

Hi there, 

We are using HPE Proliant Gen 10 DL360 Server which is connected to a Cisco Nexus Switch (Cisco Nexus 93240YC-FX2) through Gigabit port. On operational mode (HPE port is up) the speed negotiated is 1 Gb however when the HPE server is turned off it then negotiates to 100 Mbps. Is this normal behavior? This causes an issue on the switch side resulting in the switchport flapping as it can't negotiate 100 Mbps as speed. 

Also, we have noticed that the ports are still up on the Server when checked through iLO even though it is turned off. Is this also an expected behavior? 

Will appreciate your response. Thank you!

[Mayrink]

Is that port being used for the iLO as a shared port as well?

This might be affecting the case that when the server is off, the iLO takes over and might be negotiating the speed at 100mb rather than the full speed of the server.

Check the iLO configuration for shared network port and if possible put it on another port.

[wrons]

Hi, 

We are using a dedicated iLO port. 

We use ports 1 and 2 of the HPE server as WAN and LAN ports respectively. 

Is it normal for both interfaces to negotiate to 100 Mbps when it is turned off? 

[pchops]

---

Is it normal for both interfaces to negotiate to 100 Mbps when it is turned off? 

---

FWIW -- I can't provide an authoritative answer, or even a "why does this happen?" answer, but I do observe the same sort of behavior with a ProLiant DL360 Gen10 and a ProCurve switch. That is, I *don't* see the link flapping because it is a 10/100/1000 switch, however when the server is powered off, but remains on auxilliary/standby power, the embedded LOM ports step down to 100 Mbps (whereas the same ports operate a 1000 Mbps when the server is powered on). I actually never gave this much thought, because many NICs will link up even when the server is on auxilliary power, but I had not noticed the speed change until your post (you learn something new every day).

I did come across this post:

https://community.hpe.com/t5/proliant-servers-ml-dl-sl/dl380-network-detected-as-100mbps-not-1gbps/td-p/6085555

and one of the respondents stated:

Keeping up a gigabit link takes more power than a 100 Mbps link. When the server is powered down and the NIC is kept active only for the Wake-on-LAN functionality, the NIC can switch down to 100 Mbps speed to save power. When the server is running, unconfigured NICs might also be completely powered down, or in a 100 Mbps low-power state.

I can't find documentation to back that up, but if we are talking about the ProLiant DL360 Gen10 embedded LOM (331i) ports, they are based on the Broadcom 5719 chip and the latter supports a 10Base-T and 100Base-T "Low-Power Mode", if that is at all related.

Anyway, I agree that the reason for keeping NIC ports active (aside from dedicated management ports), when the server is in a powered-down state, is to facilitate features like Wake-on-LAN (WOL). That said, I can't find a way to turn off the 331i ports (short of disabling them altogether). Even if I disable WOL on the NIC port *and* on the server, the link remains active when the server is in a powered-down state. Also, in the UEFI BIOS settings for the 331i, there doesn't appear to be a way to set the speed to a "fixed" 1000 Mbps -- there is only 10Mbps Half/Full, 100Mbps Half/Full, and "AutoNeg".

What I have observed when using said ProCurve 10/100/1000 switch is if I set a switch port to "auto-1000" mode (meaning it will only auto-negotiate 1000 Mbps speed/duplex), then when the server is in a powered-down state, I no longer get a link on the NIC port, or see any activity. One thing you might try is using a similar setting (whatever that might be) on the Cisco ports in question, if possible. Just my two cents...

[Svenster]

Hello and many thanks for the explanation.
Evidently it makes a lot of sense to reduce energy consumption by tuning down the 1000 into a 100 fdx, certainly for features like Wake Up or in a scenario where someone uses the regular four NIC ports for iLO and doesn't have a dedicated baseline management card slotted in their server (cannot imagine a reason for that, those things cost what, 70,- ?)

The problem in this case is that the CISCO switch on the other end does not take this lightly and is "flapping".
It is my impression that this could be caused by the energy-saving re-negotiation where we might try 1000 FDX, then revert to 100 DFX when the operating system of the server (that'd be some CentOS flavour) is powered down, ut the server chassis as such still has power.

In response to these changes, the switch decides to deactivate the NIC switchside, which causes issues when the server-os is fired up again.

I might need to dig deeper into documentation o see if it is possible to prevent this by settign a different power-plan or such in the BIOS, i.e. stop saving energy and hope that we stay on 1000FDX - any pointer in this matter would be highly appreciated.
I don't see this as a product fault or bug, but it could deserve some TLC on the lines of Quality Of Life enhancement.

If none of this works, I'd be rather stuck with the advice to disale those checks and the deactivation on the CISCO switch but I'm not sure how this will go down with the network/switch teams.

Cheers,
Sven