iLo 4 v 2.10 DHCP fails

[tdelporto]

Environment:
HP DL360e GEN 8, iLo 4 v 2.10 (also observed in 2.03 on this server, and 2.00 on DL620p G8)
ISC-DHCPD 4.2.5

We're observing iLo failing to request a DHCP lease if a renewal is not granted.

The sequence of events is:

* DHCP server is running on network iLo is connected to
* Power is connected to server
* iLo gets a DHCP lease and we can connect to the web interface and use IPMI
* iLo continues to renew the DHCP lease every 2 minutes which is the correct renewal period
* DHCP server becomes unavailable - this is expected during software updates, etc.
* iLo requests a renewal and does not receive it
* iLo is no longer reachable via the previously leased IP
* iLo does not broadcast lease requests
* DHCP server becomes available
* iLo does not request a lease, and continues to be unreachable

At this point the only remedy is to disconnect the network cable from iLo or bounce the port on the switch. After iLo detects the interface coming up, it requests a DHCP lease and receives one from the DHCP server.

The expected behavior is that iLo would continiously request a new lease if it did not receive a renewal.

Is there a way to set iLo to request a new DHCP if a renewal fails?

Thanks,

Tony

[Jimmy Vance]

I don't have an answer for your issue, but having the lease time expire every 2 minutes is very unusual. could be an issue there. Maybe make the lease time just a bit longer than it takes to reboot your dhcp server?

[tdelporto]

Thanks for the response. I agree that a 2 minute lease is unusually short for most applications.

The application is a cluster that can change size relatively quickly, and IPMI is used for discovery and power management. Increasing the lease time could be an option, but, there would be edge cases where the time was insufficient or too long. We'd much rather find a way to treat iLo the way we treat BMCs from other vendors.

Best Regards,

Tony

[Jimmy Vance]

I understand.

Best practives would actually be to have all your BMC's on a seperate management network.

I'll pass your issue along to the iLO team. I'll let you know if I get a response

[Oscar A. Perez]

We recently discovered an issue with iLO DHCP client where iLO could give up attempting to renew its IP address if the server doesn't respond to iLO DHCP requests at T1 and T2.

Remember that DHCP uses UDP protocol and unlike TCP, UDP does NOT guarantee that the packet will be delivered. A busy network will likely result in a high number of UDP packets being dropped.

So, until we have a new iLO4 version with the fix for the DHCP client, I would suggest you to use long lease times.  Minimum 1 week.  

Also, please don't use minutes for lease time. It is completely unecessary. In my opinion, anything less than 24 hours for lease time is calling for trouble regardless of the iLO issue.   T1 is 50% of lease time so, if you have a large number of DHCP clients in a given subnet, the subnet is probably flooding with UPD traffic causing a high number of these UDP packets to get lost.

[Torsten.]

Agreed. Most DHCP related communication are broadcasts, and if you have 2min lease time each server is asking after 1min again for that address. So you're flooding the network just with DHCP traffic.

Consider to put all management related ports (ILO etc) in a separated network and use fixed addresses for them. This avoids a lot of trouble.

[Oscar A. Perez]

Also, you need to consider that iLO is not your normal "grandma"  BMC.

A BMC is normally a very simple remote management solution that runs few services on a dozen of threads.  

iLO on the other hand is a Lights Out Management that is running several dozens of services on hundreds of threads.  Since lots of these services depend on the network, changing iLO IP address every 2 minutes causes iLO to re-initialize all these services which lead to temporally outages that can last several seconds.

Using 2 minutes for a DHCP lease time would be a good stress test but, not a good idea for a production environment.

[tdelporto]

Hi All,

First, thank you very much for the well considered, timely, and informative responses. They are very helpful.

I'm glad to hear the issue with DHCP renewal is known and hope that a fix is forthcoming.

The management network is dedicated and isolated from other networks for the reasons mentioned as well as security. We create static leases/reservations for each server's iLo connection upon discovery, so iLo isn't getting a new IP per se, though it sounds like a renewal and a new lease may have the same effect on reinitializing services. That is a very helpful thing to know. We can move forward and hopefully adapt.

Best Regards,

Tony

[Jimmy Vance]

It sounds like you're already setting the IP'S via the iLO MAC address, maybe set that section of the DHCP file to have a longer release time or possibly set the renew time based on the client identifier.