HP Blade DHCP question

[Manuel E. Cotallo Torre]

Hi,

We're having trouble with HP blade ILO's attempts to gain DHCP addresses every few seconds.
We've got a dynamic dhcp/dns system, and of course ILO's are included in this subsystem, but when there is a serious incident taking down all servers, bringing them up is a chaos because of dozen of ILO's repeated attempts to re-gain IP address in few seconds take down our DHCP server

Any way to change through ILO CLI this value for DHCP dialogue interval being more conservative?

Thanks a lot!
Manuel Cotallo.

[KarloChacon]

hi

I really don't know about blades (I wish I could) but for MD/DL servers there is a feature in BIOS as well as ilo where you can can set a delay for the server to come up after a power outage for example nice really (that could avoid to take down your DHCP Serve)

but just in case ypu might see the thread below
I found this

http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1207555234549+28353475&threadId=1213613

[rick jones]

The previous suggestions to spread the power-up sound promising. Still, I'd like to address the flip side:

How many systems are you talking about here? DHCP isn't exactly featherweight, but that a DHCP server could be taken down by load seems, well, odd, unless we are talking about a _lot_ of load. What is implementing your DHCP service?

[Manuel E. Cotallo Torre]

Yeah, maybe it's not a hardware issue, although about 40 ilos spreading DHCP request every few seconds it's not also very polite ...

So, yes, maybe a DHCP3 server in a Ubuntu 6.0.6 virtual Xen box over another Ubuntu 6.0.6 physical one, sounds not very nice, but it is an inherited (evilish) design.

My other thought it is DHCP implementation at Ubuntu either is a bit buggy or has any kind of limitation concerning requests. So maybe any kind of TCP/IP tuning could do? Can DHCP daemon be tuned to accept more concurrent dialogues?

Thx a lot !

[rick jones]

If the only load here is 40 iLOs doing DHCP's every couple of seconds, that is not all that heavy a load IMO. Even if it were once a second, that is at most only 80 packets per second, which isn't even epsilon as far as network loading is concerned these days.

That your DHCP server is a virtual machine sounds suspcious as for performance. So many places for packets to be lost, where to begin? Up in the client, you should check the UDP stats with netstat -s -u:

izzy:~/linux-2.6/drivers/net# netstat -s -u
IcmpMsg:
InType8: 8703
Udp:
3752 packets received
0 packets to unknown port received.
0 packet receive errors
3743 packets sent
UdpLite:
IpExt:
InBcastPkts: 11120

And look to see if there are packet receive errors being recorded. That implies (but does not confirm) UDP socket buffer overflow, which you could try tweaking via sysctl to make sure that the default UDP socket buffer size was larger - large enough to handle a "burst" of 40 requests at one time. Start by doubling whatever it is now and see if the receive errors then go away.

As for where one checks for packet losses between the hypervisor/dom0/whatever and the client I'm not really sure. You could try some ethtool stats on the client and see if it records any errors, but after that one is left poking around in the hypervisor.

With how much "else" does this virutlaized DHCP server share the cycles on the physical hardware?

[Manuel E. Cotallo Torre]

Sounds something focusing to the point ...

Yeah, sounds completely meaningful, and we'll try.

If im allowed, i'll do a test, cause it even happens with four or five virtual machines asking at same time for DHCP IP.

And in that case, a 10 for your answer is granted :D :D :D :D :D

Thanx a lot. Let's see tomorrow if im allowed to launch experiments!