DL120G7 connecting iLO shared port interferes with second NIC

[mquezada]

I have a Proliant DL120 G7 running Ubuntu 11.04. As per the manual, NIC port 1 is shared between the iLO subsystem and network IO with the operating system. Then NIC port 2 is network IO with the OS only. Both NIC ports are configured for DHCP and both receive their IP's from the DHCP server sucessfully. Port 1 is 192.168.2.203 and port 2 is 192.168.2.202. If I connect either one of the NIC ports to the network (by itself), that port will operate as expected. However if I connect both ports to the network at the same time NIC port 2 becomes unreachable. I cannot ping it from another machine and I cannot ping or connect out from the DL120 on that port. NIC port 1 (the iLO shared port) remains functional as normal. As soon as I disconnect NIC port 1 from the network, NIC port 2 becomes responsive again.

I have not configured or ever accessed the iLO subsystem on the server. I intend to make use of it at some point but right now I need to get both ports working. As a side note, I am new to HP servers. Is this issue a problem with the server or its configuration? Why would connecting the iLO shared port interfere with the other NIC? And more importantly, how can I fix this?

Thanks,

-Marco

[Matti_Kurkela]

This might not be iLO-related at all, but more related to you having two NICs of the same system connected to the same network segment without any bonding/teaming configuration.

By default, most operating systems route outgoing traffic using the first match in their routing table. If you have multiple separate NICs connected to the same network segment, this will cause outgoing traffic for both IPs to go out from the NIC that happens to be listed first in the routing table. That may confuse the switches in the network segment. Security features like MAC/IP hijacking protection can make your switches/routers act more aggressively in situations like this.

This is a well-known situation, described as "the ARP Flux Problem" as described on page 17 (per in-document page numbering) in this PDF:

http://linux-ip.net/linux-ip.pdf

Note that the document was written in the Linux 2.4 era, so some of the configuration details may be obsolete, but the problem description is still relevant.

The most well-known solution would probably be to configure the two NICs into a bond (the choice of bonding mode depends on your use case), so that both interfaces will appear as a single "bondN" device. Then configure one of the IP addresses to the main bondN device, and the other as an IP alias (use device name "bondN:1"; it will be auto-generated when you configure it).

Another way to configure multiple IP addresses for a NIC/bond is to use "ip address add" to add the second IP. This is the newer way, and it may be confusing if you are used to use the "ifconfig" command to check your NIC parameters; the second IP won't appear in "ifconfig" output at all. Only "ip addr show" will list the extra addresses configured in this way.

If you don't need the bandwidth/redundancy of two physical NICs, you could also configure the extra IP address(es) on a single physical NIC, using the interface name "ethN:1" with ifconfig or "ip address add".

It is possible to make your current configuration work, but it requires some changes: first, you'll need to solve the ARP flux problem, and second, you will probably also have to configure your routing in a more advanced way, so that it will also consider the source IP addresses when deciding which NIC to use for outgoing packets.