bonding fails on reseting rilo security switch

[Dale Ferrier]

I have searched for "ilo bonding" but can't find a post that seems to cover my particular problem.

Here is the situation, we have an older HP server a DL380 G3 which we had to use the rilo reset switch in order to disable security so that we could define a new user and password on the rilo card. We then proceeded to install RHEL 4.8 and all was peachy. We defined two bonded interfaces bond0 and bond1 and all was well.

But then we flipped the dip switch on the motherboard back to re-enable the security on the rilo card and then booted the server.

Now we could not ping the default gateway nor any of the other static routes we had defined going over the two interfaces.

We checked the bonded interfaces and all looked normal from a linux stand point. No errors in the log and ifconfig reported all interfaces up a running normally.

Our network support could not see a mac address from any of the 4 physical interfaces on the switch however which of course would be a problem. :)

So pulling the hair a little and as a last resort I disabled bonding and brought eth0 up with the bond0 IP and eth1 up on the bond1 IP. And put in our static routes as before.

Now all was again well. And as long as we do not try to re-enable bonding everything stays well.

We did find some traffic on some forums somewhere about the linux re-enumeration problem that can affect servers on a reboot sometimes when the hardware changes. And we found a reference to a case on the DL380x series where flipping the rilo security switch might cause a re-enumeration of the devices.

But the MAC addresses for the bond0 and bond1 interfaces stayed the same (corresponded to eth0 and eth1 respectively). So I do not think re-enumeration is what was happening.

By the way bond0 was using eth0 and eth2 and bond1 was using eth1 and eth3.

So has anyone run into something like this before?

Dale

[Steven E. Protter]

Shalom Dale,

You don't ilo bond.

The ilo card needs to be left out of this mix. If it shares a NIC port with a regular NIC card, then the wiring needs to be on a network permitting ilo and regular traffic and you can bond eth0 and eth1

The search you want is redhat network bonding.

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-networkscripts-interfaces.html

Nothing in here as far as I know is RHEL specific.

SEP

[Steven E. Protter]

Sorry.

Nothing is RHEL 4 specific.

These cards on these servers can be bonded. But active-active bonding does not work well.

SEP

[Dale Ferrier]

Steven I think you may have misread my post. I am not trying to bond the rilo nic and a normal nic.

I have two onboard interfaces and a dual interface intel card installed in this server.

The two built in broadcom interfaces are eth0 and eth1 and the two intel interfaces are eth2 and eth3.

The rilo only comes into play because this problem did not present itself until I flipped the hardware security switch on the motherboard to re-enable security on the rilo.

Before I did that (in other words I installed the OS RHEL 4.8 while the switch was in the non-secure postion) the two bonded interfaces came up and ran fine.

Here are my configs:

modprobe.conf:
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter cciss
alias usb-controller ohci-hcd
alias net-pf-10 off
alias ipv6 off
alias bond0 bonding
alias bond1 bonding
options bonding max_bonds=2 mode=1 miimon=100 downdelay=5000 updelay=5000
alias eth3 e1000
alias eth2 e1000

ifcfg-bond0:
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.1.1
USERCTL=no

ifcfg-bond1:
DEVICE=bond1
BOOTPROTO=none
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.1.2
USERCTL=no

ifcfg-eth0:
DEVICE=eth0
HWADDR=00:0E:7F:F0:5E:D9
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

ifcfg-eth1:
DEVICE=eth1
HWADDR=00:0E:7F:F0:5E:D8
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no

ifcfg-eth2:
DEVICE=eth2
HWADDR=00:11:0A:57:6C:90
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

ifcfg-eth3:
DEVICE=eth3
HWADDR=00:11:0A:57:6C:91
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no

So you can see pretty straight forward.

Dale

[Steven E. Protter]

Dale,

Perhaps you mistyped.

>>
I have searched for "ilo bonding"
<<

Thanks for finally providing your OS and update level.

If you bond broadcom to broadcom and intel to intel, it will probably work.

I spent some time trying go get intel to bond with broadcom.

The result was an unstable bond and frequent loss of network connection on the system.

What you are trying to do is perfectly good from the theory stand point, but intel and broadcom NIC's don't play well in the same bond set.

There are also cisco switch issues, which come to play. If you want extra high reliability by putting your bonds on different switches, switch configuration may be needed.

>>
And we found a reference to a case on the DL380x series where flipping the rilo security switch might cause a re-enumeration of the devices.
<<

This is an OS issue. I've pushed very hard in this area and never gotten renumeration of the devices. I've got about five years experience with this server class. If you get renumeration, you will probably get it again and have stability issues down the road.

Renumeration is bad and I don't think it is happening.

As a precaution, I take the hardware address out of the configuration file when doing bonding. You might want to try taking it out or putting it in to assist with this situation.

Also, I've gotten bonding to work with hacking about in this area and then found no networking on reboot, so include that in your test plan.

I find it hard to believe both of my original posts were valueless, if you find this one valueless, I'm going to take a pass at further participation in this thread.

SEP

[Dale Ferrier]

Steven,

Nope I really didn't mistype. I just put that there to show I had done some preliminary searching for something linking the rilo part with something to do with the bonding.

I have in the past on different forums found that to be helpful to the folks reading my posts to see that I had at least put forth some effort before posting. In those forums people tended to take a dim view of someone who did not at least try searching before posting. That is how I meant for the community here to take it as well.

It seems to me that you didn't read my original post very closely or I think you would have realized I wasn't trying to bond a nic with a rilo.

And I am afraid that your original two posts were completely non germain to the problem I put before you.

I do not need help configuring bonding. I have that taken care of. Your last post I do find helpful, however.

I have had trouble with broadcom nics in the past and not just bonded ones either.

I am coming to the conclusion that the whole security switch thing may have just been coincidence and like you point out this may have always been unstable but since we very rarely re-boot or disturb this server, it just had not shown itself to be a problem in the past.