Crossover connected 1GB NIC can not communicate upon reboot

[Mel Burslan]

I have a strange problem in my hand and I don't know how to solve this. Here is the situation

Two servers, both identical configuration, 580-GL7

4 GB NICs (or I was told) on the main board.

one of these nics from each side is connected via a cross-over cable to each other, which will serve as a heartbeat for an Oracle RAC cluster.

When we reboot server both interfaces can ping themselves but can not ping one from the other.

ifconfig bond1 down; ifconfig bond1 up   <-- on both servers, peace gets restored and everything starts to work

If one side gets reboot while the other one is running, we are done. No ping between two nodes.

The  MAC address prefix on these interfaces are 1C:C1:DE: which is an HP reserved string I found by googling it. But at the same time, I know that HP basically buys a broadcom or some other manufacturer's chipset and put their own firmware on them as they are not in the chip fabrication business any more.

We are suspecting, letting these interfaces auto-neg their speed is the problem as there is no network gear in between the two. But if we can nail it down to 1000FD from the get go, this problem will go away. At least this is the inclination. We might be proven wrong but until that time, I am working towards this goal.

when I use ethtool I am getting this :

]# ethtool -s eth4 speed 1000 duplex full autoneg off
Cannot set new settings: Input/output error
  not setting speed
  not setting duplex
  not setting autoneg

which works mighty fine on any broadcom NIC

we tried to put this in the ifcfg-eth4 file asan additional line as follows

ETHTOOL_OPTS="speed 1000 duplex full autoneg off"

but upon reboot, the same story ensued.

I am under the impression that this NIC os so new that whatever database ethtool utility is looking into, to determine what commands to us, does not have the codes for these cards. (by looking at the "Input/output error" string in ethtool output)

This thing not being HPUX, where my strength is, I am pretty much clueless about the next step in the gameplan. Any suggestions are greatly appreciated.

[Hein van den Heuvel]

This question may be moot, because best  I know Oracle does NOT support cross-over cables. 

See for example: http://forums.oracle.com/forums/thread.jspa?threadID=855085

When you mention cross-over that is symbolically speaking right?

Surely you are using a regular straight cat-6 and relying on auto-MDX  ( MDI-X aka  Auto uplink and trade, Universal Cable Recognition and Auto Sensing)

Personally, I would not want to use crossover, as I want to be able to 'look at the lights' to help decide what looks good or bad. 

No free ports on a switch handy that you can isolate?

Get a $50 4 - 8 port gigabit switch from your cornerstore to try?

( and yes, of course I've used crossover when in a bind) 

fwiw,

Hein

[Mel Burslan]

Thanks for the response Hein. I know cross-over cable usage in general is a bad idea and my organization is in the process of adding a few blades to our cisco distribution switches. But the project doesn't have the time to wait for the regular procurement channels to work. Eventually, everything will be on the switches, but until the additional switch capacity exhibits itself, I need to make this work with a couple of cross-over cables. This is my dilemma.