Oracle 9i RAC cluster interconnect

[Jim Turner]

Although this is not purely an HP-UX or MCSG question, I hope you'll forgive me for tapping the brain trust which I've found to live here.

On a two-node RAC cluster, why does Oracle say that a simple crossover cable is not supported for the cluster interconnect? Oracle insists that the interconnect must go through a switch because the switch provldes some sort of heartbeat mojo that the Oracle driver needs. (SQE Test?)

I just want to understand why I'm gonna have to burn two switch ports instead of using a crossover cable.

Thanks!

(Cross-posted to hpux-admin@dutchworks.nl.)

[Steven E. Protter]

Jim,

I can see no reason at all. Heartbeat is usually a ping like functionality.

I'd say test it and see if it runs. There may be something about it that Oracle RAC, which runs on top of SG does not like.

Personally I'd like having a switch so a machine outside the cluster can monitor that networking is working well.

SEP

[Jim Turner]

Hi Steven,

Actually, the cluster interconnect (separate from MCSG heartbeat) carries a ton of traffic between the nodes. Oracle calls it "cache fusion" (something the rest of the world calls "cache coherency"). Basically it allows the SGA on one node to know what's in the SGA on the other node. What Oracle is telling us is, "Crossover Cable is not supported as an Interconnect with 9iRAC/10gRAC on any platform" as noted here:
http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix.html

There's apparently an article in Metalink that says, "A bridge or a hub is mandatory when using ethernet for the interconnect. HP does not support a straight through ethernet connection for a private ethernet interconnect.

Oracle claims that slowness in our cluster is being caused by our use of a crossover cable instead of a switch on the cluster interconnect.

But nobody can tell me exactly why. Nobody can show me the math. It's as if the guy at Oracle who designed the interconnect driver has left the company and nobody left has a clue.

[A. Clay Stephenson]

Okay Jim, here is my best guess.
Using an xover cable should work perfectly almost all the time; however when link activity is lost there is no way to determine which NIC has failed so the local node will probably assume the remote node has failed and evict it -- and this could happen on either end. Now when a switch (I assume you setup a small VLAN for this) or a hub is in place, the local NIC can tell that it still sees link activity and conclude (rightly) that the remote end has failed.

Besides, there is a lot to be said for the winky-blinky lights on the switch or hub because now us dumb humans can also tell which end of the connection has failed as well. I do admit that there is the non-zero probability of a switch failure and that is indistinguishable from the failure in the crossover case.

Finally (and the most important reason) is when you call Oracle for support and they ask about you interconnect, when you tell them about your xover cable, you are going to hear something like "your configuration is not supported."





The problem with crossover cables is that there may not be link activity
all the time and the NIC cards will sense this loss of activity and
assume the remote node has gone down and force an eviction.

[A. Clay Stephenson]

I can't think of a reason why an xover cable should be slower than a switch but this seems like something that would be rather easy to test. I would create a 1GB or so file and copy it to the other server. Now open an ftp session from Server A to Server B (and vice versa) and push this identical file simulataneously from both ftp clients and time the results. I would do this 3 or 4 times so that the buffer cache plays fair and average your results. I suggest pushing from both ends simultaneously to simulate full-duplex traffic. Now repeat your test with a switch in the middle and compare.

I think the xover/switch requirement has everything to do with Link Activity detection (from a heartbeat perspective or thr RAC equivalent) and nothing to do with performance but the above FTP test should reveal any performance differences. In any event, as long as Oracle has this excuse to fall back on (bogus or not from a performance perspective) you are really in a finger-pointing mode and not getting the real problem fixed.

[Delrish]

What happen if your cross cable cuts? both the NIC on the servers unpluged and disconnected. I think oracle face some troubles with this situation and it cannot detect what happen in cluster and does the cable disconnected or the server be failed.

[Luk Vandenbussche]

Jim,

We are running a two-node RAC with a crossover cable between the cluster interconnect cards.
This works fine, we never had any problem.