should I use infiniband as the rac interconnet?

[yao_4]

Hi, experts,

The rac include two Hp superdome using Gb fiber as interconnect, as the rac have many global cache cr request wait, I suspect may be the interconnect is the bottlenect.
I use glance to check the system,
Network utility is 53%
interface In KB Out KB
6091.9/9376.7 4703.1/5830.8
considing the Gb Fiber speed is about <100MB/s , the interface in out KB is about 10M/s
is the network is the bottleneck ? replace it as infiniband can do some good

thx a lot

[Fabien GUTIERREZ]

if i understand clearly you are using IP over FC
is your cluster a local one or are nodes more distant (how many nodes)?
we re using a close configuration to yours with HF2 links for interconnect on a local cluster with no more than 2-3Mb/s
so if it's confirmed it's a local cluster i suggest you have a look to your database configuration checking for exemple your sga size and you buffer hit ratios
oracle first check on local node the information in memory then contact the other node to get the information if none hav it then it gets data from datafile

[Court Campbell]

First off I am not a rac guru. I guess My suggestion would be to try HP's APA (port aggregation) product. Then you could aggregate multiple fiber links on the interconnect network for each node. Or if you can get it to work you can try jumbo frames. I am not sure what you interconnect block size is, but if it is 8k jumbo frames may increase performance.

[yao_4]

mu cluset is local one with two node.
anysis of oracle stat report show that :
global cache cr request wait is the top one ,
so I wish to know whether the interconnect is the bottleneck.

[Fabien GUTIERREZ]

what make me think the problem is not your interconnect is the fact that you have a very high traffic that should not occur (on my example we are handling 30.000 simultaneous connexions with a backend traffic of 2-3Mb)
so i suggest you deeply look to your oracle parameters statistics and whatever but do not look at global cache cr as the first error it may comes from another place (bad sql request)

[kevin_m]

I agree with the suggestion for jumbo frames (MTU = 9000) if your switches can support it. My company received some performance model data from a consultant based on our 16 core 2-node cluster. LMS processes may consume about 1/4 of processing power with a standard MTU size. Switching to jumbo frames could reduce that to 1/20 or less. Granted the numbers may not match with more processors but the difference is still dramatic.

Kevin