Infiniband setup

[KevB_1]

As I stumble from one crisis to another

todays bit of fun is what do I need driver wise for infiniband and how do you set these up

Running REDHAT 5 update 3 on BL460c with a BLc 4x QDR IB switch which hooks into an exadata

[Tim Nelson]

Unless you have some super special IB hca then the driver is most likely included with your distro.

lspci to show your HW.
lsmod|grep mlx (in my case because it is a Mellanox hca)

now in order to get the rest working you need to load a protocol stack.

in my case, using Oracle Enterprise Linux I downloaded openib, infiniband-diags, and rds-tools from either the distro or ULN.

I believe the openib is included in most distros, so try "yum search openib", there may also be a newer version with a new name, e.g. ofed, infiniband, or something like that.

[KevB_1]

Hi Tim

Cheers for the info - from your instructions have

InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0)

Found this at HP site

MLNX_OFED_LINUX_1.51 which has drivers and OFED assume this is what I need

Once installed what do I see -looking at the exadata which was installed by Oracle I see there is a ib0 interface but this doesnt have a ip address just what looks like a MAC

Where are these configured ?

[Tim Nelson · Accepted solution]

use either the openib or as you list the OFED 1.5 from HP. your call.

lspci should show your IB hca, if not the driver is not loaded.

the ofed contains a subnet manager daemon, you need to decide if your switches support the subnet manager. if so I would let the switche(s) do the management, reason being, if the server goes down and there is no subnet manager then you loose your entire IB network. so disable the included SM service.

"service openibd status" to check on ib stack status. may have to start it.

add entry into /etc/modprobe.conf
"alias ib0 ib_ipoib"

"service openibd status" to check on ib stack status. may have to start it.

check that stack is loaded."lsmod|grep ib_ip

edit /etc/sysconfig/network-scripts/ifcfg-ib0


ifup ib0

I am using the openib rpm which is OFED 1.4.1, the configuration settings may be different than that of 1.5, you will need to read the docs that come with 1.5. for example. 1.4.1 works best if the CONNECTED_MODE and MTU settings are set in ifcfg-ibx vs /etc/infiniband/openib.conf

check the MTU with ifconfig (needs to be 65520) if you cannot seem to get it set after stop/start then it has to do with the CONNECTED_MODE setting.

reboot when done to make sure everything comes back right.

also, ibstat ibnetdiscover are great diag tools to check on things.

[KevB_1]

Once again Tim thanks for the advice

Found that the exadata IB was setup as a bond hence no ip address on the ib0 and ib1 interface

So you set them up as you would an ethernet interface with conneted and mtu set (going by what is on the exadats as they use the same switch)

[Tim Nelson]

yep. and depending on your switch setup then either bonding mode=1 (failover) or mode=0 round-robin.

I found that setting all the bonding options, mtu and connected_mode in the ifcfg-x files seemed to be more resilient.

cat /proc/net/bonding/bondx to confirm settings .

[KevB_1]

Tim

Ok dont have bonding on the blades only in the exadata so would you set the ib0 config on the blade to contain all the ip info , mtu and connected params

Sorry if being thick here as have just done all that and cannot ping betwen the interface i have setup and the exadata - do I need to add something to the switch ( are these likely to be zoned in someway even though it is a private network) bit vague on this as it is one of those things where someone comes in and sets it all up then rides off into the sunset !!!

[Tim Nelson]

There are some things that are different with OFED 1.5.1 vs the 1.4 that I am using.

1) MTU is set / defaulted to 65520 via the /etc/infiniband/openib.conf ( CONNECTED_MODE needs to be set in openib.conf as well via SET_IPOIB_CM=yes.)

2) /etc/sysconfig/network-scripts/ifcfg-ib0 looks like this(substitue the xx for your network):
DEVICE=ib0
BOOTPROTO=static
IPADDR=xx.xx.xx.xx
NETMASK=255.255.255.0
NETWORK=xx.xx.x.0
BROADCAST=xx.xx.xx.255
ONBOOT=yes

3) service openibd restart

ifconfig should show ib0 with new IP and correct MTU

ping

use ibstat, ibstatus, ibnodes to view your network

As far as your infiniband switch configuration.. mine is a flat network with no routing, I would either configure this host on same network as Exadata, or you will need to figure out how to manage the switch to include this host.


If you are using RDS as the interconnect protocal in Oracle then there is also some more to do with that config as well.

There are tons of docs in docs directory of the ofend install.

[KevB_1]

Tim

Thanks again

Will get down to some serious reading then !!