cmquerycl doesn't recognize hyperfabric

[Kirk McGowan]

I'm having some trouble getting a cluster config to use the hyperfabric interfaces. cmquerycl seems to pick up all the lanx interfaces I have configured, but doesn't seem to recognize either of the clicx interfaces:

$ lanscan
Hardware Station Crd Hdw Net-Interface NM MAC HP-DLPI DLPI
Path Address In# State NamePPA ID Type Support Mjr#
0/0/0/0 0x00306E09013B 0 UP lan0 snap0 1 ETHER Yes 119
0/5/0/0/4/0 0x00306E044960 1 UP lan1 snap1 2 ETHER Yes 119
0/5/0/0/5/0 0x00306E044961 2 UP lan2 snap2 3 ETHER Yes 119
0/5/0/0/6/0 0x00306E044962 3 UP lan3 snap3 4 ETHER Yes 119
0/5/0/0/7/0 0x00306E044963 4 UP lan4 snap4 5 ETHER Yes 119
1/12/0/0 0x00000000 1 UP clic1 6 HyperFabricNo **
1/10/0/0 0x00000000 0 UP clic0 7 HyperFabricNo **
0/12/0/0 0x00306E034EE3 5 UP lan5 snap5 8 ETHER Yes 119
$ ifconfig lan5
lan5: flags=1843
inet 138.2.5.112 netmask ffffff00 broadcast 138.2.5.255
$ ifconfig clic0
clic0: flags=843
inet 198.162.50.8 netmask ffffff00 broadcast 198.162.50.255
$ ifconfig clic1
clic1: flags=843
inet 198.162.51.8 netmask ffffff00 broadcast 198.162.51.255
$ sudo cmquerycl -w full -l net -n atchp8

Warning: Network interface lan0 on node atchp8 couldn't talk to itself.
Warning: Network interface snap0 on node atchp8 couldn't talk to itself.
Warning: Network interface lan3 on node atchp8 couldn't talk to itself.
Warning: Network interface snap3 on node atchp8 couldn't talk to itself.
Warning: Network interface lan4 on node atchp8 couldn't talk to itself.
Warning: Network interface snap4 on node atchp8 couldn't talk to itself.

Node Names: atchp8

Bridged networks:

3 lan5 (atchp8)

IP subnets:

138.2.5.0 lan5 (atchp8)

Possible Heartbeat IPs:
138.2.5.0 138.2.5.112 (atchp8)

Possible Cluster Lock Devices:
$



This is on a 2-node N-class Oracle Parallel Server cluster.
# what /usr/lbin/cmcld
/usr/lbin/cmcld:
HP92453-02A.10.20 HP-UX SYMBOLIC DEBUGGER (END.O) $Revision: 74.03 $
Build date: Tue Oct 5 14:36:49 PDT 1999
Build id: sibld_sgops_a1109
A.11.09 Date: 08/06/1999
Daemon
Config DB
Cluster Monitor
Command Srv
CommunicationSrv
Config
Dlm
Local Comm
Network Sensor
Package Manager
Remote Comm
API
Service Sensor
Cluster LVM
Status DB
Sync
Util

Any suggestions or assistance would be greatly appreciated.

Regards,
Kirk.

[melvyn burnard]

First thing to note, you do not have ServiceGuard patched. I recommend you install PHSS_23511 on the nodes, as this is marked CRITICAL.

Seondly, I am unceratin as to what your exact problem is, but HyperFabric does not support Heartbeat, and therefore cmquerycl will not put in the HF links into the ascii file, quoting from Installing and Administering HyperFabric:

Although HyperFabric can be used by an application within a package to communicate with other nodes,
it is not possible to use HyperFabric as a heartbeat LAN. So, in a package control script, do not specify
HyperFabric IPs/subnets in the lines that contain the keywords IP[n] and SUBNET[n]. Also, cmquerycl will
not "discover" and report HyperFabric IPs and subnets.


Pleased read this manual, available at:

http://docs.hp.com/hpux/onlinedocs/B6257-90026/B6257-90026.html

[Kirk McGowan]

Hmmm. It's been a LONG time since I've had an RTFM response. ;)
I'm clearly missing something re my understanding of how this was all supposed to work. There is nothing in Oracle Parallel Server that explicitly allows you to control over which interface the DLM/PCM traffic will occur - so my assumption was that it needed to to be configured at the cluster level. I had not read the doc on hyperfabric, so appreciate the pointer. Thank-you also for the tip on the SG patch.

Regards,
Kirk.

[Carsten Krege]

In the new versions of SG OPS that use the Group Membership Services for DLM/OPS communications it is out of the control of SGOPS to specify the DLM lan (as this code is now owned by Oracle). HP only provides a shared library to give OPS an interface to the cluster software.

But the following might help: The lan ip address used for the DLM traffic is the one returned by a call to gethostbyname(). Therefore the solution is to cause this to return the "right" value. This can be achieved by modifying your DNS config in the right way (with help of /etc/hosts and /etc/nsswitch.conf ).

Not very nice, but possible.

Carsten

[Kirk McGowan]

Interestingly enough, in spite of the fact that the DLM is now integrated with OPS itself and not used as part of SGOPS, there is no direct means in OPS 8i to control which interface DLM traffic uses. (GMS was actually introduced in 8.0, and went away in 8i). So as you suggested, it appears the only way to force OPS to use hyperfabric, at least with 8i, is to mess around to get gethostbyname() to return the name associated with the clic interface. As you said, not very nice, but hopefully will work. There is apparently an Oracle init.ora parameter in 9i to provide more explicit control.
Appreciate your input.
Regards,
Kirk.

[Carsten Krege]

One note: In fact all of the current OPS server releases still support Group Member Ship services. There are currently three flavours available:

(old) Reference Implementation (DLM integrated in SGOPS): 8.0.6 32bit (last supported Oracle version that work with HP DLM).

8i native implementations (DLM in OPS) using HP's ogmsd daemon (nmapi1 library): Oracle 8.0.6

8.1.x native implementations (DLM in OPS) using HP's cmgmsd daemon (nmapi2 library): Oracle 8.1.6 and 8.1.7

So the concept of GMS has not changed, but the API provided by HP has.

Carsten