Dual lock disk issue

[Joe Short]

I am configuring an extended (campus) cluster for a client. The environment consists of
a pair of rp4440 server, and a pair of CX500 arrays. I am trying to configure a first and second lock disk. I have created the volume groups /dev/vglock1 and /dev/vglock2 on both hosts in the cluster, and entered them into the cluster config file. But when ever I execute a cmchekconf, I get the following error

"Error: First physical volume name and Second physical volume name specified correspond to the same physical device. Choose different first and second physical volume names."

These are different LUNs on the arrays, and are configured in different volume groups. I suspect this may be linked to another issue. When ever I use SAM, and list disk devices, all I see listed are the 2 array service processors, none of the LUNs are displayed.
My back is against the wall here, as the project is starting to get bogged down on this. Any help will be greatly apreciated, and points awarded.

[Steven E. Protter]

Hi Joe,

A Servicguard cluster lock disk needs to be accessible by both systems.

The disk array but present the disk to both systems (I assume this is fiber).

Once the array is set up, you'll need to do the following to make the drive visible to both systems.

insf -C
ioscan

or boot the box.

Once the disk is visible, you may need to run pvcreate to set it up for use by a logical volume.

I think once its visible, you're all set. Sorry I can't help more with making it visible from the array setup standpoint.

Regard,

SEP

[Joe Short]

The LUNs are visible, and accessible. I have configure them into volume groups. The problem appears when I try to verify my cluster configuration

cmcheckconf -C /etc/cmcluster/cluster.ascii

The check fails witht the error"

"Error: First physical volume name and Second physical volume name specified correspond to the same physical device. Choose different first and second physical volume names."

I need to determine why the software thinks this is one device, when in fact there are 2 separate LUNs, one on each array.

[Sanjay_6]

Hi Joe,

Can you post your ascii conf file for us to take a peek.

Thanks
Sanjay

[Steven E. Protter]

Visible on both boxes. Configurable into a vg.

good.

Please post the logs from servicegauard.

SEP

[Richard Perez]

which LUN does /dev/vglock1 points to in the first and second server?

Sounds like vglock1 in one server is the same LUN that the other server is seeing as vglock2.

[Joe Short]

Attched is the conf.ascii file.
The arrays are CX500, the servers are rp4440,
we are running HP-UX 11.11, with MCSG 11.16

[melvyn burnard]

`As a test, why not remove the entry for one of the VGlock vg's, and it relevant PV's, from the ascii file, and then retry the apply. Then add that VG/PV set back in and remove the other VG/PV set, and again try to apply the binary.
That may give you a pointer to an offending incorrect PV pair.

You could also do an adb on each PV to check the LVM info and see if they are indeed the same disc:

#echo â 0x2000?4X â | adb /dev/dsk/c0t6d0

Obviously replace the disk path with your disks, the sort of output you would get would be:
2000: 4C564d52 45433031 78261D3A 391C79BF where the last two words are the unique physical volume id.

Have you installed the latest available patch for 11.16? PHSS_31071 for 11.11 of HP-UX.

Also, you may find that it would be easier to use the Quorum Server product rather than the cluster lock disc.

One thing to note, HP do not support the Clariion disc arrays for use in Servicegu

[Duncan Edmonstone]

Joe,

The first thing to do is to look at the LVM record in the PVRA of these disks. Run:

echo 0x2008?4D|adb /dev/dsk/cXtYdZ

for each of the disk device files of your cluster lock disks. This should provide you with the following information:

2008: CPU-ID PV-ID CPU-ID VG-ID


So devices c10t2d7 and c18t2d7 should match, and c9t2d6 and c17t2d6 should match.


HTH

Duncan

[Joe Short]

I am able to successfully create a cluster with a single lock disk. However, this is an extended cluster so I need either 2 lock disks, or a third location to run Quorum Server. As the client has only 2 data centers, the quorum server would become a single point of failure.
I do have the patch PHSS_31071 installed as well as PHSS_31073, but they have created their own issues as they require identd, which is not running on these servers.
The results of the adb (attached)indicate that these are different devices.
How does Service Guard determine if these are different devices? Does it read lif information? Does it use the disk serial number? If I knew that much, I'd have a direction to focus my effort in.

[Duncan Edmonstone]

Joe,

I'm not really trusting the info we're getting here - you need to explicitly use an uppercase D in the adb command:

echo "0x2008?4D" | adb /dev/dsk/cXtYdZ

Cheers

Duncan

[Richard Perez]

Joe
Not the answer to your original question, but maybe a useful hint:
you don't need 2 lock disks if you are configuring an extended cluster. If you have dual lock disks, when you have a disaster in one datacenter the second node will TOC as it won't be able to secure both lock disks. You could think that mirrored lock disks could avoid that, but in such case you are increasing the chances for a split brain situation.

I would opt for single lock disk or quorum server as the chances for data corruption due to a split-brain condition are the lowest.

[melvyn burnard]

Serviceguard looks explicitly at the LVM information on each PV, therefore if these are checked and found to be the same, then this is seen as the same PV.

[Joe Short]

I had originally configure the environment with a mirrored lock disk. Between the time I built the environment, and delivery this week, I was quite busy, and lost my focus.
After testing with the mirrored lock disk, we found the cluster resonded as desired. We simulated a lost link by powering off the switches on one side of the cluster. The result was that the alternate node took no action to start the package, and registered the primary as being down. After the links where restored, we were able to reform the cluster with a cmrunnode command.
The issue with the disks being the same was never resolved. It is quite curious as the disks where different LUNs in different raid groups (one RAID 5, and one RAID 1) and configured into different volume groups.
The reslts of the adb command (using capitol D) showed that they were indeed different disks. I suspect that something within the SAN environment, the switches or the CX array was somehow blocking or providing misinformation when Service Guard polled the devices. If I knew exactly how MC/Service Guard polled the disks, and what it used to determine the identity of the disks, I could probably figure it out.
But the cluster is operational, and that is the major concern.
Thanks to all that provided assistance.