vxconfigd NOTICE Split Brain

[Muhammad Ahmad]

Hi Gurus,

A brief relevant information of our setup:

O/S:
----
B.11.23 Base Quality Pack Bundle for HP-UX 11i v2, June 2008

Products:
---------
A.11.18.00 Serviceguard Extension for RAC Toolkit SD Product
A.01.01 HP Serviceguard Cluster File System for RAC
4.1 VERITAS Oracle Disk Manager

It's 2 nodes cluster (MULTI_NODE_PACKAGES).

problem:
--------
After a disaster due to over-temperature, some of the packages are unable to start.

workaround done:
----------------
enabled the failed packages locally and globally (cmmmodpkg -e, cmmodpkg -e -n ), then tried to start but no gain.
reboot the machines one by one, then start the cluster and packages, no gain.

diagnostics:
============

from syslog:
-----------

VxVM vxconfigd NOTICE V-5-1-7900 CVM_VOLD_CONFIG command received
VxVM vxconfigd NOTICE V-5-1-7899 CVM_VOLD_CHANGE command received

VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm c4t1d3
VxVM vxconfigd ERROR V-5-1-569 Disk group dg_ora, Disk c4t1d3: Cannot auto-import group:
Serial Split Brain detected. Run vxsplitlines

VxVM vxconfigd NOTICE V-5-1-7961 establishing cluster for seqno = 0x16d3507.
VxVM vxconfigd NOTICE V-5-1-8059 master_new: no joiners/rejoiners
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm c4t1d3
VxVM vxconfigd ERROR V-5-1-569 Disk group dg_ora, Disk c4t1d3: Cannot auto-import group:
Serial Split Brain detected. Run vxsplitlines
VxVM vxconfigd NOTICE V-5-1-8059 master_new: cluster startup
VxVM vxconfigd NOTICE V-5-1-8061 master_new: no joiners
VxVM vxconfigd NOTICE V-5-1-4123 cluster established successfully

from package log:
-----------------

########### Node "NodeA": Starting package ###########
Error: "dg_ora" was not imported
ERROR: Failed to activate shared disk group "dg_ora" in "sw" mode.

Other observations:
-------------------
ioscan showing all devices as claimed.

your input required.


-Br
-Muhammad Ahmad

[VK2COT]

Hello,

The Serial Split Brain (SSB) condition
exists because Veritas Volume Manager
increments the serial ID in the disk media
record of each imported disk in all the disk
group configurations on those disks.

The value that is stored in the configuration
database represents the serial ID that the
disk group expects a disk to have. The serial
ID that is stored in a disk's private region
is considered to be its actual value.

If some disks went missing from the disk
group (physical connection problem
or power failure) and those disks were
imported by another host, the serial IDs
for the disks in their copies of the
configuration database, and also in
each disk's private region, are updated
separately on that host. When the disks are
then reimported into the original shared
disk group, the actual serial IDs on the
disks do not agree with the expected values
from the configuration copies on other disks
in the disk group.

The disk group cannot be reimported because
the databases do not agree on the actual and
expected serial IDs.

You must choose which configuration database
to use. This is a true serial split brain
condition, which Volume Manager cannot
correct automatically.

To resolve conflicting configuration
information, you must decide which disk
contains the correct version of the disk group configuration database. To assist you
in doing this, you can run vxsplitlines
command to show the actual serial ID on each
disk in the disk group and the serial ID
that was expected from the configuration database. For each disk, the command also
shows the vxdg command that you must run to
select the configuration database copy on
that disk as being the definitive copy to
use for importing the disk group.

I also hope that you have patch PHCO_36111,
which might be important.

For more details on vxsplitlines,
take a look at Veritas Volume Manager
Administrator's Guide.

Cheers,

VK2COT