Issue with missing vg for cmcheckconf

[Michele (Mike) Alberton]

Hi Folks,

I was trying to cmcheckconf my cluster configuration (nothing special a two node cluster with a shared vg (smsvg00). The vg is the cluster lock and is active on the node I'm running the cmcheckconf.

The output states:


Begin cluster verification...

Error: Volume Group /dev/smsvg00 is not found on any nodes in the cluster.
cmcheckconf : Unable to reconcile configuration file /etc/cmcluster/cmclconfig.ascii
with discovered configuration information.

But a vgdisplay smsvg00 tells me:

tbesm1a# vgdisplay smsvg00
--- Volume groups ---
VG Name /dev/smsvg00
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 18
Open LV 18
Max PV 64
Cur PV 4
Act PV 4
Max PE per PV 3199
VGDA 8
PE Size (Mbytes) 64
Total PE 8796
Alloc PE 8083
Free PE 713
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0



Any idea ?

Thanks,

Mike

[melvyn burnard]

Is the vg listed in your cluster ascii file?
If not, insert it as the last line and try again

[G. Vrijhoeven]

Hi Mike,

Take a look at this threat:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=10355

HTH,

Gideon

[Michele (Mike) Alberton]

Hi,

both nodes show on the bottom:

VOLUME_GROUP /dev/smsvg00

sigh...

Mike

[Sridhar Bhaskarla]

Hi Mike,

How come it is exclusive already if it is not part of serviceguard yet?.

-Sri

[melvyn burnard]

Hmm
Interesting
What might be a good test is tio do:
cmquerycl -v -C test.ascii -n node1 -n node2
and then examine the resultant ascii file to see what that shows regarding VG's
Failing that, you may want to check the lvmtab files, as well as /dev/smsvg00 directories on each node.

Last test would be to halt the cluster
do vgchange -c n; vgchange -a y; cmcheckconf -v -C .............
and see what happens then.
Out of interest, what version of SG and which patch for SG is installed?
do:
what /usr/lbin/cmcld

[Michele (Mike) Alberton]

Hi Sri,

the cluster was already configured but we had some HW issues on a shared disk array so that we had to apply some minor changes.

What I found is that the cmquerycl shows:

Volume groups on tbesm1a:
/dev/vg00/lvol1 FS MOUNTED /stand
/dev/vg00/lvol2
/dev/vg00/lvol3 FS MOUNTED /
/dev/vg00/lvol4 FS MOUNTED /home
/dev/vg00/lvol5 FS MOUNTED /opt
/dev/vg00/lvol6 FS MOUNTED /tmp
/dev/vg00/lvol7 FS MOUNTED /usr
/dev/vg00/lvol8 FS MOUNTED /var
/dev/vg00/SwapVol1
/dev/backupsmsvg00/DB01VolFS
/dev/backupsmsvg00/HomeVol
/dev/backupsmsvg00/NetscapeVol
/dev/backupsmsvg00/OmniVol
/dev/backupsmsvg00/OracleVol
/dev/backupsmsvg00/OrbixVol
/dev/backupsmsvg00/SmsVol
/dev/backupsmsvg00/SmsBillingVol
/dev/backupsmsvg00/SmsRptVol
/dev/backupsmsvg00/ApacheVol
/dev/backupsmsvg00/DB04VolFS
/dev/backupsmsvg00/DB05VolFS
/dev/backupsmsvg00/SmsLogVol
/dev/backupsmsvg00/DB02VolFS
/dev/backupsmsvg00/DB05ArchVol
/dev/backupsmsvg00/DB03VolFS
/dev/backupsmsvg00/DB06Vol

which is wrong since the backupsmsvg00 is a different VG.

Any idea on how to change this ?

Thanks,

Mike

[Michele (Mike) Alberton]

The cmcld is:
/usr/lbin/cmcld:
HP92453-02A.10.20 HP-UX SYMBOLIC DEBUGGER (END.O) $Revision: 74.03 $
Build date: Sun Nov 25 15:15:30 PST 2001
Build id: ibld_sgops_a1113_patch
A.11.13 Date: 11/06/2001; PATCH: PHSS_25124
Daemon
Config DB
Cluster Monitor
Command Srv
CommunicationSrv
Config
Dlm
Local Comm
Network Sensor
Package Manager
Remote Comm
Quorum Module
API
Service Sensor
Cluster LVM
Status DB
Sync
Util

[G. Vrijhoeven]

Hi Mike.

If both nodes show the disk i think it iis the cluster lock disk. If not you can at the vgname only at the bottom.

Gideon

[Dietmar Konermann]

Hmm...
ServiceGuard takes the VG's unique ID from lvmtab and tries to match it with configured disks. I attached a script called cmpdisks. Could you please run it whith both node names names as args and post the results?

./cmpdisks node1 node2 >cmpdisks.out

Best regards...
Dietmar.

[melvyn burnard]

what does ll /dev/smsvg00 show on each node?
Also, do ll /dev/*vg*/group and see what this shows.

[Sridhar Bhaskarla]

I would agree to the clean approach.

1. Halt the cluster.
2. Take the vg out of the cluster (vgchange -c n)
3. Activate the vg non-exclusive (vgchange -a y)
4. Apply the configuration.

-Sri

[Michele (Mike) Alberton]

The syslog shows:

WARNING: Cluster lock disk /dev/dsk/c10t0d0 has failed: I/O error

Actually the lvmtab does not include it:

tbesm1a# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t6d0
/dev/dsk/c2t6d0
/dev/backupsmsvg00
/dev/dsk/c10t12d5
/dev/dsk/c12t12d5
/dev/dsk/c10t12d6
/dev/dsk/c12t12d6
/dev/dsk/c10t12d7
/dev/dsk/c12t12d7
/dev/smsvg00
/dev/dsk/c10t0d4
/dev/dsk/c10t0d5
/dev/dsk/c10t0d6
/dev/dsk/c10t0d7
/dev/dsk/c12t0d4
/dev/dsk/c12t0d5
/dev/dsk/c12t0d6
/dev/dsk/c12t0d7

The PV exists but is not recorded in any vg

[Dietmar Konermann · Accepted solution]

Just saw yur latest posting... could it be possible that backupsmsvg00 is a BCV copy of smsvg00? Then it may have the same VGID, unlesse you use vgchgid after split-of.

What's the result of this script:

strings -t d /etc/lvmtab |
grep /dev |
grep -v /dev/dsk |
while read offset path; do
xd -An -j$(($offset+1024)) -N8 -tuL /etc/lvmtab |
read vgid1 vgid2
echo $path $vgid1-$vgid2
done

Best regards...
Dietmar.

[Michele (Mike) Alberton]

Hi Dietmar, yep,

that one was used for backup purposes using BC feature on a a va7400 array.

Your script aoutputs:

/dev/vg00 631339383-1042130551
/dev/backupsmsvg00 628339303-1042137717
/dev/smsvg00 628339303-1042137717

which is what you're afraid of I believe.

Thanks,

Mike

[melvyn burnard]

also, does a vgcfgrestore -n smsvg00 -l to see if that disk is actually listed.
It loks like your hardware changes may have had some nasty side effects, but what may be worth checking (if you are able) is to see WHEN you started getting the error message saying the lock disc is unavailable, as this may have been happening BEFORE you made any hardware changes.

[melvyn burnard]

So you have 2 VG's with tehsame VGID, Dietmar has got it right

[Michele (Mike) Alberton]

Yes Dietmar was right,

how should I proceed ?

Thanks !

Mike

[Dietmar Konermann]

Mike,

I think, vgexport'ing backupsmsvg00 from both nodes would help. This removes the duplicate entry from lvmtab... hopefully Serviceguard is happy again then. :)

If the problem persists, you should use vgchgid(1M) to change the VGID on all disks of backupsmsvg00. Generally I would recommend to do this after any BC split operation. Serviceguard really gets confused by these duplicates.

BTW, you should also choose a configured PV as cluster lock disk.

Best regards...
Dietmar.

[melvyn burnard]

you would need to either merge the BCV back in, or run eth vgchgid on the NEWER (backup) vg.
man vgchgid

[Michele (Mike) Alberton]

Nice shot folks !

Really thanks a lot.

I vgexported the backupsmsvg00 to clean lvmtab (I don't think I need that volume for the timebeing) and then halted the cluster and reapplied the conf (I needed to change the pvlock as well).

Everything seems good so far, I appreciated !

Cheers,

Mike