topic Re: Issue with missing vg for cmcheckconf in Operating System - HP-UX

Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 03:32:02 GMT

Hi Folks,

I was trying to cmcheckconf my cluster configuration (nothing special a two node cluster with a shared vg (smsvg00). The vg is the cluster lock and is active on the node I'm running the cmcheckconf.

The output states:

Begin cluster verification...

Error: Volume Group /dev/smsvg00 is not found on any nodes in the cluster.
cmcheckconf : Unable to reconcile configuration file /etc/cmcluster/cmclconfig.ascii
with discovered configuration information.

But a vgdisplay smsvg00 tells me:

tbesm1a# vgdisplay smsvg00
--- Volume groups ---
VG Name /dev/smsvg00
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 18
Open LV 18
Max PV 64
Cur PV 4
Act PV 4
Max PE per PV 3199
VGDA 8
PE Size (Mbytes) 64
Total PE 8796
Alloc PE 8083
Free PE 713
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

Any idea ?

Thanks,

Mike

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 03:39:37 GMT

Is the vg listed in your cluster ascii file?
If not, insert it as the last line and try again

Re: Issue with missing vg for cmcheckconf

G. Vrijhoeven — Fri, 23 Jan 2004 03:43:34 GMT

Hi Mike,

Take a look at this threat:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=10355

HTH,

Gideon

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 03:44:48 GMT

Hi,

both nodes show on the bottom:

VOLUME_GROUP /dev/smsvg00

sigh...

Mike

Re: Issue with missing vg for cmcheckconf

Sridhar Bhaskarla — Fri, 23 Jan 2004 03:49:18 GMT

Hi Mike,

How come it is exclusive already if it is not part of serviceguard yet?.

-Sri

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 03:54:15 GMT

Hmm
Interesting
What might be a good test is tio do:
cmquerycl -v -C test.ascii -n node1 -n node2
and then examine the resultant ascii file to see what that shows regarding VG's
Failing that, you may want to check the lvmtab files, as well as /dev/smsvg00 directories on each node.

Last test would be to halt the cluster
do vgchange -c n; vgchange -a y; cmcheckconf -v -C .............
and see what happens then.
Out of interest, what version of SG and which patch for SG is installed?
do:
what /usr/lbin/cmcld

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 03:54:49 GMT

Hi Sri,

the cluster was already configured but we had some HW issues on a shared disk array so that we had to apply some minor changes.

What I found is that the cmquerycl shows:

Volume groups on tbesm1a:
/dev/vg00/lvol1 FS MOUNTED /stand
/dev/vg00/lvol2
/dev/vg00/lvol3 FS MOUNTED /
/dev/vg00/lvol4 FS MOUNTED /home
/dev/vg00/lvol5 FS MOUNTED /opt
/dev/vg00/lvol6 FS MOUNTED /tmp
/dev/vg00/lvol7 FS MOUNTED /usr
/dev/vg00/lvol8 FS MOUNTED /var
/dev/vg00/SwapVol1
/dev/backupsmsvg00/DB01VolFS
/dev/backupsmsvg00/HomeVol
/dev/backupsmsvg00/NetscapeVol
/dev/backupsmsvg00/OmniVol
/dev/backupsmsvg00/OracleVol
/dev/backupsmsvg00/OrbixVol
/dev/backupsmsvg00/SmsVol
/dev/backupsmsvg00/SmsBillingVol
/dev/backupsmsvg00/SmsRptVol
/dev/backupsmsvg00/ApacheVol
/dev/backupsmsvg00/DB04VolFS
/dev/backupsmsvg00/DB05VolFS
/dev/backupsmsvg00/SmsLogVol
/dev/backupsmsvg00/DB02VolFS
/dev/backupsmsvg00/DB05ArchVol
/dev/backupsmsvg00/DB03VolFS
/dev/backupsmsvg00/DB06Vol

which is wrong since the backupsmsvg00 is a different VG.

Any idea on how to change this ?

Thanks,

Mike

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 03:57:20 GMT

The cmcld is:
/usr/lbin/cmcld:
HP92453-02A.10.20 HP-UX SYMBOLIC DEBUGGER (END.O) $Revision: 74.03 $
Build date: Sun Nov 25 15:15:30 PST 2001
Build id: ibld_sgops_a1113_patch
A.11.13 Date: 11/06/2001; PATCH: PHSS_25124
Daemon
Config DB
Cluster Monitor
Command Srv
CommunicationSrv
Config
Dlm
Local Comm
Network Sensor
Package Manager
Remote Comm
Quorum Module
API
Service Sensor
Cluster LVM
Status DB
Sync
Util

Re: Issue with missing vg for cmcheckconf

G. Vrijhoeven — Fri, 23 Jan 2004 03:58:48 GMT

Hi Mike.

If both nodes show the disk i think it iis the cluster lock disk. If not you can at the vgname only at the bottom.

Gideon

Re: Issue with missing vg for cmcheckconf

Dietmar Konermann — Fri, 23 Jan 2004 04:01:41 GMT

Hmm...
ServiceGuard takes the VG's unique ID from lvmtab and tries to match it with configured disks. I attached a script called cmpdisks. Could you please run it whith both node names names as args and post the results?

./cmpdisks node1 node2 >cmpdisks.out

Best regards...
Dietmar.

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 04:02:13 GMT

what does ll /dev/smsvg00 show on each node?
Also, do ll /dev/*vg*/group and see what this shows.

Re: Issue with missing vg for cmcheckconf

Sridhar Bhaskarla — Fri, 23 Jan 2004 04:02:53 GMT

I would agree to the clean approach.

1. Halt the cluster.
2. Take the vg out of the cluster (vgchange -c n)
3. Activate the vg non-exclusive (vgchange -a y)
4. Apply the configuration.

-Sri

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 04:03:30 GMT

The syslog shows:

WARNING: Cluster lock disk /dev/dsk/c10t0d0 has failed: I/O error

Actually the lvmtab does not include it:

tbesm1a# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t6d0
/dev/dsk/c2t6d0
/dev/backupsmsvg00
/dev/dsk/c10t12d5
/dev/dsk/c12t12d5
/dev/dsk/c10t12d6
/dev/dsk/c12t12d6
/dev/dsk/c10t12d7
/dev/dsk/c12t12d7
/dev/smsvg00
/dev/dsk/c10t0d4
/dev/dsk/c10t0d5
/dev/dsk/c10t0d6
/dev/dsk/c10t0d7
/dev/dsk/c12t0d4
/dev/dsk/c12t0d5
/dev/dsk/c12t0d6
/dev/dsk/c12t0d7

The PV exists but is not recorded in any vg

Re: Issue with missing vg for cmcheckconf

Dietmar Konermann — Fri, 23 Jan 2004 04:06:51 GMT

Just saw yur latest posting... could it be possible that backupsmsvg00 is a BCV copy of smsvg00? Then it may have the same VGID, unlesse you use vgchgid after split-of.

What's the result of this script:

strings -t d /etc/lvmtab |
grep /dev |
grep -v /dev/dsk |
while read offset path; do
xd -An -j$(($offset+1024)) -N8 -tuL /etc/lvmtab |
read vgid1 vgid2
echo $path $vgid1-$vgid2
done

Best regards...
Dietmar.

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 04:16:01 GMT

Hi Dietmar, yep,

that one was used for backup purposes using BC feature on a a va7400 array.

Your script aoutputs:

/dev/vg00 631339383-1042130551
/dev/backupsmsvg00 628339303-1042137717
/dev/smsvg00 628339303-1042137717

which is what you're afraid of I believe.

Thanks,

Mike

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 04:17:58 GMT

also, does a vgcfgrestore -n smsvg00 -l to see if that disk is actually listed.
It loks like your hardware changes may have had some nasty side effects, but what may be worth checking (if you are able) is to see WHEN you started getting the error message saying the lock disc is unavailable, as this may have been happening BEFORE you made any hardware changes.

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 04:19:15 GMT

So you have 2 VG's with tehsame VGID, Dietmar has got it right

Re: Issue with missing vg for cmcheckconf

Michele (Mike) Alberton — Fri, 23 Jan 2004 04:21:51 GMT

Yes Dietmar was right,

how should I proceed ?

Thanks !

Mike

Re: Issue with missing vg for cmcheckconf

Dietmar Konermann — Fri, 23 Jan 2004 04:31:03 GMT

Mike,

I think, vgexport'ing backupsmsvg00 from both nodes would help. This removes the duplicate entry from lvmtab... hopefully Serviceguard is happy again then. :)

If the problem persists, you should use vgchgid(1M) to change the VGID on all disks of backupsmsvg00. Generally I would recommend to do this after any BC split operation. Serviceguard really gets confused by these duplicates.

BTW, you should also choose a configured PV as cluster lock disk.

Best regards...
Dietmar.

Re: Issue with missing vg for cmcheckconf

melvyn burnard — Fri, 23 Jan 2004 04:32:03 GMT

you would need to either merge the BCV back in, or run eth vgchgid on the NEWER (backup) vg.
man vgchgid