LVM errors related to incomplete swap config?

[Ken O'Neil]

After using SAM to review the LV info on one of our 11.11 servers (SERVERA), only LV info for vg00 would display. I checked the SAM output and ran the SAM command manually and got this error:

# /usr/sam/lbin/vginfo -d
lvlnboot: Couldn't query volume group "/dev/vg01".
Possible error in the Volume Group minor number; Please check and make sure the group minor number is unique.
(repeating for vg02-09)

vgdisplay returned all the correct info for vg00 but these errors for vg01-09:

vgdisplay: Couldn't query volume group "/dev/vg01".
Possible error in the Volume Group minor number; Please check and make sure the
group minor number is unique.
vgdisplay: Cannot display volume group "/dev/vg01".
(repeating for vg02-09)

Thinking that lvmtab was off, I ran these commands:

# cd /etc
# mv lvmtab lvmtab.last
# vgscan -av
Creating "/etc/lvmtab".
Physical Volume "/dev/dsk/c0t0d0" contains no LVM information

vgscan: has no correspoding valid raw device file under /dev/rdsk.
Verification of unique LVM disk id on each disk in the volume group
/dev/vg00 failed.

/dev/vg01
/dev/dsk/c14t0d0
/dev/dsk/c14t0d1

(repeating all of the correct info for vg02-09)

Scan of Physical Volumes Complete.

Now vgdisplay is showing this:
# vgdisplay -v vg00
vgdisplay: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "vg00".

# vgdisplay
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 1279
VGDA 4
PE Size (Mbytes) 4
Total PE 2558
Alloc PE 2556
Free PE 2
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

(repeating all of the correct info for vg02-09)

So to make sure there were no disk issues I ran ioscan and all disks display as CLAIMED.

But going back to the original SAM vginfo command, and the end of the output there is a reference to /dev/vg00/lvol2, which is my primary swap file which got me thinking about swap configurations. This server (SERVERA) has no secondary swap configured, but using SAM I compared the swap configuration on this server to another identical server (SERVERB) where a vendor configured a secondary swap file, and the configuration was the same. Looks like SAM is pulling this info from /etc/fstab because on both servers fstab contains the following line:
/dev/vg00/swap2 ... swap pri=1 0 0
/dev/vg00/swap2 ... swap pri=1 0 0

But I know that a secondary swap is not configured on SERVERA but is on SERVERB, at least according to swapinfo. This led to more comparison between SERVERA and SERVERB and I found these differences on SERVERB:

in /dev/vg00:
cr--r----- 1 root sys 64 0x00000a Dec 7 20:26 rswap2
br--r----- 1 root sys 64 0x00000a Dec 7 20:36 swap2

So it appears that a secondard swap file configuration was started on SERVERA but was either incomplete or failed. Since I'm not the admin who attempted this, I can't speak to the method used but it appears that this incomplete or misconfigured secondary swap on SERVERA could be related.

I've attached a text file with complete output of all the commands I've referred to above.

Here are my questions:
Could the group minor number errors be caused by the incomplete/misconfigured/corrupted secondary swap config?
What is the best way to proceed with cleaning this up and resolving the LVM errors?

Thanks in advance!

[A. Clay Stephenson]

First things first:

ls -l /dev/vg*/group

That will determine if the minor device numbers are indeed unique. Post that output.

Next 'bdf' and post that output.

My best guess is that many of your VG's are not activated. Is this box running MC/SG? Your lvmrc file may have auto vg activation set to 0 (which is normal for MC/SG) but there must be explicit vgchange commands for those VG's not under the direct control of a package.

[Ken O'Neil]

# ls -l /dev/vg*/group
crw-r----- 1 root sys 64 0x000000 Jul 28 2004 /dev/vg00/group
crw-r--r-- 1 root sys 64 0x010000 Oct 14 2004 /dev/vg01/group
crw-r--r-- 1 root sys 64 0x020000 Oct 14 2004 /dev/vg02/group
crw-r--r-- 1 root sys 64 0x030000 Oct 14 2004 /dev/vg03/group
crw-r--r-- 1 root sys 64 0x040000 Oct 14 2004 /dev/vg04/group
crw-r--r-- 1 root sys 64 0x050000 Oct 14 2004 /dev/vg05/group
crw-r--r-- 1 root sys 64 0x060000 Oct 21 2004 /dev/vg06/group
crw-r--r-- 1 root sys 64 0x070000 Oct 21 2004 /dev/vg07/group
crw-r--r-- 1 root sys 64 0x080000 Sep 20 12:44 /dev/vg08/group
crw-r--r-- 1 root sys 64 0x090000 Jan 18 00:57 /dev/vg09/group

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 204800 104488 99552 51% /
/dev/vg00/lvol1 298928 44712 224320 17% /stand
/dev/vg09/lv_ws 157245440 126251912 30751432 80% /workspace
/dev/vg00/lvol8 4710400 632632 4047984 14% /var
/dev/vg00/lvol7 2105344 1510392 590312 72% /usr
/dev/vg05/lv_u03 41910272 39745632 2147800 95% /u03
/dev/vg04/lv_u02 10469376 2119670 8088782 21% /u02
/dev/vg03/lv_u01 10469376 3537498 6715270 35% /u01
/dev/vg00/lvol4 1048576 13720 1027656 1% /tmp
/dev/vg02/lv_sw 10469376 1465996 8722064 14% /sw
/dev/vg07/lv_src 20955136 6634344 14131428 32% /source
/dev/vg00/lvol6 2818048 2001544 810824 71% /opt
/dev/vg01/lv_ifas 10469376 4155534 6121386 40% /ifas
/dev/vg00/lvol5 524288 4552 515720 1% /home
/dev/vg08/lv_bu 125796352 75626784 49777656 60% /backup
/dev/vg00/lvol9 516096 4150 479998 1% /asprsp
/dev/vg06/lv_oal 20938752 4256 20608108 0% /archlogs
/dev/vg08/lv_obu 41918464 3616 41587464 0% /ora_backup

No MCSG.

[Ken O'Neil]

Digging further through ITRC, I found this:
http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062922068

So I ran this on both SERVERA and SERVERB:

strings -t d /etc/lvmtab | grep /dev | grep -v /dev/dsk | while read offset
path; do xd -An -j$(($offset+1024)) -N8 -tx /etc/lvmtab | read vgid1 vgid2;
echo $path $vgid1/$vgid2; done

...and got this with the last version of lvmtab in place:

SERVERA:
/dev/vg00 60404a43/4107ccfa
/dev/vg01 60404a43/416ee285
/dev/vg02 60404a43/416ee2a2
/dev/vg03 60404a43/416ee2b0
/dev/vg04 60404a43/416ee2db
/dev/vg05 60404a43/416ee301
/dev/vg06 60404a43/4178365d
/dev/vg07 60404a43/4178367b
/dev/vg08 60404a43/42828e2d

SERVERB:
/dev/vg00 60404a43/4107ccfa
/dev/vg01 60404a44/416ef1b2
/dev/vg02 60404a44/416ef1ce
/dev/vg03 60404a44/416ef1fd
/dev/vg04 60404a44/416ef20b
/dev/vg05 60404a44/416ef23b
/dev/vg06 60404a44/41783a92
/dev/vg07 60404a44/41783aae
/dev/vg08 60404a44/433073ae
/dev/vg09 60404a44/43ce1da8

I restored an older saved version of lvmtab and got this though:

SERVERA:
/dev/vg00 60404a43/4107ccfa
/dev/vg01 60404a43/416ee285
/dev/vg02 60404a43/416ee2a2
/dev/vg03 60404a43/416ee2b0
/dev/vg04 60404a43/416ee2db
/dev/vg05 60404a43/416ee301
/dev/vg06 60404a43/4178365d
/dev/vg07 60404a43/4178367b
/dev/vg08 60404a43/42828e2d
... although this doesn't include vg09, the last one created on Jan 18.

So it seems pretty clear that this is a VGID issue, as someone appears to have copied the /etc/lvmtab from SERVERB to SERVERA at somepoint.

Repeating my steps to create a new lvmtab and re-running the script above yields this:

SERVERA:
/dev/vg01 60404a43/416ee285
/dev/vg02 60404a43/416ee2a2
/dev/vg03 60404a43/416ee2b0
/dev/vg04 60404a43/416ee2db
/dev/vg05 60404a43/416ee301
/dev/vg06 60404a43/4178365d
/dev/vg07 60404a43/4178367b
/dev/vg08 60404a43/4330674f
/dev/vg09 60404a43/43ce066c

SERVERB:
/dev/vg00 60404a43/4107ccfa
/dev/vg01 60404a44/416ef1b2
/dev/vg02 60404a44/416ef1ce
/dev/vg03 60404a44/416ef1fd
/dev/vg04 60404a44/416ef20b
/dev/vg05 60404a44/416ef23b
/dev/vg06 60404a44/41783a92
/dev/vg07 60404a44/41783aae
/dev/vg08 60404a44/433073ae
/dev/vg09 60404a44/43ce1da8

This now leaves two issues:
1) vg00 doesn't come up in the scan. I get this error:
Physical Volume "/dev/dsk/c0t0d0" contains no LVM information

vgscan: has no correspoding valid raw device file under /dev/rdsk.
Verification of unique LVM disk id on each disk in the volume group
/dev/vg00 failed.

2) It appears that the current vg00 VGID on SERVERB is the same as a previous VGID for vg00 on SERVERA, although no errors have been seen on SERVERB.

What would be the best way to proceed?

[A. Clay Stephenson]

1) Do an ioscan -fn to make certain that all the disks are "seen" by the system.
2) insf # add any missing device nodes
3) mv /etc/lvmtab /etc/lvmtab.sav and then do a vgscan to rebuild /etc/lvmtab.

[Mridul Shrivastava]

It seems that u did dd from one server and installed the seconf disk on SERVERB so both server have same VGID. Is it possible to reboot the server? Then boot in lvm maintenance mode then export and import vg00.
U also run "vgchid" to change the VGID as vg shouldn't be activated while executing this.

[Ken O'Neil]

Clay,

ioscan didn't find anything missing:
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 0 0/0/3/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 4 0/1/1/1.0.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC
/dev/dsk/c3t0d0 /dev/rdsk/c3t0d0
disk 3 255/255/0/0.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d0 /dev/rdsk/c14t0d0
disk 5 255/255/0/0.1 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d1 /dev/rdsk/c14t0d1
disk 6 255/255/0/0.2 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d2 /dev/rdsk/c14t0d2
disk 7 255/255/0/0.3 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d3 /dev/rdsk/c14t0d3
disk 8 255/255/0/0.4 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d4 /dev/rdsk/c14t0d4
disk 9 255/255/0/0.5 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d5 /dev/rdsk/c14t0d5
disk 10 255/255/0/0.6 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d6 /dev/rdsk/c14t0d6
disk 11 255/255/0/0.7 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t0d7 /dev/rdsk/c14t0d7
disk 12 255/255/0/1.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d0 /dev/rdsk/c14t1d0
disk 13 255/255/0/1.1 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d1 /dev/rdsk/c14t1d1
disk 14 255/255/0/1.2 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d2 /dev/rdsk/c14t1d2
disk 15 255/255/0/1.3 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d3 /dev/rdsk/c14t1d3
disk 16 255/255/0/1.4 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d4 /dev/rdsk/c14t1d4
disk 17 255/255/0/1.5 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d5 /dev/rdsk/c14t1d5
disk 18 255/255/0/1.6 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d6 /dev/rdsk/c14t1d6
disk 19 255/255/0/1.7 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t1d7 /dev/rdsk/c14t1d7
disk 20 255/255/0/2.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d0 /dev/rdsk/c14t2d0
disk 21 255/255/0/2.1 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d1 /dev/rdsk/c14t2d1
disk 22 255/255/0/2.2 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d2 /dev/rdsk/c14t2d2
disk 23 255/255/0/2.3 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d3 /dev/rdsk/c14t2d3
disk 24 255/255/0/2.4 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d4 /dev/rdsk/c14t2d4
disk 25 255/255/0/2.5 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d5 /dev/rdsk/c14t2d5
disk 26 255/255/0/2.6 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d6 /dev/rdsk/c14t2d6
disk 27 255/255/0/2.7 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c14t2d7 /dev/rdsk/c14t2d7

[Ken O'Neil]

Clay,

I do have something extra in /dev/rdsk and /dev/dsk:
# ll
total 0
crw-r----- 1 bin sys 188 0x000000 Jul 28 2004 c0t0d0
crw-r----- 1 bin sys 188 0x0e0000 Oct 14 2004 c14t0d0
crw-r----- 1 bin sys 188 0x0e0100 Oct 14 2004 c14t0d1
crw-r----- 1 bin sys 188 0x0e0200 Oct 14 2004 c14t0d2
crw-r----- 1 bin sys 188 0x0e0300 Oct 14 2004 c14t0d3
crw-r----- 1 bin sys 188 0x0e0400 Oct 14 2004 c14t0d4
crw-r----- 1 bin sys 188 0x0e0500 Oct 14 2004 c14t0d5
crw-r----- 1 bin sys 188 0x0e0600 Oct 14 2004 c14t0d6
crw-r----- 1 bin sys 188 0x0e0700 Oct 14 2004 c14t0d7
crw-r----- 1 bin sys 188 0x0e1000 Oct 14 2004 c14t1d0
crw-r----- 1 bin sys 188 0x0e1100 Oct 14 2004 c14t1d1
crw-r----- 1 bin sys 188 0x0e1200 Oct 14 2004 c14t1d2
crw-r----- 1 bin sys 188 0x0e1300 Oct 14 2004 c14t1d3
crw-r----- 1 bin sys 188 0x0e1400 Oct 21 2004 c14t1d4
crw-r----- 1 bin sys 188 0x0e1500 Oct 21 2004 c14t1d5
crw-r----- 1 bin sys 188 0x0e1600 Oct 21 2004 c14t1d6
crw-r----- 1 bin sys 188 0x0e1700 Oct 21 2004 c14t1d7
crw-r----- 1 bin sys 188 0x0e2000 Oct 21 2004 c14t2d0
crw-r----- 1 bin sys 188 0x0e2100 Oct 21 2004 c14t2d1
crw-r----- 1 bin sys 188 0x0e2200 Sep 20 12:45 c14t2d2
crw-r----- 1 bin sys 188 0x0e2300 Sep 20 12:45 c14t2d3
crw-r----- 1 bin sys 188 0x0e2400 Sep 21 17:41 c14t2d4
crw-r----- 1 bin sys 188 0x0e2500 Sep 21 17:41 c14t2d5
crw-r----- 1 bin sys 188 0x0e2600 Jan 18 01:10 c14t2d6
crw-r----- 1 bin sys 188 0x0e2700 Jan 18 01:10 c14t2d7
crw-r----- 1 bin sys 188 0x020000 Aug 19 2004 c2t0d0
crw-r----- 1 bin sys 188 0x021000 Jan 18 01:20 c2t1d0
crw-r----- 1 bin sys 188 0x030000 Jan 18 01:20 c3t0d0

# ll
total 0
brw-r----- 1 bin sys 31 0x000000 Jul 28 2004 c0t0d0
brw-r----- 1 bin sys 31 0x0e0000 Oct 13 2004 c14t0d0
brw-r----- 1 bin sys 31 0x0e0100 Oct 13 2004 c14t0d1
brw-r----- 1 bin sys 31 0x0e0200 Oct 13 2004 c14t0d2
brw-r----- 1 bin sys 31 0x0e0300 Oct 13 2004 c14t0d3
brw-r----- 1 bin sys 31 0x0e0400 Oct 13 2004 c14t0d4
brw-r----- 1 bin sys 31 0x0e0500 Oct 13 2004 c14t0d5
brw-r----- 1 bin sys 31 0x0e0600 Oct 14 2004 c14t0d6
brw-r----- 1 bin sys 31 0x0e0700 Oct 14 2004 c14t0d7
brw-r----- 1 bin sys 31 0x0e1000 Oct 14 2004 c14t1d0
brw-r----- 1 bin sys 31 0x0e1100 Oct 14 2004 c14t1d1
brw-r----- 1 bin sys 31 0x0e1200 Oct 14 2004 c14t1d2
brw-r----- 1 bin sys 31 0x0e1300 Oct 14 2004 c14t1d3
brw-r----- 1 bin sys 31 0x0e1400 Oct 21 2004 c14t1d4
brw-r----- 1 bin sys 31 0x0e1500 Oct 21 2004 c14t1d5
brw-r----- 1 bin sys 31 0x0e1600 Oct 21 2004 c14t1d6
brw-r----- 1 bin sys 31 0x0e1700 Oct 21 2004 c14t1d7
brw-r----- 1 bin sys 31 0x0e2000 Oct 21 2004 c14t2d0
brw-r----- 1 bin sys 31 0x0e2100 Oct 21 2004 c14t2d1
brw-r----- 1 bin sys 31 0x0e2200 May 11 2005 c14t2d2
brw-r----- 1 bin sys 31 0x0e2300 May 11 2005 c14t2d3
brw-r----- 1 bin sys 31 0x0e2400 Sep 21 17:34 c14t2d4
brw-r----- 1 bin sys 31 0x0e2500 Sep 21 17:34 c14t2d5
brw-r----- 1 bin sys 31 0x0e2600 Jan 18 01:09 c14t2d6
brw-r----- 1 bin sys 31 0x0e2700 Jan 18 01:09 c14t2d7
brw-r----- 1 bin sys 31 0x020000 Jul 28 2004 c2t0d0
brw-r----- 1 bin sys 31 0x021000 Jul 28 2004 c2t1d0
brw-r----- 1 bin sys 31 0x030000 Sep 9 2004 c3t0d0

c2t0d0 doesn't show in ioscan and seems to pre-date the delivery of this system. Would allowing this to remain be causing trouble?

[Ken O'Neil]

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062922068

The method of identifying a VGID in the article above doesn't yield the same output; here's what I see:

# xd -j8200 -N16 /dev/rdsk/c*t*d*
0000000 0 0 0 0 0 0 0 0
0000010
0000010 6040 4a43 416e e1da 6040 4a43 416e e285
0000020
0000020 6040 4a43 416e e1e2 6040 4a43 416e e285
0000030
0000030 6040 4a43 416e e1e9 6040 4a43 416e e2a2
0000040
0000040 6040 4a43 416e e1f3 6040 4a43 416e e2a2
0000050
0000050 6040 4a43 416e e206 6040 4a43 416e e2b0
0000060
0000060 6040 4a43 416e e215 6040 4a43 416e e2b0
0000070
0000070 6040 4a43 416e e21c 6040 4a43 416e e2db
0000080
0000080 6040 4a43 416e e225 6040 4a43 416e e2db
0000090
0000090 6040 4a43 416e e22d 6040 4a43 416e e301
00000a0
00000a0 6040 4a43 416e e23a 6040 4a43 416e e301
00000b0
00000b0 6040 4a43 416e e247 6040 4a43 416e e301
00000c0
00000c0 6040 4a43 416e e253 6040 4a43 416e e301
00000d0
00000d0 6040 4a43 4178 35fa 6040 4a43 4178 365d
00000e0
00000e0 6040 4a43 4178 3605 6040 4a43 4178 365d
00000f0
00000f0 6040 4a43 4178 360a 6040 4a43 4178 365d
0000100
0000100 6040 4a43 4178 360e 6040 4a43 4178 365d
0000110
0000110 6040 4a43 4178 3616 6040 4a43 4178 367b
0000120
0000120 6040 4a43 4178 361e 6040 4a43 4178 367b
0000130
0000130 6040 4a43 4330 66df 6040 4a43 4330 674f
0000140
0000140 6040 4a43 4330 66e8 6040 4a43 4330 674f
0000150
0000150 6040 4a43 4331 fdc5 6040 4a43 4330 674f
0000160
0000160 6040 4a43 4331 fdcd 6040 4a43 4330 674f
0000170
0000170 6040 4a43 43ce 617 6040 4a43 43ce 66c
0000180
0000180 6040 4a43 43ce 61f 6040 4a43 43ce 66c
0000190
xd: cannot open /dev/rdsk/c2t0d0
0000190 6040 4a43 4107 ccf9 6040 4a43 4107 ccfa
00001a0
00001a0 6040 4a43 4140 c035 6040 4a43 4107 ccfa
00001b0

Is there a better syntax for xd that I should use?

[Steven E. Protter]

Shalom,

Looks to me like a disk was replaced at some point and the wrong vg information is on one disk.

Or the disk was part of shared storage in a SG cluster and the message is routine.

I would consider doing a vgexport of the volume group. pvcreate on the disk and vgimport of the volume group.

Just to be safe, if its a boot vg, Ignite make_tape_recovery first.

What does insf -C disk do?

SEP

[James R. Ferguson]

Hi Ken:

OK, things to consider and do.

There don't seem to be any non-unique minor numbers, which is good.

As for listing the LVM PVID/VGID, you can make for more readable output with:

# xd -An -j8200 -N16 -tx /dev/rdsk/cXtYdZ

The first two words are the PVID; the second two the VGID.

A PVID or a VGID is actually composed of the server's serial number (seen in 'uname') and the epoch timestamp.

At one point you show the *same* PVID and VGID for vg00 on serverA as is on serverB. This suggests that one physical disk was replicated from the other and then moved from one server to the other.

As for the error, "Physical Volume "/dev/dsk/c0t0d0" contains no LVM information". This means that no LVM signiture can be found -- no PVID (from a 'pvcreate').

The error 'xd: cannot open /dev/rdsk/c2t0d0' in the absence of any matching device file in 'ioscan' says that the device file no longer represents an attached device.

One possible reason for the error relating to vg00 could be that the server was previously installed with MC/Serviceguard. See if you have:

# /dev/slvmvg

If so, and this is not a Serviceguard cluster, remove the file and regeneate your 'lvmtab' by renaming it and running 'vgscan' as before.

Regards!

...JRF...

[Ken O'Neil]

James,

Thanks for enhanced xd command to read the VGIDs on the physical disks. I cleaned up the old c2t0d0 and re-ran it, here's the output:
# xd -An -j8200 -N16 -tx /dev/rdsk/c*t*d*
0 0 0 0

60404a43 416ee1da 60404a43 416ee285

60404a43 416ee1e2 60404a43 416ee285

60404a43 416ee1e9 60404a43 416ee2a2

60404a43 416ee1f3 60404a43 416ee2a2

60404a43 416ee206 60404a43 416ee2b0

60404a43 416ee215 60404a43 416ee2b0

60404a43 416ee21c 60404a43 416ee2db

60404a43 416ee225 60404a43 416ee2db

60404a43 416ee22d 60404a43 416ee301

60404a43 416ee23a 60404a43 416ee301

60404a43 416ee247 60404a43 416ee301

60404a43 416ee253 60404a43 416ee301

60404a43 417835fa 60404a43 4178365d

60404a43 41783605 60404a43 4178365d

60404a43 4178360a 60404a43 4178365d

60404a43 4178360e 60404a43 4178365d

60404a43 41783616 60404a43 4178367b

60404a43 4178361e 60404a43 4178367b

60404a43 433066df 60404a43 4330674f

60404a43 433066e8 60404a43 4330674f

60404a43 4331fdc5 60404a43 4330674f

60404a43 4331fdcd 60404a43 4330674f

60404a43 43ce0617 60404a43 43ce066c

60404a43 43ce061f 60404a43 43ce066c

60404a43 4107ccf9 60404a43 4107ccfa

60404a43 4140c035 60404a43 4107ccfa

...and here's how the VGIDs in the newly created lvmtab file looks:

/dev/vg01 60404a43/416ee285
/dev/vg02 60404a43/416ee2a2
/dev/vg03 60404a43/416ee2b0
/dev/vg04 60404a43/416ee2db
/dev/vg05 60404a43/416ee301
/dev/vg06 60404a43/4178365d
/dev/vg07 60404a43/4178367b
/dev/vg08 60404a43/4330674f
/dev/vg09 60404a43/43ce066c

They all match, the VGIDs for vg00 can be read from the disk but vg00 isn't being included in lvmtab when a new vgscan is performed. Here's what I get from vgdisplay after creating the new lvmtab:
# vgdisplay -v vg00
vgdisplay: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "vg00".

Remember from my earlier posts though, that if I put an older copy of lvmtab in place and rerun vgdisplay, vg00 is readable with the correct VGIDs, but all of the other VGs show this error:
# vgdisplay -v vg01
vgdisplay: Couldn't query volume group "vg01".
Possible error in the Volume Group minor number; Please check and make sure the group minor number is unique.
vgdisplay: Cannot display volume group "vg01".

So it seems that the best thing to do would be to determine why vg00 is excluded from a new lvmtab created with vgscan.

Even if we were able to determine how to get the proper VGID information for vg01-09 into an previous version of lvmtab, this really wouldn't fix the issue as it would return the next time vgscan is run.

I'll worry about the incorrect VGID for vg00 on SERVERB later, as that system isn't throwing any errors and all LVM functionality appears to be working without issue. I'm also not worried about c0t0d0 as that's just the DVD drive.

I have checked for a previous MCSG config and /dev/slvmvg doesn't exist on either box, and never has as far as I know.

Thanks for your help!

[Ken O'Neil]

Steven,

We can't afford the downtime on the box right now to restore the system volume (vg00). Do you see that as the only option to getting vg00 visible to a new vgscan?

`insf -C disk` returns nothing. The output for `ioscan -fnC disk` is posted above.

Thanks for your reply!

[Ken O'Neil]

Any more ideas on how to get vg00 to show up in a new vgscan?

[James R. Ferguson]

Hi (again) Ken:

I finally read your original attachment.

You have a mismatch between what the running kernel knows about vg00 and what '/etc/lvmtab' has recorded. That is, the current versus active physical volume counts in your 'vgdisplay' for vg00 are not equal. This signals something is amiss.

It further appears that you mirror the logical volumes of vg00 (good) and that all are in sync with no stale extents (good).

It would be useful to see a posting of the current 'lvmtab' along with another verbose 'vgdisplay' of vg00:

# strings /etc/lvmtab
# vgdisplay -v vg00

Regards!

...JRF...

[Ken O'Neil]

Hi James,

Thanks for getting back with me. It all depends on which version of the lvmtab you're interested in.

Here's the output you requested with the original lvmtab in place:

# strings lvmtab
/dev/vg00
`@JCA
/dev/dsk/c2t1d0
/dev/dsk/c3t0d0
/dev/vg01
`@JDAn
/dev/dsk/c14t0d0
/dev/dsk/c14t0d1
/dev/vg02
`@JDAn
/dev/dsk/c14t0d2
/dev/dsk/c14t0d3
/dev/vg03
`@JDAn
/dev/dsk/c14t0d4
/dev/dsk/c14t0d5
/dev/vg04
`@JDAn
/dev/dsk/c14t0d6
/dev/dsk/c14t0d7
/dev/vg05
`@JDAn
/dev/dsk/c14t1d0
/dev/dsk/c14t1d1
/dev/dsk/c14t1d2
/dev/dsk/c14t1d3
/dev/vg06
`@JDAx:
/dev/dsk/c14t1d4
/dev/dsk/c14t1d5
/dev/dsk/c14t1d6
/dev/dsk/c14t1d7
/dev/vg07
`@JDAx:
/dev/dsk/c14t2d0
/dev/dsk/c14t2d1
/dev/vg08
`@JDC0s
/dev/dsk/c14t2d2
/dev/dsk/c14t2d3
/dev/dsk/c14t2d4
/dev/dsk/c14t2d5
/dev/vg09
`@JDC
/dev/dsk/c14t2d6
/dev/dsk/c14t2d7

# vgdisplay -v vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 4
Act PV 2
Max PE per PV 4350
VGDA 4
PE Size (Mbytes) 8
Total PE 8680
Alloc PE 4014
Free PE 4666
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 304
Current LE 38
Allocated PE 76
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 512
Allocated PE 1024
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 200
Current LE 25
Allocated PE 50
Used PV 2

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 128
Allocated PE 256
Used PV 2

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 64
Allocated PE 128
Used PV 2

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 2752
Current LE 344
Allocated PE 688
Used PV 2

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 2056
Current LE 257
Allocated PE 514
Used PV 2

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 4600
Current LE 575
Allocated PE 1150
Used PV 2

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 64
Allocated PE 128
Used PV 2


--- Physical volumes ---
PV Name /dev/dsk/c2t1d0
PV Status available
Total PE 4340
Free PE 2333
Autoswitch On

PV Name /dev/dsk/c3t0d0
PV Status available
Total PE 4340
Free PE 2333
Autoswitch On

Here's the output you requested with the newly created lvmtab in place, (the one created after removing the previous lvmtab and running `vgscan -av`):

# strings lvmtab
/dev/vg01
`@JCAn
/dev/dsk/c14t0d0
/dev/dsk/c14t0d1
/dev/vg02
`@JCAn
/dev/dsk/c14t0d2
/dev/dsk/c14t0d3
/dev/vg03
`@JCAn
/dev/dsk/c14t0d4
/dev/dsk/c14t0d5
/dev/vg04
`@JCAn
/dev/dsk/c14t0d6
/dev/dsk/c14t0d7
/dev/vg05
`@JCAn
/dev/dsk/c14t1d0
/dev/dsk/c14t1d1
/dev/dsk/c14t1d2
/dev/dsk/c14t1d3
/dev/vg06
`@JCAx6]
/dev/dsk/c14t1d4
/dev/dsk/c14t1d5
/dev/dsk/c14t1d6
/dev/dsk/c14t1d7
/dev/vg07
`@JCAx6{
/dev/dsk/c14t2d0
/dev/dsk/c14t2d1
/dev/vg08
`@JCC0gO
/dev/dsk/c14t2d2
/dev/dsk/c14t2d3
/dev/dsk/c14t2d4
/dev/dsk/c14t2d5
/dev/vg09
`@JCC
/dev/dsk/c14t2d6
/dev/dsk/c14t2d7

# vgdisplay -v vg00
vgdisplay: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "vg00".

Now if you look at only this one might say, but all of your VGs seem to show up with the original version of the lvmtab, and why it looks mostly right with `strings`, it's not the right because this version was clearly copied from SERVERB and thus has SERVERB's physical VGIDs, and this results in the "group minor number" errors per the KB article I referenced above.

The only way out of this without taking the box down and rebuilding vg00 seems to be to find a way to recreate without error a new lvmtab so that all of the correct VGIDs are read from the physical disks on SERVERA. When I attempt that now, the new lvmtab file contains good info for all VGs except vg00.

What would be the best way to determine exactly what the variance with the kernel is, and how would I go about updating that variance to resolve the condition that is preventing vg00 from being included in any new vgscan?

[Devender Khatana]

Hi Ken,

================
This now leaves two issues:
1) vg00 doesn't come up in the scan. I get this error:
Physical Volume "/dev/dsk/c0t0d0" contains no LVM information

vgscan: has no correspoding valid raw device file under /dev/rdsk.
Verification of unique LVM disk id on each disk in the volume group
/dev/vg00 failed.

2) It appears that the current vg00 VGID on SERVERB is the same as a previous VGID for vg00 on SERVERA, although no errors have been seen on SERVERB.
=============================
1. The first error is for the DVD-ROM and is normal. The issue is very much clear now and what has happened is that /etc/lvmtb has been moved from serverB to serverA here and not vice versa as you are suspecting. This is the reason why there is no matching disks are found in vg00.

Also note that the total and current PVs in vg00 on giving output with old lvmtab are 4 & 2 respectively.

I am not sure but booting in single user mode and reimporting vg00 shall help is solving the issue.

Also it is the appropriate time to call support for help if available.

HTH,
Devender

[Ken O'Neil]

Thanks for the reply Devender. I've always been pretty certain that lvmtab was copied from B to A and that the DVD wasn't involved, but I might have made things a little confusing by putting too much info out there.

Nice catch though on that variance between current and active volume groups. So how would I go about determining which 4 Current PVs vgdisplay is referring to? Is there any way to correct that variance without bringing the server down into maintenance mode? (as I posted earlier, this will be difficult logistically and would like to do everything possible to avoid this)

About making that call, it is available and it was going to be my next move if I hit a dead-end here, but I'd rather try to work this out in the forums first so that the eventual solution is more readily accessible to a wider audience (i.e. web searchable), at least I hope the forum community thinks that has value.

[Devender Khatana · Accepted solution]

Hi,

Yes you can very well remove the two extra PV's mentioned in vg00 there as you are are sure it only has physically two connected and never had more than them. It can be simply done by

#vgreduce -f /dev/vg00

The VG at this stage should be activated i.e. lvmtab which allows vgdisplay for vg00 should be in place. Once done confirm with vgdisplay the current and active PVs. and if it is correct also retry recreating lvmtab again and see of the error still repeats.

HTH,
Devender

[Mridul Shrivastava]

#vgreduce -f vg00---------> this will remove all the PVs which r not listed in lvmtab so there will be no change in lvmtab after this, however current as well as active PVs should be same.
If u have any lvm extended to those pvs then u have to reduce those lvm first using key value of that pv.

[Ken O'Neil]

Thanks for that Devender (and Mridul). I'll let you know how it works.

[Ken O'Neil]

Issue Resolved! Thanks for the last piece of that Devender!!!

# vgreduce -f vg00
# rm /etc/lvmtab
# vgscan -av
# vgcfgbackup vg00