Why doesn't /dev/vg01 get created? Can't keep new volume groups!

[Trever Furnish]

Every new volume group I create disappears after a reboot, and /dev/vg01 never gets created. I'm using the same methodology I use on x86. What the heck's going on here? Help?

I'm running RHEL4 on IA64 and I'm wondering if there's some step I missing for creating volume groups.

The OS is installed as you'd expect on vg00, which shows up normally as /dev/vg00/*.

However every other volume group I create disappears after a reboot. Even BEFORE the reboot, even though the volume group is working fine, the corresponding /dev entries don't exist and 'dmsetup ls' only lists vg00.

Witness:
[root@inxx24 ~]# vgcreate -s 32M vg02 /dev/sdb /dev/sdp
Volume group "vg02" successfully created
[root@inxx24 ~]# vgdisplay -v /dev/vg02
Using volume group(s) on command line
Finding volume group "vg02"
--- Volume group ---
VG Name vg02
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 273.44 GB
PE Size 32.00 MB
Total PE 8750
Alloc PE / Size 0 / 0
Free PE / Size 8750 / 273.44 GB
VG UUID 2r65jB-tXZ5-2Y1I-8876-11aj-eZRU-dwPc00

--- Physical volumes ---
PV Name /dev/sdb
PV UUID i7Gwyl-AqNF-OYJ2-Nv61-HzyA-8sae-Mfzted
PV Status allocatable
Total PE / Free PE 4375 / 4375

PV Name /dev/sdp
PV UUID UhCFGR-PVD4-CK6Z-RTeE-80dA-g7yh-dHtadP
PV Status allocatable
Total PE / Free PE 4375 / 4375

[root@inxx24 ~]# ls -l /dev/vg02
ls: /dev/vg02: No such file or directory

Hmmm… maybe we just need to create the directory first ourselves?

[root@inxx24 backup]# vgremove vg02
Volume group "vg02" successfully removed
[root@inxx24 backup]# mkdir /dev/vg02
[root@inxx24 backup]# vgcreate -s 32M /dev/vg02 /dev/sdb /dev/sdp
/dev/vg02: already exists in filesystem
New volume group name "vg02" is invalid
[root@inxx24 backup]# ls /dev/vg02
[root@inxx24 backup]# rmdir /dev/vg02
[root@inxx24 backup]# vgcreate -s 32M /dev/vg02 /dev/sdb /dev/sdp
Volume group "vg02" successfully created
[root@inxx24 backup]# ls /dev/vg02
ls: /dev/vg02: No such file or directory

[Matti_Kurkela]

In Linux LVM2, the device nodes are created when the VG is activated, not when it's created.

If there are no LVs defined in the new VG, there is no need to create any device nodes, so the system may omit the creation of the /dev/vg02 directory too.

After a successful vgcreate, use lvcreate to set up a LV inside the VG.

After a reboot, the same caveat applies: if there are no LVs defined in the VG, you won't get any device nodes.

The "dmsetup ls" command lists only *active* LVs: if a VG is not activated, dmsetup does not know anything about the VG nor its LVs. This is by design: activating a VG *means* setting up the appropriate PV(s) -> LV(s) mapping to the device-mapper subsystem.

To query for inactive VGs, use the command "vgs".

If the device nodes get deleted or corrupted for some reason, you can use "vgscan --mknodes" to re-create them without rebooting.

MK

[skt_skt]

also try just "vgscan" after the reboot. That should scan all disks and then try "vgchange -a y vg01"

[Steven E. Protter]

Shalom,

You have run into a bug.

Few ways around it.

You can mount the volume group in the init scripts of the system.

The official fix is to create a new boot ramdisk image after you have created the volume group but not rebooted the system.

backup the old image in /boot

mkinitrd boor/initrd-$(uname -r).img $(uname -r)

SEP

[Trever Furnish]

Thanks for the comments.

I didn't realize the would be no /dev/vg02 until an lvol was allocated, but that does explain that symptom. However my example confused the issue a bit -- in my example I showed vg02, which had no lvols created, but the real problem child was vg01, which was created in the same manner and DID have an lvol. There's no /dev/vg01 after a reboot.

SEP, thanks very much for the example. I'm still not sure it's a fix, but I'll probably find out momentarily.

I should have also noted that vgscan *does* show vg01 items, but it lists two lines saying it can't find a PV (and there were two PVs in the vg), and the UUID in both lines is identical (unless I'm going blind).

[root@inxx24 ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg00" using metadata type lvm2
Found volume group "vg02" using metadata type lvm2
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Volume group "vg01" not found


Also vgdisplay for vg00 doesn't exactly show nothing:
[root@inxx24 ~]# vgdisplay vg01
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Volume group "vg01" not found


The two drives in question are still perfectly fine AFAIK. The only difference I know of that occurred during the creation of the PVs was that for the two disks in vg01 we used fdisk to delete the existing single partition, whereas for the two disks in vg02 I used "dd if=/dev/zero of=/dev/sdp bs=512 count=1".

Also -- sorry there are so many little details! -- I've determined that vg02 DOES work fine after a reboot. Only vg01 has problems. The lack of /dev/vg02 was just a red herring.

I have 28 disks to play with here, so I've got plenty of room to test. I'm going to create a few more test volumes and see what happens following reboots.

We actually only rebooted because we wanted to find out whether lvm2 mirroring with "--corelog" really requires copying a whole lvol after every reboot. I should probably have noted that the lvol on vg01 was created with "-m 1 --corelog" (lvcreate options), which I guess could be related. Probably going to go back to md for raid1 instead. :-(

[skt_skt]

"Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'" above erros makes this so intersting.

do a backup as mentioned below.

$ mv /etc/lvm/archive/* /root/lvm/archive/
$ mv /etc/lvm/backup/* /root/lvm/backup
$ rm /etc/lvm/cache/.cache

4) $ pvscan
5) $ vgscan
6) $ lvscan


If this is the issue we have seen previously, the problem is that clvmd detects change, reloads configuration but doesn't refresh cache properly. So workaround for now is - after every lvm.conf change run pvscan (vgscan).

As usual dont try anythinng in PROD w/o testing in DEV/TEST env. Let us know if your get a chance to test this

[Trever Furnish]

Um... thanks? :-) Did I just turn my box-that-loses-one-filesystem-on-boot into a box-that-won't-boot? Take a look below, let me know whether you think it's save to reboot without restoring /etc/lvm -- not that I'll hold you to it if you're wrong...

Note that the commands still complained about the missing pv, but /etc/lvm no longer contains any trace of vg01 (although it did before I removed files).

[root@inxx24 ~]# tar cvpf /root/lvm.tar /etc/lvm
tar: Removing leading `/' from member names
/etc/lvm/
/etc/lvm/lvm.conf
/etc/lvm/archive/
/etc/lvm/archive/vg02_00001.vg
/etc/lvm/archive/vg02_00002.vg
/etc/lvm/archive/vg02_00000.vg
/etc/lvm/archive/vg02_00003.vg
/etc/lvm/archive/vg01_00001.vg
/etc/lvm/archive/vg01_00000.vg
/etc/lvm/archive/vg00_00000.vg
/etc/lvm/cache/
/etc/lvm/cache/.cache
/etc/lvm/backup/
/etc/lvm/backup/vg02
/etc/lvm/backup/vg00
/etc/lvm/backup/vg01
[root@inxx24 ~]# rm /etc/lvm/archive/*
rm: remove regular file `/etc/lvm/archive/vg00_00000.vg'? y
rm: remove regular file `/etc/lvm/archive/vg01_00000.vg'? y
rm: remove regular file `/etc/lvm/archive/vg01_00001.vg'? y
rm: remove regular file `/etc/lvm/archive/vg02_00000.vg'? y
rm: remove regular file `/etc/lvm/archive/vg02_00001.vg'? y
rm: remove regular file `/etc/lvm/archive/vg02_00002.vg'? y
rm: remove regular file `/etc/lvm/archive/vg02_00003.vg'? y
[root@inxx24 ~]# rm /etc/lvm/backup/*
rm: remove regular file `/etc/lvm/backup/vg00'? y
rm: remove regular file `/etc/lvm/backup/vg01'? y
rm: remove regular file `/etc/lvm/backup/vg02'? y
[root@inxx24 ~]# rm /etc/lvm/cache/.cache
rm: remove regular file `/etc/lvm/cache/.cache'? y
[root@inxx24 ~]# pvscan
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
PV /dev/sdac4 VG vg00 lvm2 [66.84 GB / 43.84 GB free]
PV /dev/sdb VG vg02 lvm2 [136.72 GB / 126.72 GB free]
PV /dev/sdp VG vg02 lvm2 [136.72 GB / 126.72 GB free]
PV /dev/sda VG vg01 lvm2 [136.73 GB / 5.73 GB free]
PV unknown device VG vg01 lvm2 [136.73 GB / 5.73 GB free]
Total: 5 [613.74 GB] / in use: 5 [613.74 GB] / in no VG: 0 [0 ]
[root@inxx24 ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg00" using metadata type lvm2
Found volume group "vg02" using metadata type lvm2
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Volume group "vg01" not found
[root@inxx24 ~]# lvscan
ACTIVE '/dev/vg00/lvol0' [4.00 GB] inherit
ACTIVE '/dev/vg00/lvol5' [1.00 GB] inherit
ACTIVE '/dev/vg00/lvol4' [2.00 GB] inherit
ACTIVE '/dev/vg00/lvol1' [8.00 GB] inherit
ACTIVE '/dev/vg00/lvol2' [4.00 GB] inherit
ACTIVE '/dev/vg00/lvol6' [4.00 GB] inherit
ACTIVE '/dev/vg02/uvol2' [10.00 GB] inherit
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Volume group "vg01" not found
[root@inxx24 ~]# ls -lR /etc/lvm
/etc/lvm:
total 44
drwx------ 2 root root 4096 May 8 20:11 archive
drwx------ 2 root root 4096 May 8 20:11 backup
drwx------ 2 root root 4096 May 8 20:11 cache
-rw-r--r-- 1 root root 15246 Aug 24 2007 lvm.conf

/etc/lvm/archive:
total 8
-rw------- 1 root root 2549 May 8 20:11 vg00_00000.vg
-rw------- 1 root root 1933 May 8 20:11 vg02_00000.vg

/etc/lvm/backup:
total 8
-rw------- 1 root root 2548 May 8 20:11 vg00
-rw------- 1 root root 1932 May 8 20:11 vg02

/etc/lvm/cache:
total 0

[Matti_Kurkela]

vg00 looks fine, maybe also vg02. If your system partitions are on vg00 only, your system should be bootable.

What kind of storage system are you using? SCSI or FibreChannel? If it is accessible through multiple HBAs, can the storage system use all its interfaces simultaneously or does it need something (special signals or just time) to switch from the "active" interface to the "spare" interface?

Can you identify the PVs that are supposed to contain the vg01 in any other way? For example, do you know one of them should be e.g. /dev/sdc or whatever? If you know the devices, what does "pvdisplay -v " report on them?

If it does not have valid Linux LVM PV information, then what does it have? Use "fdisk -l " to see if it has a partition table.

Maybe someone or something has managed to overwrite the LVM identifier on the PVs. These identifiers are created on the PVs when "pvcreate" command is used, and these identifiers are written into LVM configuration data when vgcreate or vgextend is used to create/extend a VG onto new PVs.

Is this storage system dedicated to this server, or shared between multiple servers?

This looks suspiciously like a zoning error in a shared FibreChannel storage system, where two unrelated systems are unintentionally allowed to use the same disk area(s). As each system will assume it's the only one writing to the disk at any given time, this will inevitably lead to disk corruption.

(Server A reads something from the disk and caches it. Then it does some write operations which are cached. Then server B writes to the same disk area; server A does not know its cache is now stale. When server A flushes its write cache, the stale data overwrites whatever server B wrote.)

MK

[Trever Furnish]

The devices in vg01 are /dev/sda and /dev/sdo.

Oddly, pvdisplay produces slightly different results for each of them:

[root@inxx24 ~]# pvdisplay -v /dev/sda
Using physical volume(s) on command line
Wiping cache of LVM-capable devices
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
Couldn't find device with uuid 'dnFmXk-kaKp-q4Xm-cw2d-3d7K-Hpg6-dSMcnX'.
Couldn't find all physical volumes for volume group vg01.
get_pv_from_vg_by_id: vg_read failed to read VG vg01
Can't read : skipping

[root@inxx24 ~]# pvdisplay -v /dev/sdo
Using physical volume(s) on command line
Failed to read physical volume "/dev/sdo"

Fdisk reports no partition table:
[root@inxx24 ~]# fdisk -l /dev/sda

Disk /dev/sda: 146.8 GB, 146815737856 bytes
255 heads, 63 sectors/track, 17849 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sda doesn't contain a valid partition table

I'm having no problems with the other disks, so I think I'm going to see what happens if I zero out the start of these disks and recreate vg01.

[Trever Furnish]

Forgot to include details on the drive system.

There are two dual port HBAs attached via scsi to two full MSA30s with 14 disks. Vg01 was composed of the two disks from slot 1 on each MSA.

There are no other hosts connected.