LVM Problem vg01 will not auto activate

[Steven E. Protter]

I added a SATA disk to two systems in a cluster.

I used the same methodology on both.

fdsik
partprobe
vgcreate
lvcreate
mkfs.ext3

As a matter of fact I cut and pasted the code from one system to the other.

The second system refuses to activate vg01 at start.

I see nothing in dmesg and even tried using the lvm.conf file from the good system on the bad.

All I have to do is run:
vgchange -a y vg01

and all is good on the second system.

I know I can kludge this with a script and run the vgchange and mount commands and the system will function normally.

I'm wondering however if I can make changes to the lvm configuration to make this happen.

I've done the production process on the troublesome server twice, beginning to end, including fdisk.

I'm looking for suggestions.

Good after activate:
[root@rosh var]# vgdisplay -v vg01
Using volume group(s) on command line
Finding volume group "vg01"
--- Volume group ---
VG Name vg01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 465.76 GB
PE Size 4.00 KB
Total PE 122095952
Alloc PE / Size 122095952 / 465.76 GB
Free PE / Size 0 / 0
VG UUID Astawq-3RcG-9sHy-BNQY-45t8-zVi6-LlIB8N

--- Logical volume ---
LV Name /dev/vg01/share2
VG Name vg01
LV UUID Ltr2cw-msmr-QFz8-JbfY-Xiht-s4yJ-sBTy4n
LV Write Access read/write
LV Status available
# open 0
LV Size 465.76 GB
Current LE 122095952
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:5

--- Physical volumes ---
PV Name /dev/sda1
PV UUID 2N4UGV-2Oql-a531-3WII-kkpj-r1yQ-hh3TGx
PV Status allocatable
Total PE / Free PE 122095952 / 0

After boot
LV Status NOT available

vgchange -a y vg01 fixes it.

Let me know what you think.

RH4 update 5 Centos, fully up to date, though I'm going to yum the system again.

SEP

[Stuart Browne]

Does using the static vgscan do the same?

i.e.

/sbin/lvm.static vgchange -a y vg01

There shouldn't be any reason for it not to, but this is what's used in the boot-up sequence, so it's worth the check.

The other quick thought off the top of my head is that it somehow became exported. An 'lvscan' will show this. That being said, I wouldn't think you could 'vgchange' it at all if it was inactive, so that's probably not it (although it might be worth vgexporting it and vgimporting it again just to poke all the configs).

You could try doing a regeration of the LVM config using 'vgscan -v'. It should re-write the lvmtab and conf stuff.

You say it's a SATA disk. It looks as if 'sda1' is the LVM partition. Are there other partitions on the disk that are being used for other purposes? Is it possible the driver for the SATA controller hasn't loaded for some reason prior to one of the automated vgchange's? (i.e. not in the initrd/modprobe.conf).

.. Well, I'm pretty much rambling now .. ;) Hopefully one of these ideas will give it a kick-in-the-pants.. ;)

[Steven E. Protter]

I'll give it a try today.

Fortuneately its not production yet and can be rebooted easily.

SEP

[Steven E. Protter]

Shalom,

More full data.

/sbin/lvm.static vgchange -a y vg01

works. Apparently this command is not being executed at startup.

Must be some way to force it.

vgscan -v
first run:
[root@rosh ~]# vgscan -v
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Finding all volume groups
Finding volume group "vg00"
Fixing up missing format1 size (111.59 GB) for PV /dev/md1
Found volume group "vg00" using metadata type lvm2
Finding volume group "vg01"
Found volume group "vg01" using metadata type lvm2

second run:
[root@rosh ~]# vgscan -v
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Finding all volume groups
Finding volume group "vg00"
Fixing up missing format1 size (111.59 GB) for PV /dev/md1
Found volume group "vg00" using metadata type lvm2
Finding volume group "vg01"
Found volume group "vg01" using metadata type lvm2

Something is getting fixed but the fix is not being saved.

After restart vg01 is not activated.

The disk has no other partitions on it.

fdisk -l is a little squirrely on both systems:

[root@rosh ~]# fdisk -l

Disk /dev/hda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 26 208813+ fd Linux raid autodetect
/dev/hda2 27 14593 117009427+ fd Linux raid autodetect

Disk /dev/hdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hdb1 * 1 26 208813+ fd Linux raid autodetect
/dev/hdb2 27 14593 117009427+ fd Linux raid autodetect

Disk /dev/md0: 213 MB, 213712896 bytes
2 heads, 4 sectors/track, 52176 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Device Boot Start End Blocks Id System

Disk /dev/md1: 119.8 GB, 119817568256 bytes
2 heads, 4 sectors/track, 29252336 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/dm-0: 536 MB, 536870912 bytes
2 heads, 4 sectors/track, 131072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 114.0 GB, 114030542848 bytes
2 heads, 4 sectors/track, 27839488 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/dm-2: 3145 MB, 3145728000 bytes
2 heads, 4 sectors/track, 768000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/dm-3: 1048 MB, 1048576000 bytes
2 heads, 4 sectors/track, 256000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-3 doesn't contain a valid partition table

Disk /dev/dm-4: 1048 MB, 1048576000 bytes
2 heads, 4 sectors/track, 256000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-4 doesn't contain a valid partition table

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 60801 488384001 8e Linux LVM


Stuart is correct, the disk is /dev/sda1

SEP

[Stuart Browne]

Contents of '/etc/modprobe.conf', and are the SATA controller/lvm modules in the initrd for your kernel?

[Steven E. Protter]

modprobe.conf
# Same as working system in cluster
alias bond0 bonding
# options bond0 miimon=200 mode=active-active
options bond0 miimon=200 mode=0
alias eth0 e1000
alias eth1 e1000
alias eth2 tg3
alias scsi_hostadapter ata_piix
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd

from the initrd config file.

# CONFIG_BLK_DEV_IDE_SATA is not set
#
CONFIG_ATA=m
CONFIG_SATA_AHCI=m
CONFIG_SATA_SVW=m
CONFIG_ATA_PIIX=m
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SX4=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIL24=m
CONFIG_SATA_SIS=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m
CONFIG_SATA_INTEL_COMBINED=y


SEP

[Stuart Browne]

Well, things aren't making sense now.

The presence of the ata_piix listed in the modprobe.conf should mean that the driver is present, if not at the initial phase, and the second run of vgscanning.

If we look at the order of things in 'rc.sysinit', we first loop through the modules.conf at line 179. The first vgchange is at line 511.

So either the driver isn't getting loaded at this stage (unlikely, but easy to check), or the '/dev' isn't being populated correctly:

Line 496 : if [ -c /dev/mapper/control ]

It should be there at this point, but it might not be in the static 'dev' structure.

Line 534 : [ -e /dev/.devfsd ] && mount -f -t devfs devfs /dev

Remount with devfs if it's configured to do so. This should do the dynamic mapper stuff so it should exist later.. maybe..

The later attempt to vgchange is at line 570, surrounded by:

Line 552 : if [ -f /etc/mdadm.conf ]

and another mapper check at line 555.

I guess one way to see what's occuring is to throw a '-x' at the top of rc.sysinit, but it'll get really messy.

Try booting into 'emergency' (not single) and manually walking the following steps:

ls -ld /dev/mapper/control
/sbin/modprobe ata_piix
/sbin/lvm.static vgchange -a y

See what errors these steps throw.

[Steven E. Protter]

Shalom,

As I'm 6,000 miles from the systems in question the emergency mode boot will need to take some time.

I have a network console hung off the system but I have to go through a lengthy configuration process before I can do this work remotely.

SEP

[Steven E. Protter]

[root@rosh ~]# fdisk -l

Disk /dev/hda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 26 208813+ fd Linux raid autodetect
/dev/hda2 27 14593 117009427+ fd Linux raid autodetect

Disk /dev/hdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hdb1 * 1 26 208813+ fd Linux raid autodetect
/dev/hdb2 27 14593 117009427+ fd Linux raid autodetect

Disk /dev/md0: 213 MB, 213712896 bytes
2 heads, 4 sectors/track, 52176 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Device Boot Start End Blocks Id System

Disk /dev/md1: 119.8 GB, 119817568256 bytes
2 heads, 4 sectors/track, 29252336 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/dm-0: 536 MB, 536870912 bytes
2 heads, 4 sectors/track, 131072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 114.0 GB, 114030542848 bytes
2 heads, 4 sectors/track, 27839488 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/dm-2: 3145 MB, 3145728000 bytes
2 heads, 4 sectors/track, 768000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/dm-3: 1048 MB, 1048576000 bytes
2 heads, 4 sectors/track, 256000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/dm-3 doesn't contain a valid partition table

Disk /dev/dm-4: 1048 MB, 1048576000 bytes
2 heads, 4 sectors/track, 256000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Device Boot Start End Blocks Id System

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes


The /dev/dm-4 stuff is happening on both servers and is probably coming from the adapter card.

Just thought it might be helpful.

SEP

[Stuart Browne]

Sorry, thought it was local to you.

Quick tests to do in the mean time.

Throw markers into rc.sysinit around the vgchange lines to make sure it's actually getting executed, and do the 'ls -ld' there too. See what's there. Also verify that the 'ata_piix' is loaded with 'lsmod' at this point. It should be from all that's been shown.

See if you can trap any errors from the vgchange (> and 2>, with an 'echo $?' for good measure). Throw a '-v' (verbose) on it too. Maybe it is throwing an error, we're just not seeing it (pretty much none of the rc.sysinit output is captured).

As it's using devfs, it's sorta hard to check the contents of '/dev/' without doing funny buggers, but at the moment, it's pointing towards being that.

That being said, the second pass (which looks for the mdadm.conf) should have caught it.

Once the system boots, the vgchange works without issue, which points towards the driver being correctly loaded as well, and as far as I'm aware, there isn't a 2nd pass to load SCSI drivers later on in the boot process.

Rambling musing again.. Gotta stop that ;)

[Ivan Ferreira]

There are some things that are not pretty whell documented.

Fist, ensure that you used lvmconf to enable cluster locking.

lvmconf --enable-cluster

Then, ensure that the volume group has the "clusted" attribute:

vgchange -cy

Let me know if it helps.

[Steven E. Protter]

Shalom,

Thank's Ivan.

There is disk on both servers but none of the storage is actually shared. We use rsync to keep the two nodes up to date.

I'm going to try to debug the startup.

If I find nothing, I'm looking for ideas on where the best place to put the kludge is.

Yes, a shell script, activated at startup that does one little thing:
vgchange -a y vg01

SEP

[Ralph Grothe]

If you haven't compiled a custom kernel whith built-in support for LVM and MD meta devices
then most likely the driver is loaded as a module and VGs get activated early at system boot from within your initial RAM disk.
You can easily check by inspecting the init script of your initrd.

# zcat /boot/initrd-$(uname -r).img|cpio -ivt init
-rwxr-xr-x 1 root root 1369 Jan 18 2007 init
4544 blocks

Copy this file in somewhere and open it in a pager or editor.
But usually it contains an activation of all VGs.

# grep insmod init
insmod /lib/scsi_mod.ko
insmod /lib/sd_mod.ko
insmod /lib/mptbase.ko
insmod /lib/mptscsih.ko
insmod /lib/dm-mod.ko
insmod /lib/jbd.ko
insmod /lib/ext3.ko
insmod /lib/dm-mirror.ko
insmod /lib/dm-zero.ko
insmod /lib/dm-snapshot.ko

# grep lvm init
lvm vgscan
lvm vgchange -ay
lvm vgmknodes

[Steven E. Protter]

Thanks Ralph,

I've been backed up with functionity issues the past few days.

I will get back to this and assign points.

I'm guessing we'll find that the initrd is fine, but maybe not. I know how to create a new one if necessary.

SEP

[Steven E. Protter]

Shalom,

Performed

zcat /boot/initrd-2.6.9-55.EL.img | cpio -ivt > init

[root@rosh ~]# grep insmod init
-rwxr-xr-x 1 root root 388320 Aug 6 08:46 bin/insmod
[root@rosh ~]# grep lvm init
-r-xr-xr-x 1 root root 1261836 Aug 6 08:46 bin/lvm
drwxr-xr-x 2 root root 0 Aug 6 08:46 etc/lvm
-rw-r--r-- 1 root root 14654 Aug 6 08:46 etc/lvm/lvm.conf

The results are the same on both system.

Nothing in the boot.log

Performing a boot with a couple of set -x/set +x bracked around cricial lvm code.

SEP

[Steven E. Protter]

Shalom,

I hacked it.

/etc/rc.sysinit


at the bottom.

/sbin/vgchange -a y vg01
/bin/mount /dev/vg01/share2 /storage

I wonder if there is not a better way or place to do this.

If I set up an NFS share on /storage, I'm wondering if it will work. My assumption based on the RHCE and such is that sysinit runs runs before the nfs service so my hack is reasonable.

SEP

[Steven E. Protter]

Shalom,

When this happened on a private system, being ethical, I never considered checking with Red Hat Support via the service agreements with our office. That would be unethical.

However a week ago, we had the exact same problem at work. I have filed a case with Red Hat and they believe the problem is a bug. The work systems were pure RH, not CentOS, but a RH bug is a CentOS bug and vice versa.

I shall report further results and any fixes they provide when it happens.

SEP

[Steven E. Protter]

Solution on a SAN environment at work.



in /etc/modprobe.conf
Change:
alias scsi_hostadapter2 qla2322

to

alias scsi_hostadapter2 qla2400

mkinitrd -f initrd-$(uname -r).img $(uname -r)

reboot

I need to configure remote serial console on the system before I even think about trying this on the two Centos systems that inspired this thread.

Wondering If I'd change modprobe.conf or merely mkinitrd and see what happens.

Shmuel