LVM File System failing the booting of a RHEL5 server

[Yaboto]

A file system (/dev/vgdb/lvdb /oradb) is preventing a RHEL5 from booting. At a certain stage of the startup, it says that mkfs.ext3 cannot recognize or find or FSCK /dev/vgdb/lvdb, and gives me access to a shell command prompt.
In the shell command prompt, whenever I decide to comment or remove the(/dev/vgdb/lvdb /oradb) line in the /etc/fstab with vi command, it says that FSTAB is open in read-only and therefore cannot be amended.

Please Advice

Regards,

[Ivan Ferreira · Accepted solution]

At the prompt, first run:

mount -o rw,remount /

That will enable your root file system in read write mode to modify the file and boot or troubleshoot your missing device.

[Steven E. Protter]

Shalom,

You might want to boot your system into rescue mode in order to make repairs to the system.

fsck.ext3 might need to be run manually.

SEP

[Ralph Grothe]

Are you sure you put an ext3 file system on /dev/vgdb/lvdb?
Maybe you are using raw devices?

Do you get reasonable output when issuing

# dumpe2fs -h /dev/vgdb/lvdb

or

# tune2fs -l /dev/vgdb/lvdb


Also, maybe you inadvertently entered a wrong fstype or mount options for the lvdb entry in fstab?

Check

# grep /lvdb /etc/fstab

[Ralph Grothe]

Something else that came to my mind.
Though in most initrds you will find a "vgchange -ay" in its init, it might be that your vgdb for some strange reason hasn't been activated.

At the recovery root shell maybe try

# lvm vgscan

# lvm vgchange -a y vgdb

This should usually have open the LV lvdb as well.
You can check the LV attributes. The 5th position should read an "a" like so:

# lvm lvs -o lv_name,lv_attr vgdb
LV Attr
lvdb -wi-a-

You could still try to activate that LV explicitly by

# lvm lvchange -a y /dev/vgdb/lvddb


Remember, when issuing at a normal run level you may omit the leading word "lvm".

[Yaboto]

I was able to remove the problematique file system and my RHEL5 system can boot successfully.
One thing I would like to do is to make the file system mounting persistent after each reboot.
What is the best way to do that ?
Is it just only by putting the related filesystem in fstab?

Note: I am using DM Multipath

Here is my DF:


/dev/mapper/VolGroup01-LogVol00
130805104 3063008 120990252 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
/dev/mapper/vg01-lvol0
100791728 192272 95479456 1% /oradevap
/dev/mapper/vg02-lvol0
100791728 192272 95479456 1% /oratstap
/dev/mapper/vg03-lvol0
100791728 192272 95479456 1% /oratrnap
[root@localhost ~]#


Please Advice,

[Ivan Ferreira]

>>> Is it just only by putting the related filesystem in fstab?

Yes.

Example:

/dev/vg01/lvol0 /oradevap ext3 defaults 0 0
/dev/vg02/lvol0 /oratstap ext3 defaults 0 0

[Steven E. Protter]

Shalom,

I definitely think there is leftover entries in /etc/fstab

Make sure everything in there is valid.

mount -a

See what the errors are.

Clean em up.

SEP

[Yaboto]

Please have a look at this:

[root@rcbdev ~]#
[root@rcbdev ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol00
130805104 3500936 120552324 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
[root@rcbdev ~]#
[root@rcbdev ~]# mount -a
mount: special device /dev/vg01/lvol0 does not exist
mount: special device /dev/vg02/lvol0 does not exist
mount: special device /dev/vg03/lvol0 does not exist
[root@rcbdev ~]#
[root@rcbdev ~]# vgchange -a y vg01
1 logical volume(s) in volume group "vg01" now active
[root@rcbdev ~]# vgchange -ay vg02
1 logical volume(s) in volume group "vg02" now active
[root@rcbdev ~]# vgchange -a y vg03
1 logical volume(s) in volume group "vg03" now active
[root@rcbdev ~]#
[root@rcbdev ~]# mount -a
[root@rcbdev ~]#
[root@rcbdev ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol00
130805104 3500944 120552316 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
/dev/mapper/vg01-lvol0
100791728 2262044 93409684 3% /t24devap
/dev/mapper/vg02-lvol0
100791728 2262024 93409704 3% /t24tstap
/dev/mapper/vg03-lvol0
100791728 2262048 93409680 3% /t24trnap
[root@rcbdev ~]#

There is an improvement now, but look:

Why should I vgchange -a y the VGs anytime I reboot, before they can be mounted ?


Please Advice,

[Steven E. Protter]

Shalom,

Why should I vgchange -a y the VGs anytime I reboot, before they can be mounted ?

No, the system is supposed to do this for you as part of the boot process.

Make sure you log in as root.

root can write to /etc/fstab under most circumstances.

see the command chattr

chattr -i /etc/fstab

Then try again.

Use chattr to display other annoying attributes that might interfere with your changes.

SEP

[Ivan Ferreira]

Whare are these file systems located? On a SAN Storage? Maybe the system is not recognizing the devices during boot. Check for any extrange messages during boot phase at the "activating volume groups" stage.

[Yaboto]

The problematique mount points are on an EVA Storage through SAN.

I copy & paste some portion of /var/log/messages for your attention:


Aug 2 11:01:05 rcbdev smartd[11334]: Opened configuration file /etc/smartd.conf
Aug 2 11:01:05 rcbdev smartd[11334]: Configuration file /etc/smartd.conf was parsed
, found DEVICESCAN, scanning devices
Aug 2 11:01:05 rcbdev smartd[11334]: Problem creating device name scan list
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sda, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sda, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sda' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdb, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdb, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdb' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdc, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdc, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdc' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdd, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdd, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdd' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sde, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sde, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sde' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdf, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdf, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdf' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Monitoring 0 ATA and 0 SCSI devices
Aug 2 11:01:05 rcbdev smartd[11340]: smartd has fork()ed into background mode. New
PID=11340.

Can this be what is causing the problem ?

Please Advice,

[Ivan Ferreira]

There are some issues with EVA and qlogic adapters. Check the attached doc in the section:

Troubleshooting HSV210 devices on Linux

[Viktor Balogh]

Hi,

these are smartd messages. smartd is a daemon to periodically check the SMART database of a disk to foretell the drive failures. These *might* be related to your problem the following way: I think some drivers are missing at boot time to recognize /dev/sda etc...

[Yaboto]

Thanks !