System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

LVM File System failing the booting of a RHEL5 server

SOLVED
Go to solution
Yaboto
Super Advisor

LVM File System failing the booting of a RHEL5 server


A file system (/dev/vgdb/lvdb /oradb) is preventing a RHEL5 from booting. At a certain stage of the startup, it says that mkfs.ext3 cannot recognize or find or FSCK /dev/vgdb/lvdb, and gives me access to a shell command prompt.
In the shell command prompt, whenever I decide to comment or remove the(/dev/vgdb/lvdb /oradb) line in the /etc/fstab with vi command, it says that FSTAB is open in read-only and therefore cannot be amended.

Please Advice

Regards,
14 REPLIES
Ivan Ferreira
Honored Contributor
Solution

Re: LVM File System failing the booting of a RHEL5 server

At the prompt, first run:

mount -o rw,remount /

That will enable your root file system in read write mode to modify the file and boot or troubleshoot your missing device.
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Steven E. Protter
Exalted Contributor

Re: LVM File System failing the booting of a RHEL5 server

Shalom,

You might want to boot your system into rescue mode in order to make repairs to the system.

fsck.ext3 might need to be run manually.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Ralph Grothe
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

Are you sure you put an ext3 file system on /dev/vgdb/lvdb?
Maybe you are using raw devices?

Do you get reasonable output when issuing

# dumpe2fs -h /dev/vgdb/lvdb

or

# tune2fs -l /dev/vgdb/lvdb


Also, maybe you inadvertently entered a wrong fstype or mount options for the lvdb entry in fstab?

Check

# grep /lvdb /etc/fstab
Madness, thy name is system administration
Ralph Grothe
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

Something else that came to my mind.
Though in most initrds you will find a "vgchange -ay" in its init, it might be that your vgdb for some strange reason hasn't been activated.

At the recovery root shell maybe try

# lvm vgscan

# lvm vgchange -a y vgdb

This should usually have open the LV lvdb as well.
You can check the LV attributes. The 5th position should read an "a" like so:

# lvm lvs -o lv_name,lv_attr vgdb
LV Attr
lvdb -wi-a-

You could still try to activate that LV explicitly by

# lvm lvchange -a y /dev/vgdb/lvddb


Remember, when issuing at a normal run level you may omit the leading word "lvm".
Madness, thy name is system administration
Yaboto
Super Advisor

Re: LVM File System failing the booting of a RHEL5 server


I was able to remove the problematique file system and my RHEL5 system can boot successfully.
One thing I would like to do is to make the file system mounting persistent after each reboot.
What is the best way to do that ?
Is it just only by putting the related filesystem in fstab?

Note: I am using DM Multipath

Here is my DF:


/dev/mapper/VolGroup01-LogVol00
130805104 3063008 120990252 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
/dev/mapper/vg01-lvol0
100791728 192272 95479456 1% /oradevap
/dev/mapper/vg02-lvol0
100791728 192272 95479456 1% /oratstap
/dev/mapper/vg03-lvol0
100791728 192272 95479456 1% /oratrnap
[root@localhost ~]#


Please Advice,
Ivan Ferreira
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

>>> Is it just only by putting the related filesystem in fstab?

Yes.

Example:

/dev/vg01/lvol0 /oradevap ext3 defaults 0 0
/dev/vg02/lvol0 /oratstap ext3 defaults 0 0
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Steven E. Protter
Exalted Contributor

Re: LVM File System failing the booting of a RHEL5 server

Shalom,

I definitely think there is leftover entries in /etc/fstab

Make sure everything in there is valid.

mount -a

See what the errors are.

Clean em up.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Yaboto
Super Advisor

Re: LVM File System failing the booting of a RHEL5 server

Please have a look at this:

[root@rcbdev ~]#
[root@rcbdev ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol00
130805104 3500936 120552324 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
[root@rcbdev ~]#
[root@rcbdev ~]# mount -a
mount: special device /dev/vg01/lvol0 does not exist
mount: special device /dev/vg02/lvol0 does not exist
mount: special device /dev/vg03/lvol0 does not exist
[root@rcbdev ~]#
[root@rcbdev ~]# vgchange -a y vg01
1 logical volume(s) in volume group "vg01" now active
[root@rcbdev ~]# vgchange -ay vg02
1 logical volume(s) in volume group "vg02" now active
[root@rcbdev ~]# vgchange -a y vg03
1 logical volume(s) in volume group "vg03" now active
[root@rcbdev ~]#
[root@rcbdev ~]# mount -a
[root@rcbdev ~]#
[root@rcbdev ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup01-LogVol00
130805104 3500944 120552316 3% /
/dev/cciss/c0d0p1 101086 13830 82037 15% /boot
tmpfs 16777216 0 16777216 0% /dev/shm
none 16777216 104 16777112 1% /var/lib/xenstored
/dev/mapper/vg01-lvol0
100791728 2262044 93409684 3% /t24devap
/dev/mapper/vg02-lvol0
100791728 2262024 93409704 3% /t24tstap
/dev/mapper/vg03-lvol0
100791728 2262048 93409680 3% /t24trnap
[root@rcbdev ~]#

There is an improvement now, but look:

Why should I vgchange -a y the VGs anytime I reboot, before they can be mounted ?


Please Advice,
Steven E. Protter
Exalted Contributor

Re: LVM File System failing the booting of a RHEL5 server

Shalom,

Why should I vgchange -a y the VGs anytime I reboot, before they can be mounted ?

No, the system is supposed to do this for you as part of the boot process.

Make sure you log in as root.

root can write to /etc/fstab under most circumstances.

see the command chattr

chattr -i /etc/fstab

Then try again.

Use chattr to display other annoying attributes that might interfere with your changes.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Ivan Ferreira
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

Whare are these file systems located? On a SAN Storage? Maybe the system is not recognizing the devices during boot. Check for any extrange messages during boot phase at the "activating volume groups" stage.
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Yaboto
Super Advisor

Re: LVM File System failing the booting of a RHEL5 server


The problematique mount points are on an EVA Storage through SAN.

I copy & paste some portion of /var/log/messages for your attention:


Aug 2 11:01:05 rcbdev smartd[11334]: Opened configuration file /etc/smartd.conf
Aug 2 11:01:05 rcbdev smartd[11334]: Configuration file /etc/smartd.conf was parsed
, found DEVICESCAN, scanning devices
Aug 2 11:01:05 rcbdev smartd[11334]: Problem creating device name scan list
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sda, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sda, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sda' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdb, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdb, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdb' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdc, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdc, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdc' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdd, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdd, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdd' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sde, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sde, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sde' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdf, opened
Aug 2 11:01:05 rcbdev smartd[11334]: Device: /dev/sdf, IE (SMART) not enabled, skip
device Try 'smartctl -s on /dev/sdf' to turn on SMART features
Aug 2 11:01:05 rcbdev smartd[11334]: Monitoring 0 ATA and 0 SCSI devices
Aug 2 11:01:05 rcbdev smartd[11340]: smartd has fork()ed into background mode. New
PID=11340.

Can this be what is causing the problem ?

Please Advice,
Ivan Ferreira
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

There are some issues with EVA and qlogic adapters. Check the attached doc in the section:

Troubleshooting HSV210 devices on Linux
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Viktor Balogh
Honored Contributor

Re: LVM File System failing the booting of a RHEL5 server

Hi,

these are smartd messages. smartd is a daemon to periodically check the SMART database of a disk to foretell the drive failures. These *might* be related to your problem the following way: I think some drivers are missing at boot time to recognize /dev/sda etc...
****
Unix operates with beer.
Yaboto
Super Advisor

Re: LVM File System failing the booting of a RHEL5 server


Thanks !