fsck.ext3: no such file or directory while trying to open /dev/mapper

[Ravi S. Banda]

Hi,
First of all, I am remotely connected to this RedHatEnterprise Linux 5.0 server thru ILO2 connection in Europe. I'm in US.

when the server tries to reboot (I don't know how it went down), it comes with similar messages such as these:
fsck.ext3: no such file or directory while trying to open /dev/mapper/VolGroup15-Stageing
...........
it comes to "(Repair filesystem)" prompt.
So, I i
ls -l /dev/mapper does NOT show anything that has VolGroup15.
Could you please help!?
Thanks!
Ravi.

[Modris Bremze]

Could be that the disk devices or LVM structure is damaged. Can you see your disk device files (sd* and/or hd*) inside /dev?

[Ravi S. Banda]

pvs in lvm shows:

/dev/dm-10 VolGroup15
/dev/dm-11 VolGroup10
/dev/dm-12 VolGroup11
/dev/dm-13 VolGroup12
/dev/dm-14 VolGroup13
/dev/dm-15 VolGroup14

it means /dev/dm-10 maps to VolGroup15, correct?

[Gerardo Arceri]

By chance do these Filesystems reside on SAN? if so you can try adding the _netdev option to /etc/fstab on the fourth column so it looks like "defaults,_netdev" , make sure the netfs service is set to autostart.

[Ravi S. Banda]

Yes, they do. I will try adding the _netdev option and let you know.

but, could you also please let me know if when I do pvs in lvm, I get
/dev/dm-10 VolGroup15
/dev/dm-11 VolGroup10
/dev/dm-12 VolGroup11
/dev/dm-13 VolGroup12
/dev/dm-14 VolGroup13
/dev/dm-15 VolGroup14

does this mean /dev/... is the mapping to the /dev/mapper address?

[Gerardo Arceri]

Ravi:
pvs should list the physical volumes being used by LVM like this:

# pvs
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p2 vg01 lvm2 a- 60.33G 30.33G
/dev/emcpowera1 vg00 lvm2 a- 33.71G 0
/dev/emcpowerb1 vg00 lvm2 a- 33.71G 7.43G


It will not show the logical device names, you can use the dmsetup program to verify the mappings..

PS: Don't forget to assign points if you found our answers useful

[Wilfred Chau_1]

do you have /dev/mapper/VolGroup15-Stageing in /etc/fstab?

Does vgdisplay VolGroup15 show anything?
or lvdisplay VolGroup15/Stageing?

Could it be a temp lv?

[Gerardo Arceri]

In any case, ravi, you should be using /dev/Volgroup15/Staging as the device path in lvm, i recall having problems when using the /dev/mapper device names.

[Matti_Kurkela]

The /dev/dm-* device names are non-persistent device names used by the device-mapper subsystem. Alone, they are rather uninformative: running "dmsetup ls --tree" and comparing the device numbers (major:minor) to disk device nodes in /dev might help in identifying them.

There is a tweak that would make LVM use more informative device names: edit /etc/lvm/lvm.conf and find the "preferred_names" option. By default the list of preferred device names is empty, but there should be an example configuration line with a comment like "Try to avoid using undescriptive /dev/dm-N names". Comment out the empty default, and uncomment the example line. Then run "vgscan" to refresh the LVM metadata cache. Now run "pvs" again to hopefully get a more informative result.

Looks like the VolGroup15 is not active, yet the system is trying to run fsck on it.

Is this system part of a cluster?
The "_netdev" option tells the system "this filesystem requires network to be operational before it can be activated", which is important if VG activation depends on cluster-wide locks.

Normally the system tries to automatically activate all the VGs it sees, before starting to run file system checks. So something may have caused the activation of VolGroup15 to fail.

You might want to run "vgchange -a y VolGroup15". If the automatic activation failed, this will probably fail too, but it should display a helpful error message in that case. For example, this VG might contain more than one PV.

If the manual VG activation is successful, then I guess your system startup script might just have gone ahead faster than the SAN LUN discovery could complete. If that's the case, inserting a short delay to the FC driver loading procedure might help.

If your server really is RHEL 5.0 (i.e. no updates installed since RHEL 5 original release), then I'd strongly suggest upgrading to RHEL 5.4. There has been several kernel, LVM and dm-multipath updates to improve SAN functionality.

MK

[Ravi S. Banda]

When I do "vgchange -a y VolGroup15", it says, Volume Group "VolGroup15 is exported".

No directory called VolGroup15 under /dev. There is /dev/VolGroup00, though.

Thanks!
Ravi.

[Zinky]

What does:

multipath -ll

show?

Also -- what does pvscan show?

I think your device mapper config is most likely screwy if your multi-path devices do not come up.

[Zinky]

Oh PowerPath...
Check your Powerpath environment too... Make sure your server still see them EMC disks...

inquire (inq) tool you have?

[Gerardo Arceri]

With Powerpath installed, you will get exactly these errors if you don't use the _netdev option on fstab, since at the moment vgscan is run during startup PowerPath emcpowerxxx devices are not available, after powerpath is started it runs vgscan once again , that's why you need the _netdev option in fstab.

[Gerardo Arceri]

For further reference, from the PowerPath Admin Guide:


Excerpt from Admin Guide below:

The _netdev option in /etc/fstab
For RHEL 5, 5.1, 5.2, and 5.3, PowerPath devices should be mounted with the _netdev option instead of the defaults option in the /etc/fstab file. This will ensure that fsck is run later in the boot sequence.

http://www.emc.com/microsites/clariion-support/pdf/300-008-345.pdf

"The _netdev option in /etc/fstab. For RHEL 5, 5.1, 5.2, and 5.3, PowerPath devices should be mounted with the _netdev option instead of the defaults option"


Regards!

[Zinky]

I don;t think he is using EMC -- I thought those emc devcies above were Ravi's.

/dev/dm-NN as PVs for VGs is indicative he is using non PowerPath diskies.

He is likely having problems with his multipath configuration.

SO he needs to check:
- HBA driver settings if things changed.
- /etc/multpath.conf (what does multipath -ll show?)
- his VGs are likely on a RHCS cluster

[Gerardo Arceri]

He stated he's using PowerPath, PowerPath is an EMC Product, i think he will be good if he follows these instructions.

[Ravi S. Banda]

we're not using EMC. it is an EVA.


Since I'm using Remote Desktop => ILO2 connection, I had to take a screenshot for the multipath -ll

Please see attached.

I attached the screenshot for 'mount -a' command which gives the same error about special device not existing at the end of the document.

Thanks!
Ravi.

[Zinky]

Bingo!

Please send:

/etc/multipath.conf
complete output of "multipath -ll"
pvscan
/etc/fstab


Are you aware what RH version? (5.0, 5.2. 5.4?) and If you have the HP DM software instaled?

And are you sure this machine is not part of a CLuster where the VGs are likely activatedimportant somewhere?

[Ravi S. Banda]

complete output of "multipath -ll"
Ravi: previously attached Doc2.docx - that was the best I could do with copy & paste of screenshots from RDP->ILO2 connection.

/etc/multipath.conf
Ravi: attached multipath_conf and PVSCAN Results.docx file.

pvscan results:
Ravi: Attached multipath_conf and PVSCAN Results.docx file.

/etc/fstab results:
Ravi: attached fstab_uname_results.docx file.

Are you aware what RH version? (5.0, 5.2. 5.4?) and If you have the HP DM software instaled?
Ravi: attached fstab_uname_results.docx file contains uname -a results. Release 5 update 4.

And are you sure this machine is not part of a CLuster where the VGs are likely activatedimportant somewhere?
Ravi: I am not 100% sure. It is possible, I guess.

Thanks!

[Ravi S. Banda]

fstab_uname_results.docx

[Ravi S. Banda]

multipath_conf and PVSCAN Results.docx

[Gerardo Arceri]

You previosuly said that you were using PowerPath right ?

Please post the output of the following commands,preferably in text files:
pvs
vgs
lvs
vgscan
powermt display dev=all

Try to replace your last three "/dev/mapper/Volume15-XXXX lines" on fstab with
/dev/Volume15/XXXX, and add ",_netdev" right besides the default keyword on the fourth column, so for example
/dev/mapper/Volume15-Stageing /u06 ext3 defaults 1 2
becomes
/dev/mapper/Volume15/Stageing /u06 ext3 defaults,_netdev 1 2

Reboot and you should be able to see the logical volume mounted

[Matti_Kurkela]

If the VG has been exported, the "export" status has been written to the VG header on the disks, so rebooting won't change the status of the VG.

An exported VG cannot be activated, so the VG most likely isn't active on any other system. The system startup will attempt to activate all the VGs it sees, but it will *not* auto-import any exported VGs.

(If you know about exporting VGs on HP-UX, don't even try to apply that to Linux; "exporting a VG" is very different on Linux LVM2.)

If all the PVs belonging to this VG are available, the problem can probably be fixed by simply importing the VG, then activating it:

vgimport VolGroup15
vgchange -a y VolGroup15

Exporting a VG is always a manual sysadmin action: the system never does it on its own. It might be important to find out who did export this VG and why. In HP-UX, exporting a VG is a valid substitute for removing it, and exporting is actually easier to do. So someone who has originally learned LVM on HP-UX might have mistakenly applied HP-UX procedures on Linux here.

But the fact that the system is trying to run fsck on the LV(s) of this VG indicates that the removal procedure was not complete: before removing the VG, all the references to its LVs should have been removed from the /etc/fstab file. Whoever did the exporting, may have simply exported a wrong VG by mistake... so there might be remains of another incompletely-removed VG somewhere on your system, which should be properly cleaned up.

MK

[Ravi S. Banda]

Points assigned. Thanks for your help. Since only one file system is affected and we have backups of it, we decided to restore from the backups.

Thanks!
Ravi.