topic Re: fsck.ext3: no such file or directory while trying to open /dev/mapper in Operating System - Linux

fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Thu, 17 Jun 2010 16:06:51 GMT

Hi,
First of all, I am remotely connected to this RedHatEnterprise Linux 5.0 server thru ILO2 connection in Europe. I'm in US.

when the server tries to reboot (I don't know how it went down), it comes with similar messages such as these:
fsck.ext3: no such file or directory while trying to open /dev/mapper/VolGroup15-Stageing
...........
it comes to "(Repair filesystem)" prompt.
So, I i
ls -l /dev/mapper does NOT show anything that has VolGroup15.
Could you please help!?
Thanks!
Ravi.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Modris Bremze — Thu, 17 Jun 2010 17:09:32 GMT

Could be that the disk devices or LVM structure is damaged. Can you see your disk device files (sd* and/or hd*) inside /dev?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Thu, 17 Jun 2010 19:12:42 GMT

pvs in lvm shows:

/dev/dm-10 VolGroup15
/dev/dm-11 VolGroup10
/dev/dm-12 VolGroup11
/dev/dm-13 VolGroup12
/dev/dm-14 VolGroup13
/dev/dm-15 VolGroup14

it means /dev/dm-10 maps to VolGroup15, correct?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Thu, 17 Jun 2010 19:33:52 GMT

By chance do these Filesystems reside on SAN? if so you can try adding the _netdev option to /etc/fstab on the fourth column so it looks like "defaults,_netdev" , make sure the netfs service is set to autostart.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Thu, 17 Jun 2010 19:36:05 GMT

Yes, they do. I will try adding the _netdev option and let you know.

but, could you also please let me know if when I do pvs in lvm, I get
/dev/dm-10 VolGroup15
/dev/dm-11 VolGroup10
/dev/dm-12 VolGroup11
/dev/dm-13 VolGroup12
/dev/dm-14 VolGroup13
/dev/dm-15 VolGroup14

does this mean /dev/... is the mapping to the /dev/mapper address?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Thu, 17 Jun 2010 20:16:10 GMT

Ravi:
pvs should list the physical volumes being used by LVM like this:

# pvs
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p2 vg01 lvm2 a- 60.33G 30.33G
/dev/emcpowera1 vg00 lvm2 a- 33.71G 0
/dev/emcpowerb1 vg00 lvm2 a- 33.71G 7.43G

It will not show the logical device names, you can use the dmsetup program to verify the mappings..

PS: Don't forget to assign points if you found our answers useful

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Wilfred Chau_1 — Thu, 17 Jun 2010 20:51:45 GMT

do you have /dev/mapper/VolGroup15-Stageing in /etc/fstab?

Does vgdisplay VolGroup15 show anything?
or lvdisplay VolGroup15/Stageing?

Could it be a temp lv?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Thu, 17 Jun 2010 21:01:51 GMT

In any case, ravi, you should be using /dev/Volgroup15/Staging as the device path in lvm, i recall having problems when using the /dev/mapper device names.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Matti_Kurkela — Thu, 17 Jun 2010 21:52:00 GMT

The /dev/dm-* device names are non-persistent device names used by the device-mapper subsystem. Alone, they are rather uninformative: running "dmsetup ls --tree" and comparing the device numbers (major:minor) to disk device nodes in /dev might help in identifying them.

There is a tweak that would make LVM use more informative device names: edit /etc/lvm/lvm.conf and find the "preferred_names" option. By default the list of preferred device names is empty, but there should be an example configuration line with a comment like "Try to avoid using undescriptive /dev/dm-N names". Comment out the empty default, and uncomment the example line. Then run "vgscan" to refresh the LVM metadata cache. Now run "pvs" again to hopefully get a more informative result.

Looks like the VolGroup15 is not active, yet the system is trying to run fsck on it.

Is this system part of a cluster?
The "_netdev" option tells the system "this filesystem requires network to be operational before it can be activated", which is important if VG activation depends on cluster-wide locks.

Normally the system tries to automatically activate all the VGs it sees, before starting to run file system checks. So something may have caused the activation of VolGroup15 to fail.

You might want to run "vgchange -a y VolGroup15". If the automatic activation failed, this will probably fail too, but it should display a helpful error message in that case. For example, this VG might contain more than one PV.

If the manual VG activation is successful, then I guess your system startup script might just have gone ahead faster than the SAN LUN discovery could complete. If that's the case, inserting a short delay to the FC driver loading procedure might help.

If your server really is RHEL 5.0 (i.e. no updates installed since RHEL 5 original release), then I'd strongly suggest upgrading to RHEL 5.4. There has been several kernel, LVM and dm-multipath updates to improve SAN functionality.

MK

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Fri, 18 Jun 2010 13:20:54 GMT

When I do "vgchange -a y VolGroup15", it says, Volume Group "VolGroup15 is exported".

No directory called VolGroup15 under /dev. There is /dev/VolGroup00, though.

Thanks!
Ravi.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Zinky — Fri, 18 Jun 2010 13:47:09 GMT

What does:

multipath -ll

show?

Also -- what does pvscan show?

I think your device mapper config is most likely screwy if your multi-path devices do not come up.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Zinky — Fri, 18 Jun 2010 13:49:08 GMT

Oh PowerPath...
Check your Powerpath environment too... Make sure your server still see them EMC disks...

inquire (inq) tool you have?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Fri, 18 Jun 2010 14:26:54 GMT

With Powerpath installed, you will get exactly these errors if you don't use the _netdev option on fstab, since at the moment vgscan is run during startup PowerPath emcpowerxxx devices are not available, after powerpath is started it runs vgscan once again , that's why you need the _netdev option in fstab.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Fri, 18 Jun 2010 14:34:27 GMT

For further reference, from the PowerPath Admin Guide:

Excerpt from Admin Guide below:

The _netdev option in /etc/fstab
For RHEL 5, 5.1, 5.2, and 5.3, PowerPath devices should be mounted with the _netdev option instead of the defaults option in the /etc/fstab file. This will ensure that fsck is run later in the boot sequence.

http://www.emc.com/microsites/clariion-support/pdf/300-008-345.pdf

"The _netdev option in /etc/fstab. For RHEL 5, 5.1, 5.2, and 5.3, PowerPath devices should be mounted with the _netdev option instead of the defaults option"

Regards!

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Zinky — Fri, 18 Jun 2010 14:43:43 GMT

I don;t think he is using EMC -- I thought those emc devcies above were Ravi's.

/dev/dm-NN as PVs for VGs is indicative he is using non PowerPath diskies.

He is likely having problems with his multipath configuration.

SO he needs to check:
- HBA driver settings if things changed.
- /etc/multpath.conf (what does multipath -ll show?)
- his VGs are likely on a RHCS cluster

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Gerardo Arceri — Fri, 18 Jun 2010 15:17:34 GMT

He stated he's using PowerPath, PowerPath is an EMC Product, i think he will be good if he follows these instructions.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Fri, 18 Jun 2010 17:19:58 GMT

we're not using EMC. it is an EVA.

Since I'm using Remote Desktop => ILO2 connection, I had to take a screenshot for the multipath -ll

Please see attached.

I attached the screenshot for 'mount -a' command which gives the same error about special device not existing at the end of the document.

Thanks!
Ravi.

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Zinky — Fri, 18 Jun 2010 18:02:24 GMT

Bingo!

Please send:

/etc/multipath.conf
complete output of "multipath -ll"
pvscan
/etc/fstab

Are you aware what RH version? (5.0, 5.2. 5.4?) and If you have the HP DM software instaled?

And are you sure this machine is not part of a CLuster where the VGs are likely activatedimportant somewhere?

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Fri, 18 Jun 2010 18:18:39 GMT

complete output of "multipath -ll"
Ravi: previously attached Doc2.docx - that was the best I could do with copy & paste of screenshots from RDP->ILO2 connection.

/etc/multipath.conf
Ravi: attached multipath_conf and PVSCAN Results.docx file.

pvscan results:
Ravi: Attached multipath_conf and PVSCAN Results.docx file.

/etc/fstab results:
Ravi: attached fstab_uname_results.docx file.

Are you aware what RH version? (5.0, 5.2. 5.4?) and If you have the HP DM software instaled?
Ravi: attached fstab_uname_results.docx file contains uname -a results. Release 5 update 4.

And are you sure this machine is not part of a CLuster where the VGs are likely activatedimportant somewhere?
Ravi: I am not 100% sure. It is possible, I guess.

Thanks!

Re: fsck.ext3: no such file or directory while trying to open /dev/mapper

Ravi S. Banda — Fri, 18 Jun 2010 18:42:30 GMT

fstab_uname_results.docx