Stuck in runlevel 3 with / as read only

[Oscar Garcia]

Hi Guys,

I am running Suse SLES 8 in an HP proliant DL385 and after trying to configure nfs and samba I ended up with an unusable machine.
Basically all the filesystems are readonly. I tried to mount / in single user mode, but it comes with:
mount: can't find / in /etc/fstab or /etc/mntab

Any ideas how to make it available? Any thing I tried to do just tells me is read-only...

Thanks in advance.

[xyko_1]

Hi Oscar,

would you please attach boot.log, boot.msg and messages from /var/log ?

Regards,
Xyko

[Oscar Garcia]

Thanks xyko,

I cannot telnet or ssh to that server so getting those files is going to be hard...
I will try to post what seems the mos relevant entries.

Ta

[Oscar Garcia]

Well to make things worst, there is not /var/log only /var/lib and /var/run...

The only filesystems that appear in the fstab are nfs ones.

I think is important to mention that I am using LVM and I have defined only one vg00, where everythings hangs of except /boot and swap.
One of the first errors I notice is something like:
Can't fsck because root is not read-only

and these are the last lines of the boot:
Failed services in runlevel 5: random hotplug sshd nfs xdm
Skipped services in run level 5: isdn nmb sshd cups hwscon postfix smb atd cron nscd inetd

[xyko_1]

Oscar,

show us the result of the command

#mount


Xyko

[Stuart Browne]

Yeah, the major issue is that you've fried your '/etc/fstab'.

There should be (at a minimum) a '/' and 'swap' entry in there for local devices.

You've also mentioned that '/boot' is missing.

Now, assuming you've got all your LVM modules loaded, and the machine does a 'vgscan' (and 'vgchange -a y')on boot-up, you can probably try the following:

mount LABEL=/boot /boot

(as a test). Most modern distro's "LABEL" the filesystems so you can use this method.

If that works, then add entries similar to:

LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2

to your '/etc/fstab', and reboot.

Going back a bit, to explain why you're seeing what you are seeing...

As the system boots (of an initrd etc.), it creates a fake-root of the initrd's details. It then 'pivot_root's to your real '/' device, and continues the booting from there. This then uses your '/etc/fstab' to correctly mount the remainder of your filesystems, usually in '/etc/rc.d/rc.sysinit', including the re-mounting of root (with a command similar to "mount -n -o remount,rw /") into read-write mode. As '/etc/fstab' doesn't exist, the short-hand method of mount (using only the existing mount-point) doesn't work, as there's no map-entry in the fstab.

Unf... yay...

Fix fstab, reboot, all should be well.

[Matti_Kurkela]

Note that the list of mounted disks provided by "mount" command comes from /etc/mnttab, which might have stale data if the root filesystem is in read-only state.

In a situation like this, you should check /proc/mounts. It may not have all the details /etc/mnttab has, but if /proc is properly mounted by the startup scripts, the data will be current (/proc is a pseudo-filesystem; its data is generated from the kernel memory structures on demand).

To make the root filesystem writeable, you should run "mount -o remount,rw /". This is often the first command you'll need to use when booting to the single user mode, so it's good to remember it.

[Oscar Garcia]

Thanks guys for your replies.

The result of mount is:
proc on /proc type proc(rw)

The result of mount LABEL=/boot /boot
mount:no such partition

and this is what I have in /proc/mounts :
rootfs / rootfs rw 0 0
/dev/root / reiserfs ro 0 0
proc /proc proc rw 0 0
devpts /dev/pts devpts rw 0 0
shmfs /dev/shm shm rw 0 0
lindev7:/my_remote_dir /my_local nfs rw, v2, reserved,hard,udp,lock, addr=... 0 0

I have changed to init 1 and when trying to mount -o remount,rw / it comes back with this:
mount: can't find / in /etc/fstab or /etc/mtab

So it seems that the only hope is to use /proc, but I do not know what to put there...

Regards,

[Matti_Kurkela · Accepted solution]

OK, so you'll need to figure out your disk structure first...

Begin by checking out /proc/partitions: it's a list of all disk partitions the system currently has detected. Do you see any "dm-" -style entries? If you do, LVM might already be active. If not, you must activate it manually.

As you have a LVM-based system, a "vgscan -v" might help to understand the current situation. Then activate all available volume groups with "vgchange -a y". If the LVM device nodes are missing, "vgscan --mknodes" should fix that. (None of these commands should do any harm, even if they aren't needed.)

You currently seem to have both /etc/fstab and /etc/mtab in unusable state. First you need to try to reconstruct the /etc/fstab, I think.

Have you already checked whether there is a backup version of /etc/fstab still available in the system? (ls -l /etc/fstab*)

[Oscar Garcia]

Matti,

I cannot see any "dm-" -style entry. However there is a list of what seems to be the partition map. It has 15 columns and start with "major minor #blocks name ..."
the first entry is
58 0 1048576 lvma and 11 columns with 0

the last 4 entries show what it seems to be the partitions for c0d0, c0d0p1 /boot, c0d0p2 swap and c0d0p3 lvm.

I ran vgchange -ay with "vgchange -- volume group "vg00" already active" as result.

I also found a /etc/fstab.orig file. I think there are a couple of filesystems missing, but I can afford to loose them.

Ta

[Matti_Kurkela]

My mistake. SLES8 is apparently kernel 2.4-based, so it uses "lvm*" naming style in /proc/partitions instead of "dm-*". No problem.

Good thing you found the fstab.orig file.

Now the first thing to do would be to create a valid "basic" /etc/fstab. Even just having a correct line for the root filesystem would be an improvement, but I think you have more than that in there.

To make your root filesystem writeable when /etc/fstab is incorrect, you need the device name of the partition or logical volume that contains the root filesystem.

"ls -l /dev/vg00" should display the names of all the logical volumes of vg00, in case you've forgotten and the fstab.orig does not mention that. "vgdisplay -v" might be useful too.

If you only have LABELs in the fstab.orig file, they *should* work instead of device name at this point... but in case they don't, the way to peek at the LABELs of each is to use the appropriate "tunefs" command.
For reiserfs, the command seems to be "reiserfstune "; for ext2 and ext3 filesystems, the command is "tune2fs -l ".

(Disclaimer: I've never had to use the reiserfstune command, and don't have a system with reiserfs available for checking. But as I understand the man page, this would be the correct syntax to view the filesystem parameters, including the LABEL.)

The command to explicitly remount the root filesystem to writeable state, without needing to refer to /etc/fstab and without writing /etc/mtab would be:

mount -n -o remount,rw /

When you get the root filesystem to writeable state, replace the option "-n" (don't update /etc/mtab) with "-f" (do everything but the actual mount) and run the command again to get the root filesystem line of your /etc/mtab up to date.

Now you should be able to fix the /etc/fstab file!

Remember: if you lose only /etc/fstab, it only means the system does not mount your partitions and logical volumes automatically. The data is still there on the disks, you just need to tell the system the correct places to mount the volumes.

When your root filesystem is mounted read-only and nothing else is mounted, you may be restricted to a limited set of commands (if /usr is a separate filesystem and still unmounted). When you get the root filesystem and /usr back to normal state, it should be much easier to identify and mount the rest.

[Oscar Garcia]

Matti you are a wizard!
I've got all the information I needed from the commands/files you recommended and I executed the mount -n -o remount,rw / followed by the -f command.
It worked as a dream! I was able to edit the /etc/fstab and put all the filesystems back.
After re-starting the server I noticed that / was full as somebody created a tmpswap for an oracle installation...
I have also noticed that creating lvms on the fly is not the best practice as it is only after reboot that something goes wrong.

By the way, I did not run the reiserfstune command.

Thanks a lot!