topic Re: Root file system is mounted read-only??? in Operating System - Linux

Root file system is mounted read-only???

Qcheck — Wed, 16 Jun 2010 16:51:50 GMT

Recently one of our servers rebooted and the root file system is mounted read only. We can't touch any files in that file system. How can I mount rw?

When I tried # mount -n -o remount /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

O/S is 5.1. Any help?

Re: Root file system is mounted read-only???

Tim Nelson — Wed, 16 Jun 2010 16:59:39 GMT

how about

mount -o remount,rw /

Re: Root file system is mounted read-only???

Qcheck — Wed, 16 Jun 2010 17:02:09 GMT

No lick Tim:

# mount -o remount,rw /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

Do u think hard disk is bad?

Re: Root file system is mounted read-only???

Qcheck — Wed, 16 Jun 2010 17:02:36 GMT

Sorry I meant no Luck.

Re: Root file system is mounted read-only???

Gerardo Arceri — Wed, 16 Jun 2010 18:16:55 GMT

try before remounting:
lvchange -pw /dev/VolGroup00/LogVol00

Are you sure that /dev/VolGroup00/LogVol00 is your root filesystem ?

Re: Root file system is mounted read-only???

Modris Bremze — Wed, 16 Jun 2010 19:03:53 GMT

LVM probably has a good reason for disabling writes to /. If you say the server rebooted - was it unexpectedly? If so, did you look at dmesg or /var/log/messages output for any LVM related errors?

Running fsck on a mounted filesystem is not very safe, it being a read-only filesystem could also make it not very productive. You could try booting OEL installation CD1 in recovery mode and running fsck on your LV then.

Re: Root file system is mounted read-only???

Qcheck — Wed, 16 Jun 2010 19:08:36 GMT

Thank you Gerardo and Medris. Yes /dev/VolGroup00/LogVol00 is my / filesystem and when I tried the command, "lvchange -pw /dev/VolGroup00/LogVol00", it errors out as w as an invalid argument even though I saw in the man page.

And I don't see any errors in the log files abt lvm, however I will check thoroughly. I couldn't able to right to / file system, that is how I found the issue. Yes, I know I should boot the server in rescue mode but I am working remotely. Notified this to the management in the afternoon. Probably tomorrow they will decide how to follow up.

Thanks again for all your help.

Re: Root file system is mounted read-only???

Qcheck — Thu, 17 Jun 2010 12:46:01 GMT

This is what I noticed in the log files:

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
Jun 14 18:11:10 rac3 kernel: sda: Write Protect is off
Jun 14 18:11:10 rac3 kernel: SCSI device sda: drive cache: write back w/ FUA
Jun 14 19:15:46 rac3 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Jun 14 19:15:46 rac3 kernel: sda: Current: sense key: Hardware Error
Jun 14 19:15:46 rac3 kernel: Add. Sense: Internal target failure
Jun 14 19:15:46 rac3 kernel:
Jun 14 19:15:46 rac3 kernel: Info fld=0x0
Jun 14 19:15:46 rac3 kernel: end_request: I/O error, dev sda, sector 356269
Jun 14 19:15:46 rac3 kernel: Aborting journal on device dm-0.
Jun 14 19:15:47 rac3 kernel: ext3_abort called.
Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only

Re: Root file system is mounted read-only???

P Muralidhar Kini — Thu, 17 Jun 2010 13:00:20 GMT

Hi Pratibha,

>> This is what I noticed in the log files:
This looks like a hardware error on that device.

Regards,
Murali

Re: Root file system is mounted read-only???

AnthonySN — Sun, 20 Jun 2010 18:19:33 GMT

was there any san lun presented to the server.
we faced a similar issue when the san lun was unpresented and we rebooted the server but there was an entry in the fstab of the san mounted partition hence it would boot and mount the filesystem in readonly.

Re: Root file system is mounted read-only???

AnthonySN — Sun, 20 Jun 2010 18:25:25 GMT

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
>>440077 MB approx 430 GB means its not local disk.

Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only
>>device dm-0 means a san disk

so check your san disk which may be unpresented

Re: Root file system is mounted read-only???

Thomas Callahan — Mon, 21 Jun 2010 12:16:18 GMT

DM-0 doesn't strictly mean it's a SAN disk. Device-Mapper can manage anything - including scsi, local attached, in the nose, or SAN. In fact, all LVM devices are managed under DM, and those are just logical pointers, per se.

If it IS a SAN disk, and has been unpresented, there is no reason the server should still be able to see the disk. There should be no path to the disk if it has been "properly" unpresented.

This looks more like a volume failure, more specifically an EXT3 failure. I would reboot in rescue mode, and run an fsck on the disk to attempt to clear the errors. If that fails, you may be looking at a restore from backup.

Re: Root file system is mounted read-only???

Qcheck — Mon, 21 Jun 2010 12:24:55 GMT

Thomas, Thank you and everyone who is responding to this.

In order to reboot in rescue mode, I need the boot disk right? I am afraid the client has that. Physically I can't access the server as it is 3 hrs away and we r supporting remotely.

So now, over the weekend, the server went down, tried to reboot I guess and this is where it stopped:

"Reboot and Select proper Boot device
or Insert Boot Media in selected Boot device and press a key"

So If I ask them to power off and on the button, do you think it will come up? Can I try that first as they don't have the disk1 to mount? Please let me know asap?

Thanks again.

Re: Root file system is mounted read-only???

AnthonySN — Mon, 21 Jun 2010 12:58:10 GMT

check the bios for boot order change, any usb disk or floppy attached to the server.

Re: Root file system is mounted read-only???

Qcheck — Mon, 21 Jun 2010 13:23:27 GMT

SASJ, Thank you for the response.

We asked the client to power cycle the server and everything came up clean. Now, curious of what happened? Now, no read-only file system. I can write to the / file system. Happies....but wht happened? Nothing suspicious in the log file.

Re: Root file system is mounted read-only???

AnthonySN — Mon, 21 Jun 2010 18:09:02 GMT

that is indeed gr8 news...
All is Well that ends Well.
cheers...

Re: Root file system is mounted read-only???

Thomas Callahan — Tue, 22 Jun 2010 11:28:22 GMT

Sounds like whatever is presenting your storage hiccuped or the server hiccuped.

The screen you said it was stuck at was a BIOS screen where it was looking for a Boot Device but not finding one.

I would ask the client where the storage is coming from, and if there were any problems with that storage at the time this occurred.

Re: Root file system is mounted read-only???

Qcheck — Tue, 22 Jun 2010 16:28:45 GMT

After power cycling the server, problem has gone away......