Root file system is mounted read-only???

[Qcheck]

Recently one of our servers rebooted and the root file system is mounted read only. We can't touch any files in that file system. How can I mount rw?

When I tried # mount -n -o remount /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

O/S is 5.1. Any help?

[Tim Nelson]

how about

mount -o remount,rw /

[Qcheck]

No lick Tim:

# mount -o remount,rw /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only.

Do u think hard disk is bad?

[Qcheck]

Sorry I meant no Luck.

[Gerardo Arceri · Accepted solution]

try before remounting:
lvchange -pw /dev/VolGroup00/LogVol00

Are you sure that /dev/VolGroup00/LogVol00 is your root filesystem ?

[Modris Bremze]

LVM probably has a good reason for disabling writes to /. If you say the server rebooted - was it unexpectedly? If so, did you look at dmesg or /var/log/messages output for any LVM related errors?

Running fsck on a mounted filesystem is not very safe, it being a read-only filesystem could also make it not very productive. You could try booting OEL installation CD1 in recovery mode and running fsck on your LV then.

[Qcheck]

Thank you Gerardo and Medris. Yes /dev/VolGroup00/LogVol00 is my / filesystem and when I tried the command, "lvchange -pw /dev/VolGroup00/LogVol00", it errors out as w as an invalid argument even though I saw in the man page.

And I don't see any errors in the log files abt lvm, however I will check thoroughly. I couldn't able to right to / file system, that is how I found the issue. Yes, I know I should boot the server in rescue mode but I am working remotely. Notified this to the management in the afternoon. Probably tomorrow they will decide how to follow up.

Thanks again for all your help.

[Qcheck]

This is what I noticed in the log files:

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
Jun 14 18:11:10 rac3 kernel: sda: Write Protect is off
Jun 14 18:11:10 rac3 kernel: SCSI device sda: drive cache: write back w/ FUA
Jun 14 19:15:46 rac3 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Jun 14 19:15:46 rac3 kernel: sda: Current: sense key: Hardware Error
Jun 14 19:15:46 rac3 kernel: Add. Sense: Internal target failure
Jun 14 19:15:46 rac3 kernel:
Jun 14 19:15:46 rac3 kernel: Info fld=0x0
Jun 14 19:15:46 rac3 kernel: end_request: I/O error, dev sda, sector 356269
Jun 14 19:15:46 rac3 kernel: Aborting journal on device dm-0.
Jun 14 19:15:47 rac3 kernel: ext3_abort called.
Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only

[P Muralidhar Kini]

Hi Pratibha,

>> This is what I noticed in the log files:
This looks like a hardware error on that device.

Regards,
Murali

[AnthonySN]

was there any san lun presented to the server.
we faced a similar issue when the san lun was unpresented and we rebooted the server but there was an entry in the fstab of the san mounted partition hence it would boot and mount the filesystem in readonly.

[AnthonySN]

Jun 14 18:11:10 rac3 kernel: SCSI device sda: 859525120 512-byte hdwr sectors (440077 MB)
>>440077 MB approx 430 GB means its not local disk.

Jun 14 19:15:47 rac3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Jun 14 19:15:47 rac3 kernel: Remounting filesystem read-only
>>device dm-0 means a san disk

so check your san disk which may be unpresented

[Thomas Callahan]

DM-0 doesn't strictly mean it's a SAN disk. Device-Mapper can manage anything - including scsi, local attached, in the nose, or SAN. In fact, all LVM devices are managed under DM, and those are just logical pointers, per se.

If it IS a SAN disk, and has been unpresented, there is no reason the server should still be able to see the disk. There should be no path to the disk if it has been "properly" unpresented.

This looks more like a volume failure, more specifically an EXT3 failure. I would reboot in rescue mode, and run an fsck on the disk to attempt to clear the errors. If that fails, you may be looking at a restore from backup.

[Qcheck]

Thomas, Thank you and everyone who is responding to this.

In order to reboot in rescue mode, I need the boot disk right? I am afraid the client has that. Physically I can't access the server as it is 3 hrs away and we r supporting remotely.

So now, over the weekend, the server went down, tried to reboot I guess and this is where it stopped:

"Reboot and Select proper Boot device
or Insert Boot Media in selected Boot device and press a key"

So If I ask them to power off and on the button, do you think it will come up? Can I try that first as they don't have the disk1 to mount? Please let me know asap?

Thanks again.

[AnthonySN]

check the bios for boot order change, any usb disk or floppy attached to the server.

[Qcheck]

SASJ, Thank you for the response.

We asked the client to power cycle the server and everything came up clean. Now, curious of what happened? Now, no read-only file system. I can write to the / file system. Happies....but wht happened? Nothing suspicious in the log file.

[AnthonySN]

that is indeed gr8 news...
All is Well that ends Well.
cheers...

[Thomas Callahan]

Sounds like whatever is presenting your storage hiccuped or the server hiccuped.

The screen you said it was stuck at was a BIOS screen where it was looking for a Boot Device but not finding one.

I would ask the client where the storage is coming from, and if there were any problems with that storage at the time this occurred.

[Qcheck]

After power cycling the server, problem has gone away......