Suddenly the / filesystem is readonly

[GnanaShekar]

Hi,

We have a SUSE server that was fine until a couple of days. Suddenly, I cannot create a file or folder in the root filesystem. I can't even edit. It says "Read-only file system"

bangpcsl1:~ # mkdir 123
mkdir: cannot create directory `123': Read-only file system

bangpcsl1:~ # more /etc/SuSE-release
SUSE LINUX Enterprise Server 9 (i586)
VERSION = 9
PATCHLEVEL = 3

Find below the contents of /var/log/messages
Sep 6 04:25:46 bangpcsl1 kernel: SCSI error : <0 0 1 0> return code = 0x8000002
Sep 6 04:25:46 bangpcsl1 kernel: Info fld=0x6382974, Current sdb: sense key Hardware Error
Sep 6 04:25:46 bangpcsl1 kernel: Additional sense: Mechanical positioning error
Sep 6 04:25:46 bangpcsl1 kernel: end_request: I/O error, dev sdb, sector 104343923
Sep 6 04:25:46 bangpcsl1 kernel: Buffer I/O error on device sdb2, logical block 12779926
Sep 6 04:25:46 bangpcsl1 kernel: lost page write due to I/O error on sdb2
Sep 6 04:25:56 bangpcsl1 kernel: SCSI error : <0 0 1 0> return code = 0x8000002
Sep 6 04:25:56 bangpcsl1 kernel: Info fld=0x80c1ecb, Current sdb: sense key Hardware Error
Sep 6 04:25:56 bangpcsl1 kernel: Additional sense: Mechanical positioning error
Sep 6 04:25:56 bangpcsl1 kernel: end_request: I/O error, dev sdb, sector 135012043
Sep 6 04:25:56 bangpcsl1 kernel: Buffer I/O error on device sdb2, logical block 16613441
Sep 6 04:25:56 bangpcsl1 kernel: lost page write due to I/O error on sdb2
Sep 6 04:26:58 bangpcsl1 kernel: REISERFS: abort (device sdb2): Write error while pushing transaction to disk in flush_journal_list
Sep 6 04:26:58 bangpcsl1 kernel: REISERFS: Aborting journal for filesystem on sdb2
Sep 6 04:26:58 bangpcsl1 kernel: ReiserFS: sdb2: warning: clm-6006: writing inode 152475 on readonly FS
Sep 6 04:26:58 bangpcsl1 last message repeated 5 times
Sep 6 04:26:58 bangpcsl1 kernel: ReiserFS: sdb2: warning: clm-6006: writing inode 57768 on readonly FS
Sep 6 04:26:58 bangpcsl1 last message repeated 5 times
Sep 6 04:26:58 bangpcsl1 kernel: ReiserFS: sdb2: warning: clm-6006: writing inode 152475 on readonly FS
Sep 6 04:26:58 bangpcsl1 last message repeated 5 times

Please suggest,
Thanks & Regards,

[Matti_Kurkela]

This is what disk hardware failure looks like.

Linux kernel can behave in various ways when it detects a disk error that caused data to be lost: it can either panic (=crash) immediately to avoid further damage, keep running but remount the filesystem as read-only, or do nothing special. The default action is to remount the filesystem as read-only. In a critical environment where "server stops at the first sign of trouble" is better than "incomplete or corrupted data", you may want to set the server to panic immediately in situations like this.

If you have anything irreplaceable on the disk (on any filesystem located on that physical disk), backup it *RIGHT NOW*.

When you reboot the server, you can expect at least a full filesystem check, which may take some time and may require some manual actions in single-user mode. Do not trust the disk after that: if the disk surface is physically damaged, there may be loose crumbs of magnetic material moving around inside the disk housing. Over time, the damage can only get worse.

The worst possibility is that the disk may fail its self-diagnostics at the next reboot and be totally unaccessible after that.

Prepare to replace the disk as soon as possible.

[sathish kannan]

Hi GnanaShekar,

Looks like you have some hardware issue.

ep 6 04:25:46 bangpcsl1 kernel: SCSI error : <0 0 1 0> return code = 0x8000002
Sep 6 04:25:46 bangpcsl1 kernel: Info fld=0x6382974, Current sdb: sense key Hardware Error

Check your hard drive "sdb"

Have you tried unmounting the filesystem and ran fsck?

Regards
Sathish

[Nandu]

Hi

Please try the following command which will remount the filesystem in rw

#mount -o remount /

then during maintenance window try the following command to check the disk sdb2 by

#dd if=/dev/sdb2 of=/dev/null

if this output shows any io error problem with you disk.

[rmueller58]

Is it possible you mounted / as an NFS mount point and made a typo? check your /etc/fstab make sure / is not referenced for any mount points on other devices.

Reboot the system.

[Steven E. Protter]

Shalom,

If a filesystem is no umounted cleanly it can be left dirty and therefore marked in such away that it can not be mounted cleanly rw next boot.

Next boot fsck should fix it though you may need to do this manually in single user mode.

SEP