One of file systems suddenly became read-only on Suse Linux (suse 10 sp3)

[senthil_kumar_2]

Hi All,

One of file systems suddenly became read-only on Suse Linux (suse 10 sp3)..

Even root user is not able to access...

Actually It is a SAN disk coming from Hitachi box...

But It is showing as RW
(Read/Write) while I ran mount command...

# mount
/dev/sda2 on /win_appl type reiserfs (rw,acl,user_xattr

My Questions:

1)How to trouble shoot this that Where is the issue OS side / SAN side?

2)Will It be resolved after reboot of the server...

3)Is it possible to resolve the issue with out reboot..

[senthil_kumar_2]

Hi All,

When I checked the filesystem using the command dumpe2fs...I am getting following error...

# dumpe2fs /dev/sda2
dumpe2fs 1.38 (30-Jun-2005)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sda2
Couldn't find valid filesystem superblock.


How to resolve this...

But still I am able to read the file system...

[Wilfred Chau_1 · Accepted solution]

you have acl turned on on the file system.

what is the output of
getfacl /dev/sda2
looks like? Do you have the "w" for root user?

[macosta]

The dumpe2fs command won't tell you anything about that filesystem because the FS type is ReiserFS. dumpe2fs is for ext{2..4} filesystems.

My guess would be that it is read-only because of a configuration error or a metadata operation encountered an error. Remounting might help, but you need to look for the cause.

Check the system logs around the time it happened to see if you received any I/O errors. Other than that, check the permissions on the filesystem and the filesystem in which it's mounted.

[Matti_Kurkela]

The "mount" command just displays what it wrote into /etc/mtab when each filesystem was mounted. For current information in situations like this, you should look into /proc/mounts.

Please run these commands:
grep "ro," /proc/mounts
dmesg | less

You will probably find out that the filesystem is now in read-only mode, and the "dmesg" output might be showing some disk I/O errors.

If there are so many errors in the dmesg output that the original system boot messages are no longer visible, look into logfiles in /var/log to see when the errors began happening and what were the _first_ error messages: these usually give the most clues about what went wrong.

Answers to your questions:

1.) Read the logs, as mentioned above. Then read SuSE's support resources (e.g. knowledge base, bug database) to see if this is a known FC driver bug and if there is an update to fix it.

Read the compatibility lists provided by your SAN system manufacturer: there might be a recommendation or even a requirement to install a particular update version.

Also check if your FC HBA vendor has an updated driver available for the HBA, and read its release notes if one exists.

If necessary, install the updates that are documented to fix problems like you are having.

2.) Your filesystem will need a filesystem check before it can be mounted in read-write mode again. Normally, a brief check is performed automatically at system boot: if the state of the filesystem indicates it has had problems since the last check, a full check may happen automatically. But an automatic check may not always solve all problems: sometimes it will be necessary to run the filesystem check in interactive mode, so that the fsck program can present questions to the sysadmin, who can then choose the appropriate way to solve some filesystem problems.

NOTE: the output of your "mount" command indicates the filesystem type is reiserfs. Apparently a damaged reiserfs is harder to fix than other filesystem types: with some error conditions, reiserfsck might take a long time and/or discover (parts of) already-deleted files. The resulting confusion could, in worst cases, be worse than the original error.

To be safe, I'd strongly suggest that you should ***first take a full backup*** of the filesystem as it exists now, before you reboot or do anything else. Some people have had bad experiences running (an old version of) reiserfsck.

3.) Yes, the issue can be resolved without a reboot... but probably not without unmounting the filesystem.

The recommended procedure would be:
- first, take a backup of the filesystem as it is now

- Use "fuser -m /dev/sda2" to identify processes that are still attempting to use the filesystem, and stop them.

- Unmount the filesystem:
umount /dev/sda2

- Run reiserfsck on the filesystem. For extra advice on how to do it, read this:

http://www.cyberciti.biz/tips/repairing-reiserfs-file-system-with-reiserfsck.html

- If the reiserfsck is successful, mount the filesystem again using the options configured in /etc/fstab:
mount /dev/sda2

- restart any applications, if necessary.

MK

[senthil_kumar_2]

Hi All,

Have fixed the issue using following steps:

1) # umount /

2) # reiserfsck /dev/sda2

3)Ran reiserfsck with --rebuild-tree and --fix-fixable as the above command asked to me to run to fix some errors faced by file system..

# reiserfsck --rebuild-tree /dev/sda2

# reiserfsck --fix-fixable /dev/sda2

4)Againa ran reiserfsck to check whether the error is coming again:

But this time did not get error...

5)Mounted the drive back to file system