EXT3-fs error:: Remounting filesystem read-only

[Antoine Hatzistamatiou]

Hi experts,

from time to time we face the following problem and although the extensive google search we couldn't find a solution. You are our only hope!

Hardware: HP ProLiant DL380 Generation 4 Packaged Cluster with Modular Smart Array 500 G2
Operating System: RedHat Enterprise 4
Setup: 2 DL servers are connected through SCSI Controller Smart Array 642 to the MSA500. MSA500 has a RAID5 with 3 physical drives 146Gb each.
Using LVM we created 5 logical volumes (i.e. A, B, C, D, E) and share those volumes between the servers. Any given time, no 2 servers acccess the same volume (i.e. Server1 has A and C, Server2 has B, D and E)

The problem is: from time to time one of servers (either 1 or 2) presents an EXT3-fs error and remounts filesystem as read-only. Check below the messages from /var/log/messages

--------/var/log/messages------------------

Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_free_inode: bit already cleared for inode 4063305
Apr 27 18:35:37 sms3 kernel: Aborting journal on device dm-0.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0) in ext3_delete_inode: IO failure
Apr 27 18:35:37 sms3 kernel: ext3_abort called.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Apr 27 18:35:37 sms3 kernel: Remounting filesystem read-only

--------/var/log/messages------------------

One or two minutes after this event, the other server presents the same error on his mounted filesystems.

How we solve the problem when it appears?
We unmount all 5 filesystems from both servers, fsck them and remount.

What we cannot find, is the source of the problem. Do you have any idea?

[Rob Leadbeater]

Hi,

When you say you "share those volumes between the servers" what exactly do you mean...? Are you running any clustering software ?

Are all 5 LUNs visible to both servers at the same time ?

It sounds very much like both systems are accessing the LUNs at same time without any clustering software which is causing the file system to become corrupt...

Hoep this helps,

Regards,

Rob

[Antoine Hatzistamatiou]

Hi Rob,

we are using HighAvailability version 1(http://www.linux-ha.org/) as clustering software.

"Are all 5 LUNs visible to both servers at the same time ?" NO they aren't. Actually there are 6 LUNs

Our current configuration is:
Server 1 has 1 LUN
Server 2 has rest of the LUNs

df on Server 1:
/dev/mapper/db_vg-lv1
120758436 57693168 59384768 50% /svc/db


df on Server 2:
/dev/mapper/mcp_vg-lv1
19610300 771108 17843048 5% /svc/mcp
/dev/mapper/sms_gw_vg-lv1
19610300 6449548 12164608 35% /svc/sms_gw
/dev/mapper/web_vg-lv1
19610300 5868448 12745708 32% /svc/web
/dev/mapper/tomcat_vg-lv1
19610300 608252 18005904 4% /svc/tomcat
/dev/mapper/mcp2_vg-lv1
18578172 1010392 16624064 6% /svc/mcp2

regards,

Antoine

[Ivan Ferreira]

Please check this page and follow the instructions there, in the section:

Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?

Remember that SAN is just SCSI over fibre, so the same rules could apply.

http://www.redhat.com/magazine/026dec06/features/tips_tricks/

[palaniappan.sp]

Hi,
Did u try by running fsck during booting time??
after booting,
#cd /
#touch /forcefsck
#reboot
Next time when stystem boots,system has to run fsck b4 mounting the filesystem.

In scsi bios setup all hdd is showing ok??check the scsi cables and hdd also..

regds,
palani

[Antoine Hatzistamatiou]

Ivan hi,

thank you for pointing out something we didn't had in mind at all.

Although no changes described in the article you sent us happened in our systems. Actually no changes at all happens.

However we will try to monitor SCSI timeouts and somehow corelate them with our problem. Which will be the difficult task since we can not reproduce it. We just wait to happen.

Could you pls give us input on how to monitor SCSI timeouts (or other parameters needed at your opinion) on our systems?

thanks in advance
\Antoine