DL380 G4 Packaged Cluster with MSA500 G2::EXT3-fs error, Remounting filesystem read-only

[Antoine Hatzistamatiou]

Hi experts,

from time to time we face the following problem and although the extensive google search we couldn't find a solution. You are our only hope!

Hardware: HP ProLiant DL380 Generation 4 Packaged Cluster with Modular Smart Array 500 G2
Operating System: RedHat Enterprise 4
Setup: 2 DL servers are connected through SCSI Controller Smart Array 642 to the MSA500. MSA500 has a RAID5 with 3 physical drives 146Gb each.
Using LVM we created 5 logical volumes (i.e. A, B, C, D, E) and share those volumes between the servers. Any given time, no 2 servers acccess the same volume (i.e. Server1 has A and C, Server2 has B, D and E)

The problem is: from time to time one of servers (either 1 or 2) presents an EXT3-fs error and remounts filesystem as read-only. Check below the messages from /var/log/messages

--------/var/log/messages------------------

Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_free_inode: bit already cleared for inode 4063305
Apr 27 18:35:37 sms3 kernel: Aborting journal on device dm-0.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0) in ext3_delete_inode: IO failure
Apr 27 18:35:37 sms3 kernel: ext3_abort called.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Apr 27 18:35:37 sms3 kernel: Remounting filesystem read-only

--------/var/log/messages------------------

One or two minutes after this event, the other server presents the same error on his mounted filesystems.

How we solve the problem when it appears?
We unmount all 5 filesystems from both servers, fsck them and remount.

What we cannot find, is the source of the problem. Do you have any idea?

thanks in advance

[Serviceguard for Linux]

What clustering SW are you using?

If you are not using clustering SW then you need to understand that ext3 and LVM are not designed to be shared between servers. The clustering SW controls access and prevents potential data corruption.

If both servers need access to the same file system at the same time, then you need a clustered file system as well.

[Antoine Hatzistamatiou]

We are using HA (www.linux-ha.org) software for clustering.