Simpler Navigation for Servers and Operating Systems
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
General
cancel
Showing results for 
Search instead for 
Did you mean: 

EXT3-fs error:: Remounting filesystem read-only

Antoine Hatzistamatiou
Occasional Contributor

EXT3-fs error:: Remounting filesystem read-only

Hi experts,

from time to time we face the following problem and although the extensive google search we couldn't find a solution. You are our only hope!

Hardware: HP ProLiant DL380 Generation 4 Packaged Cluster with Modular Smart Array 500 G2
Operating System: RedHat Enterprise 4
Setup: 2 DL servers are connected through SCSI Controller Smart Array 642 to the MSA500. MSA500 has a RAID5 with 3 physical drives 146Gb each.
Using LVM we created 5 logical volumes (i.e. A, B, C, D, E) and share those volumes between the servers. Any given time, no 2 servers acccess the same volume (i.e. Server1 has A and C, Server2 has B, D and E)

The problem is: from time to time one of servers (either 1 or 2) presents an EXT3-fs error and remounts filesystem as read-only. Check below the messages from /var/log/messages

--------/var/log/messages------------------

Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_free_inode: bit already cleared for inode 4063305
Apr 27 18:35:37 sms3 kernel: Aborting journal on device dm-0.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0) in ext3_delete_inode: IO failure
Apr 27 18:35:37 sms3 kernel: ext3_abort called.
Apr 27 18:35:37 sms3 kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Apr 27 18:35:37 sms3 kernel: Remounting filesystem read-only

--------/var/log/messages------------------

One or two minutes after this event, the other server presents the same error on his mounted filesystems.

How we solve the problem when it appears?
We unmount all 5 filesystems from both servers, fsck them and remount.

What we cannot find, is the source of the problem. Do you have any idea?
5 REPLIES
Rob Leadbeater
Honored Contributor

Re: EXT3-fs error:: Remounting filesystem read-only

Hi,

When you say you "share those volumes between the servers" what exactly do you mean...? Are you running any clustering software ?

Are all 5 LUNs visible to both servers at the same time ?

It sounds very much like both systems are accessing the LUNs at same time without any clustering software which is causing the file system to become corrupt...

Hoep this helps,

Regards,

Rob
Antoine Hatzistamatiou
Occasional Contributor

Re: EXT3-fs error:: Remounting filesystem read-only

Hi Rob,

we are using HighAvailability version 1(http://www.linux-ha.org/) as clustering software.

"Are all 5 LUNs visible to both servers at the same time ?" NO they aren't. Actually there are 6 LUNs

Our current configuration is:
Server 1 has 1 LUN
Server 2 has rest of the LUNs

df on Server 1:
/dev/mapper/db_vg-lv1
120758436 57693168 59384768 50% /svc/db


df on Server 2:
/dev/mapper/mcp_vg-lv1
19610300 771108 17843048 5% /svc/mcp
/dev/mapper/sms_gw_vg-lv1
19610300 6449548 12164608 35% /svc/sms_gw
/dev/mapper/web_vg-lv1
19610300 5868448 12745708 32% /svc/web
/dev/mapper/tomcat_vg-lv1
19610300 608252 18005904 4% /svc/tomcat
/dev/mapper/mcp2_vg-lv1
18578172 1010392 16624064 6% /svc/mcp2

regards,

Antoine
Ivan Ferreira
Honored Contributor

Re: EXT3-fs error:: Remounting filesystem read-only

Please check this page and follow the instructions there, in the section:

Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?

Remember that SAN is just SCSI over fibre, so the same rules could apply.

http://www.redhat.com/magazine/026dec06/features/tips_tricks/
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
palaniappan.sp
Regular Advisor

Re: EXT3-fs error:: Remounting filesystem read-only

Hi,
Did u try by running fsck during booting time??
after booting,
#cd /
#touch /forcefsck
#reboot
Next time when stystem boots,system has to run fsck b4 mounting the filesystem.

In scsi bios setup all hdd is showing ok??check the scsi cables and hdd also..

regds,
palani



Everything is Possible and Anything is Feasible if u try
Antoine Hatzistamatiou
Occasional Contributor

Re: EXT3-fs error:: Remounting filesystem read-only

Ivan hi,

thank you for pointing out something we didn't had in mind at all.

Although no changes described in the article you sent us happened in our systems. Actually no changes at all happens.

However we will try to monitor SCSI timeouts and somehow corelate them with our problem. Which will be the difficult task since we can not reproduce it. We just wait to happen.

Could you pls give us input on how to monitor SCSI timeouts (or other parameters needed at your opinion) on our systems?

thanks in advance
\Antoine