EXT3-fs error getting on DELL 2650 system.

[Ameet_HP]

We are facing "EXT3-fs error" problem very often on Dell 2650 systems, running Red Hat Linux Advanced Server release 2.1AS/i686, 2.4.9-e.24enterprise. This error does not even allow to login on console, forces to cold boot the server. Check Errors in attached jpg file. Dell is unable to fix it in past more than a month. Did anyone face the same problem & it is cleared ? Your comments please..

[Steven E. Protter]

This could be a hardware problem.

Are all the disks good?

What is your storage setup? Could be an issue with your fiber connection if you are using one.

I've been running various versions of Red Hat on Dell hardware for years and have never run into this issue.

Its always a good idea to provide more data to get better help around here.

SEP

[Jerome Henry]

It definetely seems to be hardware related, seemingly to your disk chip or controler. Does fschk run at boot time ? fix the issue ? Or fail ?
If it fixes, then the problem should come from your disk at random places (such a deficient head, or dust moving on disk)
If it doesn't, then you have deficient clusters.

You may run fschk on unmounted system manually too...

Seems to be time to replace HD...

J

[Ameet_HP]

Belive or not.. Hard Disks are replaced 5 times in just 3 months on 4 brand new Dell systems. We ae using "DELL PowerEdge Expandable RAID Controller 2 Command Line Interface" on 4 systems. Firmware is up to date. We have created to Controllers, first is mirror userd for Root partition of Linux, second is Stripe with 2 disks, used for data which makes 2 * 18GB disks in one contrller = 36 GB for big oracle data. Sometimes I see 1 Disk missing from controller. Dell asks to run SCSI utility with it to verify disk where it passes 100 % but does not come in Controller.

System runs fsck on all filesystems & cleans the error I have shown in jpg file. Sometimes it requires to change disk, but mostly with booting issue gets cleared. It does not have any external disks or Fibre cards. Still getting I/O error is unknown. Dell is asking me to upgrade Linux to e27 kernel version which is funny solution to upgrade OS to a month back new version to save hardware problem.

[Jerome Henry]

Wow ! My 2 cents there is a power problem on their mobo !

5 disks in such a short time... or can be that they are not able to issue correct drivers for the hardware... reminds me problems on the first day of 2.4 release... many manufacturers werea nouncing hardware spcifications... that were required 100 % by the kernel... and making the hardware hang, as the real specifications were far lower than the one the marketing team was claiming... Could be the same issue here...

Fun to think that maybe turning to older version could require less from the hardware...
:]

J

[Stuart Browne]

I agree it's a hardware problem, but as it is only affecting the SCSI IO subsystems, I'd beleive it to be either a hot-swap backplane or cabling.

I know on the DELL's I've dealt with, they are rarely 'spacious' on the inside. I've seen cables in dell machines twisted, turned, folded, squished, turned, unfolded, then plugged into a split back-plane. *whee!* Just how many folds can these cables take? :P

The last Dell server we had go out for one of our clients (they bought it), we had similar issues (it was running SCO OSR5, not Linux). It turned out to be one of the cables shorting out with the case cover.

It took the replacing of pretty much everything before they figured that one out! Much fun.

[Huc_1]

Testing one piece of hardware compoment at a time could be a very lenghty and costly process, ex; change cables ... wait for it to perhaps fail , remplace backplane wait for it to perhaps fails, place power supply wait for it to perhaps fail.

and each time stopping users/applic and downtime...

The easy solution could be to negotiate with Dell they remplace all the box expect your disks then if this work well the "box" is at fault and disk are good

Could point out to Dell you already have sufferd a lot and that it is becoming costly etc ..

just a 0.02 Euro cent.

J-P

[Joan Dolson]

On our Dell 1650 systems with the RAID controller and running Red Hat 7.3, we had to apply a kernel patch to correct problems with ext3 filesystems. We used Red Hat up2date to apply the latest kernel patches available at the time, which corrected the problem.

[Ameet_HP]

Joan,

Dell is insisting me same thing to upgrade kernel version to the latest e27. In fact, if your problem in RH 7.3 was fixed with some patch, RH 8 should have it in built already. Currently, I have upgraded BIOS version on all Dell systems. Lets see if we get this problem again. If yes, I will upgrade to e24 to e27 all systems.

These systems do not have cables, HDs are connected with Backplane directly. It has been replaced on 2 systems of which 1 system is still facing this problem.

Keep ading your comments.

Thanks,
Ameet

[Ameet_HP]

Joan,

Dell is insisting me same thing to upgrade kernel version to the latest e27. In fact, if your problem in RH 7.3 was fixed with some patch, RH 8 should have it in built already. Currently, I have upgraded BIOS version on all Dell systems. Lets see if we get this problem again. If yes, I will upgrade to e24 to e27 all systems.

These systems do not have cables, HDs are connected with Backplane directly. It has been replaced on 2 systems of which 1 system is still facing this problem.

Keep adding your comments.

Thanks,
Ameet

[Kevin McMahon]

I'm seeing the same issue with my Dell 2650 systems. I went from a RedHat 7.2 based OS to a RedHat 9 based OS, running a 2.4.20-18.9 kernel, but I'm still getting the problem on multiple machines. Have you found a fix for the problem?

[Mark Grant]

IN all probablility, this is a hardware error but it doesn't actually have to be.

I would attempt to backup your data, destroy the file system completely, re-create it again and see if that helps.

It might also be worth trying a different filesystem type. Maybe create a reiserfs file system and give it a bit of a pounding.

[Ameet_HP]

Guys,

Just for your information, Dell 2650 does not have any solution till the date. As I have upgraded kernel till 2.4.9e30, it is not fixed yet. Also, I changed Network Module to bcm5700, tg3 with latest updates, but still same.. system keeps on giving this error.

Ameet