topic Re: OS / kernel / memory corruption? in Operating System - Linux

OS / kernel / memory corruption?

Matt Shaffer_1 — Wed, 21 May 2008 19:27:26 GMT

running rhel3u4 on hp ml350 g4. mirrored hard drives. 1G of mem and 1 CPU. after a reboot friday night we've been seeing strange things on server. example: when run I run ls -la in this one particular dir I get I/O errors and some of the files have time stamp of 1970.

Also, I can create a file in this dir and I can rsync to the dir but not all files get rsynced.

I see errors when I run dmesg and there are errors in /var/log/messages.1. I've attached the I/O errors and the errors from messages.1.

Re: OS / kernel / memory corruption?

Ivan Ferreira — Wed, 21 May 2008 19:35:39 GMT

The I/O errors is to be worried about. How is mirrored hardware/software? What is the current status of the mirrors? I have seen hardware RAID with a disk failure to produce file system corruption (No RAID 0). This could be a problem with the module or the controller itself.

You should take a backup of what you can, check the mirror status, and run fsck.

Re: OS / kernel / memory corruption?

Matt Shaffer_1 — Wed, 21 May 2008 19:42:14 GMT

The mirror is on the hardware level. we have a smart array 641 card installed.

Re: OS / kernel / memory corruption?

Matt Shaffer_1 — Wed, 21 May 2008 19:42:44 GMT

mirror looks good

Re: OS / kernel / memory corruption?

macosta — Thu, 22 May 2008 12:27:24 GMT

I agree with Ivan - this smells of filesystem corruption.

Also, on a box with 8 CPUs, why are all disabled except CPU0 and CPU4?? According to the DOC you attached, this is the case.

Re: OS / kernel / memory corruption?

Matt Shaffer_1 — Thu, 22 May 2008 14:43:48 GMT

this server only has 1 cpu.

Re: OS / kernel / memory corruption?

macosta — Thu, 22 May 2008 15:05:11 GMT

Matt, are you sure you posted the right messages file then? I see a system startup, no obvious filesystem errors, and this:

May 17 00:23:21 7758 kernel: 8 CPUs total
May 17 00:23:21 7758 kernel: Local APIC address fee00000
...
May 17 00:23:21 7758 kernel: Processors: 2

This indicates 8 physical processors, of which 2 are enabled and available to the OS.

In the end, backup and either fsck or start with a fresh filesystem and restore.

Re: OS / kernel / memory corruption?

Matt Shaffer_1 — Tue, 23 Nov 2010 15:29:54 GMT

mirror was corrupt. had to restore using HP DPX OBDR.