Rhel 5u3 issues

[brian_31]

We have a server(5u3) which would not boot and here is what i have noticed..

The dev/sda (which has the root disk) seems to have lost root. We have a oscopy script which does dd internally which fails with the following error..Kernel Panic..not syncing. Even if we try to create a partition on /dev/sda(that part is sucessfull) and try to create a filesystem on it it errors with the same message."Kernel Panic..not syncing".what could this message mean? I also noticed that the fstab has /dev/sdb listed as swap but the partition number was 83 (i thought 82 should be for swap). We went into linux rescue but we are stuck as we are not able to do the oscopy to have the root built (on /dev/sda) because of this error. Any ideas?

Thanks

Brian.

[Matti_Kurkela]

"Kernel Panic..not syncing" is only part of the panic message. It means something may have corrupted kernel's code or data structures, and the only safe thing the kernel can do is to stop *everything*.

The complete Linux kernel panic message should look like this:
http://en.wikipedia.org/wiki/File:Ubuntu-linux-kernel-panic-by-jpangamarca.JPG

The "Call Trace" part would allow someone who knows about the kernel's internal structures to figure out what was happening when the error was detected. In the picture, it looks like a problem in libata's interrupt handler, so perhaps an IDE/SATA controller returned corrupted data or did something else the driver did not expect, or perhaps something else had corrupted the memory pointer that told the system where the data from disk should be stored.

You cannot use dd to overwrite a root filesystem while the system has it mounted in read/write mode; that would be a sure way to crash the system. Find out what the oscopy does (you said it's a script, so *read* it!) and do the equivalent operation manually using rescue mode.

If the rescue mode crashes too when trying to write to the root disk, then you may have a hardware problem: a subtle kind of disk failure, or even a disk controller failure.

The partition type numbers are pretty unimportant for Linux: if /etc/fstab says some partition should be used for swapping and it has been initialized with the mkswap command, then it *will* be used as swap, no matter what partition type number is used. (The only exception is partitions used for Linux software RAID: the RAID driver will auto-activate RAID partitions only if they have the correct type.)

Using the correct partition type numbers anyway is a good practice, because it allows third-party tools (like Partition Magic, for reorganizing traditional PC disk partitions) to easily recognize the partition and choose the correct method for handling it.

MK

[brian_31]

Thanks MK!. I will try to do the command in the script (I am sure it is dd command) and post the results.

Best Regards

Brian.

[brian_31]

One thing though, why would i get the same Kernel Panic message when i create a small partition (in rescue mode) on the same /dev/sda and try to do a filesystem on it. Would that confirm a subtle harddisk failure or a harddisk controller failure? Any other logs etc i could see to confirm this error that i could provide to the vendor?

Best Regards

Brian.

[Matti_Kurkela]

Yes, if even the rescue mode fails when trying to create a filesystem, then it suggests a hardware problem of some sort.

A disk failure would be statistically more probable: try replacing the disk. If the problem goes away, the disk had a problem of some unusual type. In that case, it might be interesting to try the problematic disk on a system with a newer kernel, for example a RHEL 5.5 system. If it is able to produce a more informative error message, fine. If it crashes too, you might want to submit a bug report to RedHat, with the full panic message attached if at all possible. (If the problem is reproducible, the panic message might be easiest to capture by using a serial console.)

If the problem remains the same after replacing the disk, then the disk controller is probably faulty. That's a pretty rare type of hardware failure, but it happens sometimes.

MK