Kernel panic - not syncing : Attempted to kill init !

[Matti_Kurkela]

This means GRUB has successfully loaded your kernel and initrd. LVM seems to be OK, but it looks like your root filesystem cannot be found for some reason.

RHEL 4 used the driver disk images differently than the later RHEL releases. If you have another Linux system available, try loop-mounting the uncompressed driver image file:

mkdir -p /mnt/driverdisk
mount -o loop,ro cpq_cciss-2.6.20-34.rhel4.i686.dd /mnt/driverdisk

 Then burn all the files you find at /mnt/driverdisk to a CD, and try giving this CD to the installer/rescue mode as a driver disk.

[Afaq]

HI,

After installing the drivers now i can able to see the logical disk in /dev/cciss/* and /proc/partitions.

Spoiler

Spoiler

If the rescue mode works with the driver disk and the partitions/logical volumes get mounted to /mnt/sysimage, the next step is to check the /boot filesystem (now /mnt/sysimage/boot). Each kernel file (vmlinuz-<version>) should have a corresponding initrd file (initrd-<version>.img).

and in resucue mode volumes are mounted on /mnt/sysimage

/dev/VolGroup00/LogVol00                                       /mnt/sysimage

/dev/cciss/c0d0p1                                                        /mnt/sysimage/boot

/dev/VolGroup00/LogVol02                                       /mnt/sysimage/data

i check the /boot filesystem (/mnt/sysimage/boot), each Kernel file (vmlinuz- 2.6.9-67.0.20.EL) have corresponding initrdfile ( initrd-2.6.9-67.0.20.EL.img)

shell i create a new initrd file ?

please suggest me....

[Matti_Kurkela]

>...each Kernel file (vmlinuz-2.6.9-67.0.20.EL) have corresponding initrdfile ( initrd-2.6.9-67.0.20.EL.img)

(The  vmlinuz-2.6.9-67.0.20.EL is a single-processor kernel. Proliant DL360 G5 has either dual-core or quad-core CPU(s), so it would benefit from a multi-processor kernel. With the single-processor kernel, only one core is being used.)

Wait... I think I understand now.

2.6.9-67.0.20.EL is an errata kernel.

The original release kernel for RHEL 4 Update 6 was 2.6.9-67.EL (or .ELsmp for a multi-processor flavor) - and that version is still used by the rescue mode.

I guess that when the system was installed, a driver disk was used, and then an updated cpq_cciss RPM may or may not have been installed.

At some later date, someone updated the kernel version to 2.6.9-67.0.20, but forgot to check for corresponding updates to the cpq_cciss RPM - and apparently did not check that the system was bootable afterwards. Because it was not bootable.

The latest cpq_cciss RPM that supports RHEL 4 Update 6 (cpq_cciss-2.6.20-34.rhel4.x86_64.rpm) has this text in its Release Notes:

SUPPORTED KERNELS:
The kernels of Red Hat Enterprise Linux 4 (AMD64/EM64T) supported by this binary rpm are:
2.6.9-67.EL - Red Hat Enterprise Linux 4 Update 6 (AMD64/EM64T)
2.6.9-67.0.1.EL
2.6.9-67.0.4.EL
2.6.9-67.0.7.EL
2.6.9-67.0.15.EL
2.6.9-78.EL - Red Hat Enterprise Linux 4 Update 7 (AMD64/EM64T)
2.6.9-89.EL - Red Hat Enterprise Linux 4 Update 8 (AMD64/EM64T)
2.6.9-89.0.7.EL
2.6.9-89.0.9.EL
2.6.9-89.0.11.EL
2.6.9-89.0.16.EL
2.6.9-89.0.18.EL
2.6.9-89.0.20.EL

 Note that version 2.6.9-67.0.20 is not listed - so it is not supported.

The newest cpq_cciss RPM for RHEL4 (cpq_cciss-2.6.20-36.rhel4.x86_64.rpm) only supports RHEL 4 Update 7 and Update 8.

Now you must choose what you want to do:

• the best option would probably be to update the kernel to RHEL 4 Update 9 levels (version 2.6.9-100.* or newer), since the RHEL 4 Update 9 kernels include the updated cciss driver as standard. If you do this, the cpq_cciss RPM should become completely unnecessary and can be removed.
• If you cannot upgrade that far, choose one of the supported kernel versions for RHEL 4 Update 7 or 8 (see above).
• if you cannot upgrade at all, consider downgrading the kernel to 2.6.9-67.0.15, the last HP-supported RHEL 4 Update 6 kernel.
• If you absolutely must stay with the 2.6.9-67.0.20, find the sources for the updated cciss module at /opt/hp/storage_drivers (if any version of the cpq_cciss RPM is installed) and try and compile it yourself.

Whatever you choose to do, first run "chroot /mnt/sysimage" in rescue mode, so that you can use the complete set of system commands, and the package management tools will work without extra tricks.

If you choose to upgrade, first install the higher version kernel (or kernel-smp) RPM, then the appropriate cpq_cciss RPM (if necessary). The installation of the cpq_cciss RPM will automatically create a new initrd for you, after replacing the standard cciss module with the updated version.

Unfortunately I don't have a RHEL 4 test VM at the moment, so I cannot check how hard would it be to compile the updated cciss module manually.

[Afaq]

Thank you so much MK.

let me check wheather we can upgrade or not.

in rescue mode "chroot /mnt/sysimage" was not working its giving " cannot run command '/bin/sh': input/output error".

now i'm in rescue mode i can able to see all the mount point and files. Is it possible to take the backup of my files ?

[Matti_Kurkela]

Apparently there is some corruption in your root filesystem after all.

You can use the rescue mode without entering the "chroot /mnt/sysimage" command too, but then you will have a reduced set of commands available, and all pathnames of your files will have /mnt/sysimage prefixed to them. If that's enough to back up your files, then do it. Otherwise you'll need to find the problem that stops you from running the standard shell in chroot mode, and fix it.

I think "rpm --root /mnt/sysimage -Va" should allow you to verify all the files managed by the RPM package manager on your system. The command should list all the files that have been modified after they have been installed from RPMs. For some files (e.g. configuration files and log files), this is expected. But if you see something like /bin/bash or the system libraries in /lib have changed, you might want to reinstall the respective RPM.

It should be possible to reinstall RPMs in rescue mode without the chroot command, using commands like "rpm --root /mnt/sysimage --replacepkgs -ivh <pathname of the RPM>". Once the chroot command starts working, many things - including backing up your files - will be easier.

Alternatively, you can find some Live-CD-type Linux distribution, and boot the system from it.

(For example, Knoppix used to be good for rescue purposes: in the past, I've used it a few times for purposes like this.)

You may have to activate the VG and mount the filesystems manually, but otherwise the Live CD environment might be more user-friendly and have more tools available than the RHEL 4 rescue mode.