Kernel panic - not syncing : Attempted to kill init !

[Afaq]

HI experts,

My RHEL server was crashed and giving the below message

" kernel panic - not syncing: Attempted to kill init !" at boot time.

i was tried to restore the grub from the resuce mode, but its giving device not found and going to shell prompt, there its not showing /mnt/sysimage.

when i tried to mount manually it its giving device not found.

please help me to recover operating system without loss of any data.

server model : hp dl360 g5

having 3 HDD, using smart array created mirrored logical volume of raid 1/0 and one disk was hot spare.

Best Regards

Afaq

[Matti_Kurkela]

> " kernel panic - not syncing: Attempted to kill init !" at boot time.

This is not a GRUB problem, then. If the kernel is panicing, it means GRUB has already successfully loaded the kernel (and maybe initrd too?). So at least the /boot directory/filesystem is somewhat readable.

The first "init" process is a script within the initrd: it is supposed to handle the early boot tasks (mostly finding and mounting the real root filesystem in read-only mode) and then exec() the real /sbin/init. When a RHEL server is failing with this message, it probably means the real root filesystem could not be found or accessed.

In rescue mode, is the cciss.ko module loaded? Run "modprobe cciss" to load it if necessary.

Then, does the /dev/cciss/ directory contain any device nodes?

If no device nodes are visible, it means the RAID controller cannot present any RAID logical disks, even in degraded mode.

In that case, reboot the system, and watch the BIOS messages. When you see the SmartArray controller messages, press F8 to access the SmartArray configuration. Does it list any logical disks? Are they OK, degraded or outright failed?

What is the status of the physical disks?

If you interrupt the automatic boot of the GRUB boot loader and then press "e" to view/edit the boot entry, what is your current "root=" parameter? Is it like "root=/dev/cciss/<something>", or something else (like "root=/dev/VolGrp00/LogVol00", indicating that LVM is being used)?

[Afaq]

HI,

1. In rescue mode cciss.ko module was not loaded and i was try to load it but it was failed

2. Smart Array Controller displays one logical disk raid 1/0 and one is spared they are ok

3. physical disks are ok

4. i interrupt the automatic Grub boot loader and check the current root parameter

it was "root=/dev/VolGrp00/LogVol00"

[Matti_Kurkela]

> 1. In rescue mode cciss.ko module was not loaded and i was try to load it but it was failed

The version of the cciss driver on your install media is probably too old, and a driver disk was used when installing the OS.

You'll need to use a driver disk with the rescue mode too.

> 2. Smart Array Controller displays one logical disk raid 1/0 and one is spared they are ok

> 3. physical disks are ok

Good, so your data might be recoverable, maybe even without too much effort.

> 4. it was "root=/dev/VolGrp00/LogVol00"

Looks like a default LVM layout - OK.

Go to http://www.hp.com/go/support and select "Drivers & Software". Type in your server model: "DL360 G5" and press Enter. You should get to a page that lists all the supported OS versions for the server model (direct link here).

Choose the RHEL version and architecture (32 or 64 bit) and you'll get to the download page.

Check the category "Software - Driver update": there you'll find the driver disk images for your OS version.

Click on the name of the download, not the blue "Download" button: you'll see a page that contains more information about that download, including the installation instructions. The procedure varies a bit between OS versions. Depending on the RHEL version, you might also be able to mount the disk image file with "mount -o loop,ro" on another Linux system and copy the contents (or even just the .dd file itself) onto a USB stick or burn it to a CD.

The RHEL Installation Guide for your RHEL version will have more information on all the different ways to use a driver disk image.

Then add "dd" to the rescue mode boot parameters (e.g. if you normally boot to rescue mode by typing "rescue" to the boot prompt, now type "rescue dd".)

The rescue mode system should now automatically prompt you to insert the rescue disk media in an appropriate point. With the updated cciss driver, it should recognize the controller and load the appropriate module automatically. At that point, you should at least see the partitions of the Smart Array logical disk in /dev/cciss/* and /proc/partitions. If your system is not too badly damaged, the rescue mode should now automatically mount the partitions to /mnt/sysimage.

When in rescue mode, all the LVM commands are packaged into a single binary named "lvm" to save space, so instead of "vgscan" and "vgchange -a y", you should use commands "lvm vgscan" and "lvm vgchange -a y". (If the rescue mode still fails to find your OS installation, those are the commands you need to try and activate the VG manually.)

If the rescue mode works with the driver disk and the partitions/logical volumes get mounted to /mnt/sysimage, the next step is to check the /boot filesystem (now /mnt/sysimage/boot). Each kernel file (vmlinuz-<version>) should have a corresponding initrd file (initrd-<version>.img).

If the initrd file for your current kernel is missing, it is easy enough to recreate it: first "chroot /mnt/sysimage", then "mkinitrd /boot/initrd-<version>.img <version>". Note that you will have to enter the kernel version string twice, and the string must exactly match the version string of the kernel file. After entering the chroot command, you should have all the normal RHEL system commands available to you, almost as if the system was in a normal single user mode.

If the rescue mode fails to mount your filesystems automatically, you may have to manually run a filesystem check (fsck) to one or more partitions and/or LVM logical volumes, then try again.

[Afaq]

HI MK,

i was downloaded the storage controller driver which is a .rpm file, how can i use this rpm as a driver could you give me the steps its very helpfull.

[Matti_Kurkela]

Sorry, but that's not the right file. You cannot use the .rpm file to load an updated driver for the installer/rescue mode.

You'll want a download named like "Driver Diskette for HP Smart Array Controller (<architecture>) for RedHat Enterprise Linux <version>". It is in the "Software - Driver Update" category. It should have a suffix like ".dd.gz". It is a specially packaged version of the driver that can be directly used by the installer/rescue environment.

And *please* tell me your RHEL version.

For this server, HP offers driver disks for RHEL 3, 4, 5 and 6, and most of them have both 32- and 64-bit versions, and at least RHEL 5 has different driver disk versions for different RHEL 5.x Update levels too. If I knew your RHEL version, I could give you direct links to both the download and the instructions.

For finding the correct driver diskette image, it would be essential to know the version and update level of the RHEL media you're using to start the rescue mode. Look at the beginning of the RELEASE-NOTES-<language>.html files in the root directory of the RHEL media. If it's RHEL 5 or 6, the version should be indicated as "5.x" or "6.x"; if there is no number like that visible, your media is of the "initial release" version. If it is RHEL 4 or older, the version can be listed as e.g. "RHEL 4 Update 9".

[Afaq]

HI,

I was tried by using the HP insight manager disk for drivers but its not recognizing.

right now we have RHEL 4 32bit OS and the kernel release  is 2.6.9-67.ELamp

[Matti_Kurkela]

Here's a direct link to the driver disk download page.The page also contains instructions for using it - but I guess you probably don't have a floppy disk drive in your system.

An alternative way to use the driver disk image is to uncompress the file:

gunzip cpq_cciss-2.6.20-34.rhel4.i686.dd.gz

Then burn the file to a CD-ROM, so that the root directory of the CD has only the file "cpq_cciss-2.6.20-34.rhel4.i686.dd".

Then boot the server to the rescue mode as you've done before, but add the "dd" boot option.

Before switching to rescue mode, the installer will ask: "Do you want to use a driver disk?" Answer Yes, then insert the CD with the driver disk image (or the floppy, if you followed HP's instructions and have a floppy drive available). After reading the driver disk, the installer will ask "Do you have any more driver disks?" Answer No.

Now the rescue mode should start with the updated cciss driver.

In RHEL 4, kernel version 2.6.9-67.ELsmp is the release kernel version for RHEL 4 Update 6. That's actually the oldest kernel version that is supported by this driver disk.

[Afaq]

i was downloaded the file cpq_cciss-2.6.20-34.rhel4.i686.dd.gz and uncompressed it, then burn it to CDROM.

i used a command at boot time "linux dd" it was asked "Do you want to use a driver disk?" then i select yes and insert the driver disk into CDROM device /dev/hda then i select ok, but its not reading the disk. saying insert driver disk into /dev/hda !!!

[Afaq]

HI,

below is the error message i'm getting

Uncompressing Linux......ok, booting the kernel.

Red Hat nash version 4.2.1.13 starting

Reading all physical volumes. This may take a while .....

Found volume group "VolGroup00" using metadata type lvm2

3 logical volume(s) in volume group "VolGroup00" now active

WARNING: can't access (null)

exec of init ((null)) failed !!! : 14

umount /initrd/dev failed : 2

Kernel panic - not syncing : Attempted to kill init !

[Matti_Kurkela]

This means GRUB has successfully loaded your kernel and initrd. LVM seems to be OK, but it looks like your root filesystem cannot be found for some reason.

RHEL 4 used the driver disk images differently than the later RHEL releases. If you have another Linux system available, try loop-mounting the uncompressed driver image file:

mkdir -p /mnt/driverdisk
mount -o loop,ro cpq_cciss-2.6.20-34.rhel4.i686.dd /mnt/driverdisk

 Then burn all the files you find at /mnt/driverdisk to a CD, and try giving this CD to the installer/rescue mode as a driver disk.

[Afaq]

HI,

After installing the drivers now i can able to see the logical disk in /dev/cciss/* and /proc/partitions.

Spoiler

Spoiler

If the rescue mode works with the driver disk and the partitions/logical volumes get mounted to /mnt/sysimage, the next step is to check the /boot filesystem (now /mnt/sysimage/boot). Each kernel file (vmlinuz-<version>) should have a corresponding initrd file (initrd-<version>.img).

and in resucue mode volumes are mounted on /mnt/sysimage

/dev/VolGroup00/LogVol00                                       /mnt/sysimage

/dev/cciss/c0d0p1                                                        /mnt/sysimage/boot

/dev/VolGroup00/LogVol02                                       /mnt/sysimage/data

i check the /boot filesystem (/mnt/sysimage/boot), each Kernel file (vmlinuz- 2.6.9-67.0.20.EL) have corresponding initrdfile ( initrd-2.6.9-67.0.20.EL.img)

shell i create a new initrd file ?

please suggest me....

[Matti_Kurkela]

>...each Kernel file (vmlinuz-2.6.9-67.0.20.EL) have corresponding initrdfile ( initrd-2.6.9-67.0.20.EL.img)

(The  vmlinuz-2.6.9-67.0.20.EL is a single-processor kernel. Proliant DL360 G5 has either dual-core or quad-core CPU(s), so it would benefit from a multi-processor kernel. With the single-processor kernel, only one core is being used.)

Wait... I think I understand now.

2.6.9-67.0.20.EL is an errata kernel.

The original release kernel for RHEL 4 Update 6 was 2.6.9-67.EL (or .ELsmp for a multi-processor flavor) - and that version is still used by the rescue mode.

I guess that when the system was installed, a driver disk was used, and then an updated cpq_cciss RPM may or may not have been installed.

At some later date, someone updated the kernel version to 2.6.9-67.0.20, but forgot to check for corresponding updates to the cpq_cciss RPM - and apparently did not check that the system was bootable afterwards. Because it was not bootable.

The latest cpq_cciss RPM that supports RHEL 4 Update 6 (cpq_cciss-2.6.20-34.rhel4.x86_64.rpm) has this text in its Release Notes:

SUPPORTED KERNELS:
The kernels of Red Hat Enterprise Linux 4 (AMD64/EM64T) supported by this binary rpm are:
2.6.9-67.EL - Red Hat Enterprise Linux 4 Update 6 (AMD64/EM64T)
2.6.9-67.0.1.EL
2.6.9-67.0.4.EL
2.6.9-67.0.7.EL
2.6.9-67.0.15.EL
2.6.9-78.EL - Red Hat Enterprise Linux 4 Update 7 (AMD64/EM64T)
2.6.9-89.EL - Red Hat Enterprise Linux 4 Update 8 (AMD64/EM64T)
2.6.9-89.0.7.EL
2.6.9-89.0.9.EL
2.6.9-89.0.11.EL
2.6.9-89.0.16.EL
2.6.9-89.0.18.EL
2.6.9-89.0.20.EL

 Note that version 2.6.9-67.0.20 is not listed - so it is not supported.

The newest cpq_cciss RPM for RHEL4 (cpq_cciss-2.6.20-36.rhel4.x86_64.rpm) only supports RHEL 4 Update 7 and Update 8.

Now you must choose what you want to do:

• the best option would probably be to update the kernel to RHEL 4 Update 9 levels (version 2.6.9-100.* or newer), since the RHEL 4 Update 9 kernels include the updated cciss driver as standard. If you do this, the cpq_cciss RPM should become completely unnecessary and can be removed.
• If you cannot upgrade that far, choose one of the supported kernel versions for RHEL 4 Update 7 or 8 (see above).
• if you cannot upgrade at all, consider downgrading the kernel to 2.6.9-67.0.15, the last HP-supported RHEL 4 Update 6 kernel.
• If you absolutely must stay with the 2.6.9-67.0.20, find the sources for the updated cciss module at /opt/hp/storage_drivers (if any version of the cpq_cciss RPM is installed) and try and compile it yourself.

Whatever you choose to do, first run "chroot /mnt/sysimage" in rescue mode, so that you can use the complete set of system commands, and the package management tools will work without extra tricks.

If you choose to upgrade, first install the higher version kernel (or kernel-smp) RPM, then the appropriate cpq_cciss RPM (if necessary). The installation of the cpq_cciss RPM will automatically create a new initrd for you, after replacing the standard cciss module with the updated version.

Unfortunately I don't have a RHEL 4 test VM at the moment, so I cannot check how hard would it be to compile the updated cciss module manually.

[Afaq]

Thank you so much MK.

let me check wheather we can upgrade or not.

in rescue mode "chroot /mnt/sysimage" was not working its giving " cannot run command '/bin/sh': input/output error".

now i'm in rescue mode i can able to see all the mount point and files. Is it possible to take the backup of my files ?

[Matti_Kurkela]

Apparently there is some corruption in your root filesystem after all.

You can use the rescue mode without entering the "chroot /mnt/sysimage" command too, but then you will have a reduced set of commands available, and all pathnames of your files will have /mnt/sysimage prefixed to them. If that's enough to back up your files, then do it. Otherwise you'll need to find the problem that stops you from running the standard shell in chroot mode, and fix it.

I think "rpm --root /mnt/sysimage -Va" should allow you to verify all the files managed by the RPM package manager on your system. The command should list all the files that have been modified after they have been installed from RPMs. For some files (e.g. configuration files and log files), this is expected. But if you see something like /bin/bash or the system libraries in /lib have changed, you might want to reinstall the respective RPM.

It should be possible to reinstall RPMs in rescue mode without the chroot command, using commands like "rpm --root /mnt/sysimage --replacepkgs -ivh <pathname of the RPM>". Once the chroot command starts working, many things - including backing up your files - will be easier.

Alternatively, you can find some Live-CD-type Linux distribution, and boot the system from it.

(For example, Knoppix used to be good for rescue purposes: in the past, I've used it a few times for purposes like this.)

You may have to activate the VG and mount the filesystems manually, but otherwise the Live CD environment might be more user-friendly and have more tools available than the RHEL 4 rescue mode.