Panic on boot after patch bundle install

[marnold_1]

Hi,

First: I'm not much of an HP-UX sysadmin, my background is mainly development, and then on Solaris.

I was asked to install Oracle 9iR2 on an old HP11.00 server. For this, it was necessary to install some patches - I installed the last patch bundle for HP11.00, QPK1100(B.11.00.64.4).

This included some patches to the kernel, so after installation was complete, the system rebooted.

Now the system panics as it boots when it gets to 'Configuring all unconfigured software filesets'. Output:

HP-UX Start-up in progress
__________________________

Configure system crash dumps .............................. OK
Mount file systems ........................................ OK
Update kernel and loadable modules ........................ N/A
Initialize loadable modules ............................... OK

Start Software Distributor agent daemon ................... OK
Configuring all unconfigured software filesets ............


Stored message buffer up to panic:
NOTICE: nfs3_link(): File system was registered at index 3.

System Panic:

linkstamp: Tue Aug 28 10:16:55 BST 2007
_release_version: @(#)B2352B/9245XB HP-UX (B.11.00) #1: Thu Nov 6 01:58:21 PS
T 1997
__kern_ci_revision: $Header: kern_sequence.c,v 1.6.106.512 97/11/05 18:01:46 mso
sa Exp $

panic: Break instruction trap

So I tried to boot to single-user mode, using 'hpux -is' at the ISL> prompt, and this appears to work, but when it gets to the # prompt, the system becomes completely unresponsive. I'm connected over the serial port to hyperterminal on windows.

Can anyone help me please? I think I could work out how to resolve this myself if I could only boot to single-user run-level using a terminal over the serial port.

[Pete Randall]

Try your previous kernel. At the ISL prompt, enter "hpux vmunix.prev".


Pete

[TY 007]

Hi,

Boot to LVM Maintenance Mode:
ISL> hpux -lm

# vgchange -a y /dev/vg00
# mountall

If the system still become completely unresponsive, possible bad sector(s), in your case, possible just happen to be at the swap area.

Thanks

[Matti_Kurkela · Accepted solution]

First try booting to the single-user mode using the previous kernel. At the ISL> prompt, use the following command:

hpux -is /stand/vmunix.prev

Note that when you do this, any commands that refer to the kernel name list (for example, dmesg) will not work correctly unless you tell them to look into vmunix.prev instead of the default vmunix (i.e. use "dmesg /stand/vmunix.prev" instead of just "dmesg").

With the single-user boot, the root filesystem is in read-only mode and other filesystems are not mounted at all. The easiest way to deal with this is to use the "mountall" command.

If using vmunix.prev allows you to bring the system up to single-user mode, your new kernel may have been corrupted while it was generated (as the patch bundle was installed). You can then replace the corrupted kernel with the "previous" kernel:

cd /stand
mv vmunix vmunix.failed
mv vmunix.prev vmunix
mv dlkm dlkm.vmunix.failed
mv dlkm.vmunix.prev dlkm
mv system system.failed
mv system.prev system

This removes the need to specify vmunix.prev when using commands like dmesg. Be aware that you're now running with a kernel that does not have the Quality Pack patches applied: you should rebuild your kernel to make the kernel patches of the Quality Pack take effect. If /stand is getting full, move the *.failed files and directories away from /stand (or just delete them, if you aren't going to analyze them any further).

Examine the state of the physical disks: check the dmesg output for SCSI errors. If the system disk is mirrored, run "vgdisplay -v vg00" and check that all logical volumes are in state "available/syncd".

You didn't mention the server model: knowing that would have helped to determine the general age of the server and allowed us to find out whether there are any special issues to be aware of.

MK

[Steven E. Protter]

Shalom,

Boot off the previous kernel.

Perform q4 analysis on the crash dump and submit the findings to HP.

swverify \* you may have corrupt patches and clues that could lead to resolution along with the swagentd logs and such.

SEP

[marnold_1]

Thanks to everyone who took time to reply.

I successfully booted off the previous kernel, (thanks Pete). Then moved the new vmunix, dlkm, system to *.failed and moved *.prev back (thanks MK).

I tried rebuilding a new kernel, but it failed with the same outcome.

But in the process of poking around trying to find the problem, I discovered, incredibly, that the kernel is 32-bit, and thus will never run Oracle (at least >=9i), which is 64-bit only. It's an old PA8000 CPU - which seems to be the first 64-bit PA-RISC chip.

I have now parked this problem with the recommendation that we upgrade our hardware, and then reinstall HP-UX completely.

Thanks again for your help - my first time using HP forums - better than most around the web.