Unable to Boot HP-UX 10.20

[Sri Narayanan]

Hi,

We have a HP-UX 10.20 box that is used only for Q&A testing. Resently there was a power failure to this box (not sure if this is the cause for the problem) and I'm not able to Boot the system. This is what I see.

Boot
:disc (4.5.0;0) /stand/vmunix
3614452+319488+426048 start 0x1b4c68

(Then scrolls thru a bunch of stuff and comes back and displays the following:)

System Panic (Display == 0xbf00, flags ==....
...
Unable to determine the cause for the System crash by the Kernel

FLTBF00

Warning: Core Dump
Warning: Dumpsys( ) 42949667295

(Then reboots on its own......and does this over and over again)

Please help. Thank you so much for your time.

Thanks again,
Sr

[harry d brown jr]

Boot into single user mode and run an fsck.

live free or die
harry

[Ian Dennison_1]

FLTBF00

FLT = 'Fault'
BF00 = Fault Code

It sounds like you need an engineer onsite!

Have you thought about trying to boot off another form of media, to see if it is the boot disk?

Share and enjoy! Ian

[Sri Narayanan]

Does the same thing when I boot in single user mode. Actually the messages are from booting in single user mode.

Thanks,
Sri

[Sridhar Bhaskarla]

If you carefully go through the messages you may find some clues like scsi errors etc.,

This could be due to a lot of reasons. What is the model of this box?. If it is crashing after the LED state CEF2, it could be mostly due to SCSI hang or any hardware failure. Make sure the SCSI ids and the connections are proper and there are no open terminations.

Or open a hardware call with HP and they should be able to determine based on the console messages and status codes.

-Sri

[Sri Narayanan]

It is a HP9000 700/800 D class box. Yes I checked the SCSI ids and their connections. they seem to be okay. Could there be any other problem? and why would it suddenly stop working?

Thanks so much,
Sri

[Sridhar Bhaskarla]

Hi Sri,

I suspect it could be due to a hardware failure as you mentioned that there was a power failure. You may want to open a Hardware call with HP and go through the console messages and chassis codes to find out the reason.

-Sri

[Roger Baptiste]

hi,

what model of system is it? Based on that you can look up the meaning of Fault codes.

If you have a support , boot from the support cd and see whether you can mount the root disk.

If your root disk is mirrored, try booting from the mirror disk.

HTh
raj

[Volker Borowski]

Hi,

maybe someone junked your kernel ?

Did you try to boot /stand/vmunix.prev ?

Do not know if this helps,
just an option.
Indeed I also suspect hardware trouble.

Second option:
boot into LVM maintanance-mode

Good hunting
Volker

[Sri Narayanan]

Thanks for all your responses. Well I was on maternity leave (baby boy) until last week. so could be someone did something while i was gone. will find out about that. all i know is last monday we had a power shutdown for a few minutes and i didnt check this server then as i was busy fixing our production servers.

btw how do i boot using /stand/vmunix.prev? and
boot into LVM maintenance mode?

could you please give me the steps. I'm a rookie in Unix system admin. I'm basically a Oracle DBA taking care of this unix box where the DB resides.

Thanks again for all your great help.
Sri

[Sridhar Bhaskarla]

Hi Sri,

I am not sure if it is due to kernel problem. If somebody built the new kernel, there is a possibility that they might have booted the system later and it was up and running without a problem.

However, there is a chance that they rebuilt the kernel but didn't boot the system. To boot the system with vmunix.prev, you need to interact with ISL. Get into bootadmin prompt. While it boots, it will ask you to press any key to interact. Once you are at the boot admin prompt. Do a boot pri isl. At the isl prompt do

isl> hpux /stand/vmunix.prev

This should boot the system with the previous kernel.

If this also doesn't work, then you need to call HP. It's mostly a hardware failure.

-Sri

[Steve Labar]

It sounds like you lost either your boot disk or a primary swap device in your power loss. To boot to a different kernel follow the same steps as you would to enter single user mode. Interrupt your boot sequence by pressing any key when prompted. Once you enter the boot admin mode, you can do a search and verify the devices you expect to see are listed. If you don't see the devices you expect, you need to clear those problems up first. If all the devices are present you can do a boot pri. When it asks if you want to interact with IPL? enter a 'y'. At the ISL> prompt enter HPUX /stand/vmunix.prev to boot from a previous kernel or HPUX -lm /stand/vmunix to boot into lvm maintenance mode. I have seen panics during boot if a primary swap space is built into the kernel, but the device is not available during boot. If this device is missing the only way to boot would be from a kernel that doesn't have it built in.
Good Luck.

Stev

[Sanjay_6]

Hi Sri,

First let us congratulate you on the baby boy.

Next to boot the system in single user mode, try this,

interrupt the boot procedure when it says press any key to interrupt the boot procedure,

At the prompt type "bo pri" to boot through the primary disk. Select "Y" when it says do you want to interact with IPL. The system will boot and it will come to the IPL prompt. At the ipl prompt type "hpux -is" to boot the system in single user mode.

IPL> hpux -is

To boot the system using the default kernel

IPL> hpux (;0)/stand/vmunix

To boot the system using alternate kernel

IPL> hpux (;0)/stand/vmunix.prev

Here vmunix.prev is an alternate kernel. To locate bootable kernels in /stand try

IPL> hpux ll (;0)/stand

If there is a disk problem in the root VG, you can try and boot without quorum,

IPL> hpux -lq (;0)/stand/vmunix

Also take a look at the thread below for more guidance,

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2355-90742/B2355-90742_top.html&con=/hpux/onlinedocs/B2355-90742/00/00/41-con.html&toc=/hpux/onlinedocs/B2355-90742/00/00/41-toc.html&searchterms=system%20boot&queryid=20011127-094820

Hope this helps.

Regds

[Sri Narayanan]

Thanks Sanjay.

Well i was doing lots of trial & error. And guess what I was able to reboot when I disconnected the HP Storage (with all the SCSI disks) from the Box. Now I've to figure why they dont work. There is no discrepancy in the SCSI ids. When I did a search on the Boot Prompt, I was able to see all of the SCSI disks there.

Please help if you have any suggestions.

Thanks again for all your responses.
Sri

[Sanjay_6]

Hi Sri,

Try to boot your system in single user mode with all your disks connected. If you are able to do so, try mounting the filesystems manually. Do remember that if you want to manually mount filesystems for VG other than VG00 (root vg), you need to do a vgchange -a y /dev/vg_name. Don't have to mount all the filesytems. Just boot into single user mode, mount the filesystems for the root VG and then just activate all the other VG's, deactivate them and then reboot and try booting into multiuser mode. Give a more detailed description of the error you are getting. The earlier error was not of much help. Also make sure you have posted the right error code diskplayed on the control panel of your system.

Hope this helps.

Regds

[Sri Narayanan]

Hi Sanjay,

I was not able to boot the system in single user mode when connected to the all the disks. This is what i did.

Break when booting to go to the Boot Prompt
boot pri
IPL - Y (yes)
hpux -is

(The following message is displayed)

Boot
:disc (4.5.0;0) /stand/vmunix
3614452+319488+426048 start 0x1b4c68
(Hangs here for a long time maybe about 15-20 mins, Then scrolls thru a bunch of numbers)

(Finally comes down to this)
NCR Chip register dump
...
Zalon register dump
IO_STATUS:
f90008c0
IO_EIM
00000000

SCSI Resetting SCSI - lbolt: 21509, bus:0
SCSI Reset detected - lbolt:21509, bus:0

System Panic:
9245xB HP-UX (B 10.20) #1: Sun Jun 9 06:31:19 PDT 1996
panic: (display==0xbf00, flags==0x0) I/O system hang

PC-Offset Stack Trace (Read accross, most recent is 1st)
...

It was not possible for the kernel to find the process that cause this crash.
Dumpsys ( ) called

Warning: Early dump not configured; ** Contact your HP support representative**
Warning: Cant initialize dump device

Dumpsys ( ) failed, returned 4294967245

(From here it reboots on its own)

This goes on & on until I turn the switch off.

Please let me know if you need more information. Also when I disconnect the HP Storage completly from the server (ie. remove the connecting cable & close the SCSI chain) it boots okay in single user mode. It just give bunch of warning messages that not able to locate the other VG's and lvol's.
Warning: Insufficient space on dump device.
Warning: Time-of-day register time is out of range.

When i try to shut down, it gave me the following error. (When booted in single user mode without the SCSI disks)

Trasition to run-level 0 is complete
Wait for Transition to run-level s (allow 20 seconds)
Wait for Transition to init state timed out - exiting shutdown.

(I manually turned the switch off on the box)

Thanks so much for your help,
Sri

[Sridhar Bhaskarla]

That goes to my first reply of checking the SCSI chain. You need to make sure that the SCSI ids of the extenal array do not conflict with the internal disks. Also make sure you have the terminations proper on the storage.

-Sri

[Sri Narayanan]

Hi Sri,

I checked them, there are no conflicts. They all have different ids.

They are as follows:
4.5.0 (vg00)
4.8.0 (mirror root disk)

from the external HP storage unit:
4.0.0
4.1.0
4.2.0
4.4.0
4.9.0
4.10.0

And the terminator is in place correctly. so there is no broken SCSI chain.

Thanks,
Sri

[Sanjay_6]

Hi Sri,

lbolt errors are primarily due to SCSI bus problems. The bus 0 in the error message refers to the c0 on your system. If your Internal and external disks are on the same bud, check the SCSI ID's. From your feedback it looks the SCSI ID's are okay. If so, is the mirror on external disk. In case the mirror is on external disk, can you disconnect that disk and try to boot. To boot without mirror you should use "hpux -lq (;0)/stand/vmunix" at the IPL prompt. You can also check and see if termination exist on the controller if your internal and external disks are on the same controller.

when you do a boot of the system without external disks, run a "ioscan -fn" to make note of the SCSI ID of your SCSI controller. This will also help you in isolating the clash. If the disks you are using are Hot-swap, remove one disk at a time and try the boot. If there is a clash of SCSI id's, you may be able to isolate that by doing this.

Hope this helps.

Regds

[Scott_14]

Hello:

I have had somthing like this in the past, were, I had to power down the system a few times, and it came up, but what I found was the main boot disk, actually was going bad, and I had booted off the mirror, to get the system up, and called in service to get that drive replaced.

At first the disk didnt seem like it was bad, but I beleive it was failing, and it finally
failed where it wouldnt come up.

Good luck.

scott

[Sri Narayanan]

Thanks to all of you who responded to my query. Well this is what i was able to figure out. I tried booting without the HP storage unit and it worked. Then according to Sanjay's suggestion, I tried to boot in single user mode with one disk at a time. Well looks like there is a problem with one side of the External HP storage. Because all the disks work fine with the same SCSI ids in the right side (side B) but none of them work when put on the left side (side A). The system just hangs while booting for about 15-20 minutes and comes back with the same error as before. well off i go to figure out if i can find anything more than this. will let you know how that goes.

meanwhile if anyone has any more suggestion in trouble shooting please let me know. btw can one side of the HP storage unit go bad? i would think maybe there is a problem with the cross-over SCSI cable that links side A & side B. maybe it got busted during power shutdown.

Thanks again for all your help, Made it very very easy for me to figure out what is wrong.
Sri

[Sanjay_6]

Hi Sri,

You can check the port and see if the terminator or cable is getting plugged in properly. Sometimes while plugging the cable in and out, a pin gets bent and that damages the holes on the port. You can check and make sure something like that is not the case over here.

Hope this helps.

Regds

[Sri Narayanan]

IT WORKED!!! I was able to Boot the system!!!

I un-hooked the SCSI cables and hooked them again that didnt work. Then I put a new set of SCSI Cross-over cable to go from side A to side B which did the trick. I guess the cross over cable went bad during power shutdown.

Without all your help i would not have been able to figure this out. I would like to thank all of you for helping me get back on the track.

Just one more question,

When i try to shut down, in single user mode. The following message displayed. and I had to manually shut the machine by switching the off key on the box.

Trasition to run-level 0 is complete
Wait for Transition to run-level s (allow 20 seconds)
Wait for Transition to init state timed out - exiting shutdown.
#

Thanks again for helping me get the system up & running.
Sri

[Sanjay_6]

Hi Sri,

So your system finally worked. Thanks for posting your results.

To shutdown from multiuser mode to power off, use

shutdown -h now
or
shutdown -h 0

To shutdown from Runlevel 3 to single user mode, use

init 2;exit
then type

shutdown
or
shutdown 0

To shutdown from single user mode to power off, use

shutdown -h now
or
shutdown -h 0

Hope this helps.

Regds

[Sri Narayanan]

Hi Sanjay,

Thanks for a quick response. tried what you said from single user mode, but gives me the same error (i dont know if this is a error message, also does it automatically power off the HP box or should i manually switch it off after this.)

Wait for Transition to run-level s (allow 20 seconds)
Wait for Transition to init state timed out - exiting shutdown.
#

Thanks a bunch for all your responses,
Sri