System Crash - continual reboot

[Joe Angeline_1]

Hi everyone. I've got a problem with a 9000. The following message is given -

Stored message buffer up to panic:
gate64: sysvec_vaddr = 0xc0002000 for 2 pages
NOTICE: autofs_link(): File system was registered at index 3.
NOTICE: cachefs_link(): File system was registered at index 5.
NOTICE: nfs3_link(): File system was registered at index 6.
0 sba
0/0 lba
0/0/0/0 btlan
0/0/1/0 c720
0/0/1/0.7 tgt
0/0/1/0.7.0 sctl
0/0/1/1 c720
0/0/1/1.2 tgt
0/0/1/1.2.0 sdisk
0/0/1/1.7 tgt
0/0/1/1.7.0 sctl
0/0/2/0 c720
0/0/2/0.2 tgt
0/0/2/0.2.0 sdisk
0/0/2/0.7 tgt
0/0/2/0.7.0 sctl
0/0/2/1 c720
0/0/2/1.2 tgt
0/0/2/1.2.0 sdisk
0/0/2/1.7 tgt
0/0/2/1.7.0 sctl
0/0/4/0 func0
0/0/4/1 asio0
0/1 lba
0/2 lba
0/3 lba
0/4 lba
0/5 lba
0/8 lba
0/8/0/0 btlan
0/9 lba
0/10 lba
0/10/0/0 td
td: claimed Tachyon TL/TS Fibre Channel Mass Storage card at 0/10/0/0
0/10/0/0.1 fcp
0/10/0/0.1.18.255.0 fcpdev
I/O hardware probe timed out at path 0/10/0/0.1.18.255.0.0

System Panic:

linkstamp: Fri Jun 14 03:58:54 EDT 2002
_release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
panic: I/O system hang

PC-Offset Stack Trace (read across, top of stack is 1st):
0x00209c50 0x0037e760 0x0073fffc
0x0003f990 0x0025e390 0x00165cdc
0x003f1be8 0x00176f3c 0x001754dc
End Of Stack
I know how to stop the boot process but am not sure how to begin to troubleshoot this.

Any infomation would be great.

Thanks,

Joe

[Patrick Wallek · Accepted solution]

It appears that the system was doing a probe of your fibre attached devices and timed out on the path 0/10/0/0.1.18.255.0.0. I'd start by checking that path, the devices, cables, controllers,etc. You could also try unplugging the cable from the card on the 0/10 patch and see what happens.

If the system boots without panicing, then you've narrowed down the problem. You can then narrow further by replacing cables, checking devices, etc.

Good luck!

[James Murtagh]

Hi Joe,

If I was you I would take the cable out of the FC card, terminate it, boot to single user mode and disable AUTO_VG_ACTIVATE in /etc/lvmrc, by setting this to 0, then reboot.

There is obviously a problem with a device on your SAN, although it appears your switch is OK, it is what is hanging off it that is causing you problems. Once the server is up you can replug the FC cable and troubleshoot from there, it shouldn't panic the server.

I'd imagine you will need to install the ioscan performance improvement patches, but I would need to know:

1) What OS? I know its 64bit but not the version.
2) What class of server?
3) swlist -l product|grep -i ioscan

Regards,

James.

[Stuart Abramson_2]

Well, I think you have a hardware problem, and should call HP immediately.

But, bring it up into single user mode, (the steps depend on what model this is, it looks like an L or N), and check the following logs:

/var/tombstones/ts99 (or latest)
look for HPMC in tombstone
/etc/shutdownlog
look for info about panic

I forget how to go to single user mode on an N, but you do something like this:

get console
reboot machine
interrupt boot sequence at warning
ISL
bo -is

[Joe Angeline_1]

Hi everyone,

Thanks for your help so far. I have two fc cards going to two brocades. I have a twin machine doing the same, and two VAs. So, I termed each of the two cards and booted - I got the same panic when either card was used, going to eiter switch. When I termed both cards, the box booted fine. I assume that since the same conditions exist for both cards, on either switch, its not a card or cable problem. The other "twin" is working fine and is using both switches so I'm totally confused. Any ideas?

Thanks,

Joe

[Steven E. Protter]

Several choices:

1) Boot off previous kernel

10 second prompt
bo
Y interact
hpux /stand/vmunix.prev

2) Boot of install media, run a recovery shell or just load the os so you can look around.

3) single user boot

10 second prompt
bo
Y interact
hpux -is

4) If its an older box, disconnect the fibre cable and see if it boots.

Somehow you have a bad kernel. Lots of stuff could have caused it.

What is the last thing you did to this machine. Hopefully that was not reinstalling PHKL_18543. That would e bad.

SEP

[melvyn burnard]

Can you boot the system to multi user mode with the fibre cables disconnected?
If so, what happens if you then plug in the cables?
What does an ioscan reveal?
Did this used to work or has it never worked?


You may want to check the version of the td driver using swlist

One document to look for in the ITRC is:
KBRC00009415 I/O hardware probe timed out -or- I/O system hang panic

Also what OS version, and what 9000 model?
and finally what FC switches are you using, and what is the firmware level of those switches?

[Colin Topliss]

This might help you a bit.

I came across the same sort of problem when someone decided to put in a fibre hub on the back of our EMC array (which in itself acts like a hub anyway). We were getting duplicate ALPA addresses, and this in turn caused the system to panic continually during the system hardware probe. Like you, we could resolve the issue by disconnecting the affected fibre channel (the hub was only placed on one channel of the array). Once the hub was removed, all was OK again.

So, if anything has been changed in respect to array configuration/cabling, then check that.

Hope it helps.

[Joel Shank]

I had a similar problem. It turned out that I had two devices with the same SCSI ID.

If you added new devices, I'd check them for this.

Hope this helps,
jls

[Joe Angeline_1]

Hi everyone,

To answer Melvin's questions -
Can you boot the system to multi user mode with the fibre cables disconnected?

yes

If so, what happens if you then plug in the cables?

Nothing

What does an ioscan reveal?

It hangs

Did this used to work or has it never worked?

It has been in production for over a year.

Thanks again for everyone taking the time to help.

Joe