topic Re: shutdown problem for rx3600 in Operating System - HP-UX

shutdown problem for rx3600

donna hofmeister — Mon, 15 Oct 2012 19:42:19 GMT

i've got a rx3600 (11.31, of course) that always crashes on shutdown -r.

when i was first made aware of the problem, the system would actually get into a crash-fail-to-dump loop that was only resolved by powercycling the system. the powercycle always resolved the issue (albeit unelegantly) and the system would come up as expected. we manually booted (hpux> boot /stand/current/vmunix) over the weekend thinking the system was confused about what kernel to boot from. that only marginally helped since the system still crashes on shutdown but at least now it produces a crashdump (see attached).

to me, this crashdump looks like there's a hardware problem -- but nothing is raising a red flag.

i noticed the (dire :-) warning about USB driver in the crashinfo file. could that be it? the driver is old (C.01.05.08). the HAA boot device is USB:

Primary bootpath : 0/6/0/0/0/0/1/0/0/0.0x0.0x4000000000000000 (/dev/rdisk/disk3)
HA Alternate bootpath : 0/0/2/1.0x0.0x10 (/dev/deviceFileSystem/Usb/MassStorage/dsk/disk@hp-1008+294=A60020000001)

the system has been recently patched with the latest support bundle, fwiw.

any ideas?

Re: shutdown problem for rx3600

Bill Hassell — Tue, 16 Oct 2012 00:01:06 GMT

Defintely get that bad USB driver out of there. And unless you have a goiod reason for the USB path in HAA path, use setboot to set HAA to real boot path.

Also, latest patch bundle? That would be Sep 2012 for QPK, Feature and HWE. I don't see recent ARPA/TCP cumulative patch in the crashinfo list (PHNE_41436) but it may not be reported.

The double panic may indeed point to a hardware error. Connect to the (real) console and get a list of the last 10-20 entries in the SL (hint: SL, then E for error logs, and F for forward progress). That may provide a better clue as to what is going on.

Re: shutdown problem for rx3600

donna hofmeister — Tue, 16 Oct 2012 16:11:34 GMT

the both system logs look clean.

i'm arranging for the latest USB driver to be installed and we'll see if that brings us any joy.

about setboot...i've notes that say do "setboot -h <primary boot disk>" but is this is for pa-risc boxes only? does it work for itaniums as well?

about the patches....i spent an entire minute (ha!) looking at the list of patches in the latest bundle and it sure looks to me that the latest arpa patch (for example and since we're using it as a guinea pig) is NOT included. i suspect i missed something...but....

Re: shutdown problem for rx3600

Bill Hassell — Tue, 16 Oct 2012 23:49:19 GMT

setboot -h <legacy hardware path, lunpath hardware path or a persistent dsf>

No difference between PA and IA as far as setboot is concerned. It is just setting stable storage on the system board.

As far as PHNE_41436, oops -- that should have been PHNE_41617, 41714 or 42017 for 11.31 (41436 is for 11.23). The ARPA cumulative patch is full of fixes for panics and other networking issues. But you have the second most recent so you should be OK. There are warnings for 41714 as well as its successor. For 41714 (which you have), it is only a problem when you have HP-UX Transport SRP installed. Check with:

swlist -l bundle HPUXTransportSRP

If nothing is found, then 41714 is OK.

Your best bet is to look at the SL logs.

Re: shutdown problem for rx3600

donna hofmeister — Wed, 24 Oct 2012 14:48:17 GMT

an update...

updating the usb driver has helped....somewhat. at least now the box doesn't crash while shutting down.

it still won't reboot on it's own without a powercycle. (don't ask for console log...it didn't get captured this time around...)

has anyone heard of /sbin/reboot actually failing? i was wondering if we did:

1) shutdown (with no parms) and then

2) reboot

...would that be safe and helpful?

Re: shutdown problem for rx3600

Dennis Handly — Wed, 24 Oct 2012 20:03:36 GMT

>has anyone heard of /sbin/reboot actually failing?

What part is failing? The shutting down or the booting?

Re: shutdown problem for rx3600

donna hofmeister — Wed, 24 Oct 2012 20:07:41 GMT

shutdown (imo)

it reboots fine -- provided you powercycle the box.

if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

you can see in previous console logs where it is (for example) closing lvm volumes, so i don't believe this is a problem with some rc script.

Re: shutdown problem for rx3600

Dennis Handly — Wed, 24 Oct 2012 20:17:19 GMT

>if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

Where does it stop? Anything in the shutdown log?

>you can see in previous console logs

I don't see those?

Have you tried disconnecting that USB disk?

Re: shutdown problem for rx3600

donna hofmeister — Wed, 24 Oct 2012 20:23:39 GMT

the shutdownlog isn't helpful...just the normal one-liner.

anyhow, in this last go-round (where the usb driver was updated) we didn't get the shutdown dialog because the connection was lost.

the previous console/shutdown logs are a horror to look at and probably not helpful at this point.

afaik, the usb dvd drive is still connected. but between updating the driver and modifying setboot, i don't think it's an issue any longer. if you think otherwise, let me know.

i'm hoping for (yet another) reboot over the weekend....

Re: shutdown problem for rx3600

Dennis Handly — Wed, 24 Oct 2012 22:05:22 GMT

>the shutdownlog isn't helpful... just the normal one-liner.

That means it got from shutdown(1m) to reboot(1m), so your suggestion probably wouldn't work?

>if you think otherwise, let me know.

Only if you have nothing better to do. ;-)

Re: shutdown problem for rx3600

Matti_Kurkela — Fri, 26 Oct 2012 14:39:46 GMT

Look into /etc/rc.log.OLD: if something went wrong in the shutdown, this file is where the error messages would be recorded.

(Originally it would have been /etc/rc.log, but renamed to /etc/rc.log.OLD in the next system startup.)

Re: shutdown problem for rx3600

donna hofmeister — Mon, 12 Nov 2012 21:41:06 GMT

ok...with the dvd drive pulled, 'shutdown -r' is still resulting in the box panicing upon shutdown completion.

the console listing is attached. this still feels like something hardware related...

Re: shutdown problem for rx3600

Dennis Handly — Fri, 16 Nov 2012 20:00:39 GMT

>the console listing is attached.

These errors indicate you have a garbage file left in /etc/rc.config.d/:

/sbin/rc[82]: 1500: not found.

...

/sbin/rc1.d/K410Rpcd[30]: 1500: not found.

I don't think it is related to your crash (unless you incorrectly fiddled with a config file) but you should fix it.

>Closing open logical volumes... Done

>panic : Failed to halt all processors

...

>Closing open logical volumes... Done

>Bad News: pr == 0x144000406c61001d

It seems that this is where it goes wrong. During the shutdown.

Re: shutdown problem for rx3600

donna hofmeister — Mon, 12 Nov 2012 22:39:55 GMT

yeah...i noticed the rc.config.d messages. i, too, don't think they're related.

yeah #2 -- that's where it all goes wrong. perhaps some kind hp-insider could plug some of these messages/number into their call tracker and say <this> is the problem? :-)

Re: shutdown problem for rx3600

Dennis Handly — Tue, 13 Nov 2012 03:23:35 GMT

>I noticed the rc.config.d messages.

But have you fixed it yet? :-)

>perhaps some kind hp-insider could plug some of these messages/number into their call tracker ...

That's what the HPSC is for.

Re: shutdown problem for rx3600

Todd McDaniel_1 — Fri, 16 Nov 2012 18:51:09 GMT

Donna, I don't see that you have reviewed the shutdown log. any data in there?

also, please bring the box down into single user mode first, then note any errors, and reboot from there. I will bet you a donut it will come up clean.

Re: shutdown problem for rx3600

donna hofmeister — Fri, 16 Nov 2012 18:54:46 GMT

there's nothing (interesting) in the shutdown log. i think we (collectively) have decided that the machine is actually shuting down ok.

it's when the machine goes to restart (we're at the hardware level now) that it has problems.

if you are correct, how are you going to collect your donut? :-)

Re: shutdown problem for rx3600

Todd McDaniel_1 — Fri, 16 Nov 2012 19:13:46 GMT

virtual donut!

I'd suggest going down to single in steps and watch the console to see anything irregular.

I'm pretty sure something is hanging when it goes down, I'd almost bet its not umounting all the disks or other hardware device for some reason and actually hanging at the very bottom of the process, before it attempts to come up on reboot.

do you have a large amount of san attached disks? or nfs mounts on this server?

Regards,

Todd McDaniel

Re: shutdown problem for rx3600

Dennis Handly — Fri, 16 Nov 2012 20:05:52 GMT

>I think we (collectively) have decided that the machine is actually shutting down ok.

I don't think so. It is panicking when you are trying to shut it down: ux06.console.listing.txt

panic : Failed to halt all processors

Seems like a hardware problem.

>I don't see that you have reviewed the shutdown log.

Did you mean the rc.log? There were problems with it. Possibly unrelated?

Re: shutdown problem for rx3600

Todd McDaniel_1 — Fri, 16 Nov 2012 20:49:46 GMT

yea rc.logs, iirc the shutdown log only shows when the box went down.

you might also inspect your /dev/ (devices) directory for any unusal entries. Make sure you can account for every device, hopefully not too many there!

grasping for straws here, but if possible, recreate your /dev/random and /dev/zero devices

Review swlist for any unnecessary products you can remove.

Random 3rd party software not managed by your depot...