shutdown problem for rx3600

[donna hofmeister]

i've got a rx3600 (11.31, of course) that always crashes on shutdown -r.  

when i was first made aware of the problem, the system would actually get into a crash-fail-to-dump loop that was only resolved by powercycling the system.  the powercycle always resolved the issue (albeit unelegantly) and the system would come up as expected.  we manually booted (hpux> boot /stand/current/vmunix) over the weekend thinking the system was confused about what kernel to boot from.  that only marginally helped since the system still crashes on shutdown but at least now it produces a crashdump (see attached).

to me, this crashdump looks like there's a hardware problem -- but nothing is raising a red flag.

i noticed the (dire :-) warning about USB driver in the crashinfo file.  could that be it?  the driver is old (C.01.05.08).  the HAA boot device is USB:

Primary bootpath : 0/6/0/0/0/0/1/0/0/0.0x0.0x4000000000000000 (/dev/rdisk/disk3)
HA Alternate bootpath : 0/0/2/1.0x0.0x10 (/dev/deviceFileSystem/Usb/MassStorage/dsk/disk@hp-1008+294=A60020000001)

the system has been recently patched with the latest support bundle, fwiw.

any ideas?

[Bill Hassell]

Defintely get that bad USB driver out of there. And unless you have a goiod reason for the USB path in HAA path, use setboot to set HAA to real boot path.

Also, latest patch bundle? That would be Sep 2012 for QPK, Feature and HWE. I don't see recent ARPA/TCP cumulative patch in the crashinfo list (PHNE_41436) but it may not be reported.

The double panic may indeed point to a hardware error. Connect to the (real) console and get a list of the last 10-20 entries in the SL (hint: SL, then E for error logs, and F for forward progress).  That may provide a better clue as to what is going on.

[donna hofmeister]

the both system logs look clean.

i'm arranging for the latest USB driver to be installed and we'll see if that brings us any joy.

about setboot...i've notes that say do "setboot -h <primary boot disk>" but is this is for pa-risc boxes only?  does it work for itaniums as well?

about the patches....i spent an entire minute (ha!) looking at the list of patches in the latest bundle and it sure looks to me that the latest arpa patch (for example and since we're using it as a guinea pig) is NOT included.  i suspect i missed something...but....

[Bill Hassell]

setboot -h <legacy hardware path, lunpath hardware path or a persistent dsf>

No difference between PA and IA as far as setboot is concerned. It is just setting stable storage on the system board.

As far as PHNE_41436, oops -- that should have been PHNE_41617, 41714 or 42017 for 11.31 (41436 is for 11.23). The ARPA cumulative patch is full of fixes for panics and other networking issues. But you have the second most recent so you should be OK. There are warnings for 41714 as well as its successor. For 41714 (which you have), it is only a problem when you have HP-UX Transport SRP installed. Check with:

    swlist -l bundle HPUXTransportSRP

If nothing is found, then 41714 is OK.

Your best bet is to look at the SL logs.

[donna hofmeister]

an update...

updating the usb driver has helped....somewhat.  at least now the box doesn't crash while shutting down.

it still won't reboot on it's own without a powercycle.  (don't ask for console log...it didn't get captured this time around...)

has anyone heard of /sbin/reboot actually failing?  i was wondering if we did:

1) shutdown (with no parms) and then

2) reboot

...would that be safe and helpful?

[Dennis Handly]

>has anyone heard of /sbin/reboot actually failing?

What part is failing? The shutting down or the booting?

[donna hofmeister]

shutdown (imo)

it reboots fine -- provided you powercycle the box.

if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

you can see in previous console logs where it is (for example) closing lvm volumes, so i don't believe this is a problem with some rc script.  

[Dennis Handly]

>if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

Where does it stop?  Anything in the shutdown log?

>you can see in previous console logs

I don't see those?

Have you tried disconnecting that USB disk?

[donna hofmeister]

the shutdownlog isn't helpful...just the normal one-liner.

anyhow, in this last go-round (where the usb driver was updated) we didn't get the shutdown dialog because the connection was lost.

the previous console/shutdown logs are a horror to look at and probably not helpful at this point.

afaik, the usb dvd drive is still connected.  but between updating the driver and modifying setboot, i don't think it's an issue any longer.  if you think otherwise, let me know.

i'm hoping for (yet another) reboot over the weekend....

[Dennis Handly]

>the shutdownlog isn't helpful... just the normal one-liner.

That means it got from shutdown(1m) to reboot(1m), so your suggestion probably wouldn't work?

>if you think otherwise, let me know.

Only if you have nothing better to do.  ;-)

[Matti_Kurkela]

Look into /etc/rc.log.OLD: if something went wrong in the shutdown, this file is where the error messages would be recorded.

(Originally it would have been /etc/rc.log, but renamed to /etc/rc.log.OLD in the next system startup.)

[donna hofmeister]

ok...with the dvd drive pulled, 'shutdown -r' is still resulting in the box panicing upon shutdown completion.

the console listing is attached.  this still feels like something hardware related...

[Dennis Handly]

>the console listing is attached.

These errors indicate you have a garbage file left in /etc/rc.config.d/:

/sbin/rc[82]: 1500: not found.

...

/sbin/rc1.d/K410Rpcd[30]: 1500: not found.

I don't think it is related to your crash (unless you incorrectly fiddled with a config file) but you should fix it.

>Closing open logical volumes... Done

>panic : Failed to halt all processors

...

>Closing open logical volumes... Done

>Bad News: pr == 0x144000406c61001d

It seems that this is where it goes wrong.  During the shutdown.

[donna hofmeister]

yeah...i noticed the rc.config.d messages.  i, too, don't think they're related.

yeah #2 -- that's where it all goes wrong.  perhaps some kind hp-insider could plug some of these messages/number into their call tracker and say <this> is the problem?  :-)

[Dennis Handly]

>I noticed the rc.config.d messages.

But have you fixed it yet?  :-)

>perhaps some kind hp-insider could plug some of these messages/number into their call tracker ...

That's what the HPSC is for.

[Todd McDaniel_1]

Donna, I don't see that you have reviewed the shutdown log. any data in there?

also, please bring the box down into single user mode first, then note any errors, and reboot from there. I will bet you a donut it will come up clean.

[donna hofmeister]

there's nothing (interesting) in the shutdown log.  i think we (collectively) have decided that the machine is actually shuting down ok.

it's when the machine goes to restart (we're at the hardware level now) that it has problems.

if you are correct, how are you going to collect your donut?  :-)

[Todd McDaniel_1]

virtual donut!

I'd suggest going down to single in steps and watch the console to see anything irregular.

I'm pretty sure something is hanging when it goes down, I'd almost bet its not umounting all the disks or other hardware device for some reason and actually hanging at the very bottom of the process, before it attempts to come up on reboot.

do you have a large amount of san attached disks? or nfs mounts on this server?

Regards,

Todd McDaniel

[Dennis Handly]

>I think we (collectively) have decided that the machine is actually shutting down ok.

I don't think so.  It is panicking when you are trying to shut it down:  ux06.console.listing.txt

panic : Failed to halt all processors

Seems like a hardware problem.

>I don't see that you have reviewed the shutdown log.

Did you mean the rc.log?  There were problems with it.  Possibly unrelated?

[Todd McDaniel_1]

yea rc.logs, iirc the shutdown log only shows when the box went down.

you might also inspect your /dev/ (devices) directory for any unusal entries. Make sure you can account for every device, hopefully not too many there!

grasping for straws here, but if possible, recreate your /dev/random and /dev/zero devices

Review swlist for any unnecessary products you can remove.

Random 3rd party software not managed by your depot...

[donna hofmeister · Accepted solution]

just to *wrap* this issue up (ho ho ho)...

turned out the MP board was faulty.    it took a fair amount of digging on the CE's part to diagnose the issue...but all is well now.

[Torsten.]

Since the "MP-board" also known as the UCIO
(Unified Core-I/O) board has not only the MP/iLO but also BMC and other components, a failure could affect more than the MP only.


From your initial post I noticed you still use the OLD USB driver (for the internal DVD and vMedia) - you should update this to get USB 2.0 support.

[donna hofmeister]

the usb driver was was one of the first things we fixed.