topic Re: HPUX Server crash in Operating System - HP-UX

HPUX Server crash

Tobin Marchbanks — Mon, 21 Apr 2008 14:23:30 GMT

Here is what I see in the /etc/shutdownlog...

17:14 Fri Apr 18 2008. Reboot after panic: INIT, IIP:0xe000000000635350 IFA:0x9fffffffbf675000

Any thoughts on what I can do to figure out what these codes mean?

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 14:30:01 GMT

Anything in /var/adm/crash?
Whats the last line in /var/adm/syslog/OLDsyslog.log?

Any fault/attention lights on the front of the server?

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 14:32:26 GMT

Also may want to Connect to GSP and have a look at the chassis Logs type SL at GSP prompt OR Run stm (or xstm if you have an xwindows session) and look at the information on the memory. See if the memory error log is empty, or not.

Re: HPUX Server crash

Tobin Marchbanks — Mon, 21 Apr 2008 14:38:30 GMT

/var/adm/crash

@(\
rboundcrash.Â²crash.1mp.tarÂºcrashdump.tar.gzcrÂ¨Â¤crash.2

Last line in /var/adm/syslog/OLDsyslog.log

Apr 18 16:50:01 lawpr1v2 sshd[2780]: Accepted publickey for oracle from 10.101.60.156 port 61345 ssh2

Re: HPUX Server crash

melvyn burnard — Mon, 21 Apr 2008 14:50:32 GMT

It looks like you have had a panic on your Integrity server, you could make sure that you are patched up to the latest support pack, after that you should log a call with the HP Response Centre

Also check /etc/shutdownlog, and see if there is an INDEX file inteh crash subdirectory, that may give you a clue

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 14:54:20 GMT

Also, another place to look is the EMS event log, in /var/opt/resmon/log/event.log. Any serious events should also have been flagged in syslog, (some get missed)but the full information will be here.

Have you ran all of the normal stuff? like ioscan -funCdisk? Is the system reporting any disk failures or hardware issues? I know that HP-UX can reboot on "it's own" due to CPU failures and whatnot. What type of system are you running?

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 14:55:12 GMT

also, don't forget points! Cheers!

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 15:41:55 GMT

Is your system back up and running. or is it down still?

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 15:51:19 GMT

Did you unmirror any disks or remove veritas or anything right before the crash? You might be missing a patch or something.

Re: HPUX Server crash

Don Morris_1 — Mon, 21 Apr 2008 16:03:48 GMT

There's no codes there -- that's an Interrupt Instruction Pointer (what instruction bundle was being executed at the time of panic). That's tells you what code [in the kernel] was running. The Interrupt Fault Address is the data address referenced at the time of panic. For dereference panics, that's the address which caused the panic. For non-dereference panics [such as INIT usually], this is less meaningful.

I won't claim there may be some variant of this I haven't seen -- but usually INIT panics are Transfer-Of-Controls (TOC). Either someone pushed the TOC button on your IPF server, ServiceGuard or other clustering packages are running and decided to bring down the node because it was thought to be hung, or the GSP console was used to do CM>TC or something.

Presuming that you have sufficient dump space that the dump [and there should have been one] was saved, go to /var/adm/crash and look for a crash.XX directory (where XX starts at 0 and goes up based on the last crash). Check for one from April 18th and give the dump to support if you wish to find out exactly what happened.

If you want to try to figure it out yourself, you really should contact support to get the crashinfo tool [since otherwise, you'll spend a lot of time figuring out how to get what it will summarize for you]. Or use kwdb -q4 in the dump directory, and post the result of "examine &msgbuf+0x8 using s" -- the message buffer up to the panic (and any stack) may give us some idea.

If it really was an externally driven TOC, though, the dump won't know much other than "We got a TOC". If ServiceGuard did it, the dump could indicate why with a bit of work (again, crashinfo would help sum this up).

Re: HPUX Server crash

Tobin Marchbanks — Mon, 21 Apr 2008 17:17:16 GMT

The system is operational, It rebooted and is running fine. I am trying to determine the cause.

Adam,
Nothing changed, as far as I know.

Re: HPUX Server crash

Tobin Marchbanks — Mon, 21 Apr 2008 17:22:07 GMT

INDEX File:

comment savecrash crash dump INDEX file
version 2
hostname lawpr1v2
modelname ia64 hp server rx7640
panic INIT, IIP:0xe000000000635350 IFA:0x9fffffffbf675000
dumptime 1208556292 Fri Apr 18 17:04:52 CDT 2008
savetime 1208556596 Fri Apr 18 17:09:56 CDT 2008
release @(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
memsize 12221382656
chunksize 268435456
defcompchunk 0x0000000000000000

virtual to physical information for IA64
registers 0x24b98fd0
vhpt 0xe000000140000000 2097152
itr 0x24bbb000 31
dtr 0x25213c00 45

module /stand/crashconfig/vmunix vmunix 61228952 1307126553
module /stand/crashconfig/mod/fdd fdd 607304 1793790626
module /stand/crashconfig/mod/dmpaa dmpaa 20384 2270847948
module /stand/crashconfig/mod/dmpaaa dmpaaa 21160 1064655384
module /stand/crashconfig/mod/dmpap dmpap 20808 1051718878
module /stand/crashconfig/mod/dmpapg dmpapg 25080 271114488
module /stand/crashconfig/mod/dmpapf dmpapf 27216 549770395
module /stand/crashconfig/mod/dmpjbod dmpjbod 20440 2589002619
module /stand/crashconfig/mod/dmphpalua dmphpalua 28064 2861264650
module /stand/crashconfig/mod/dmphdsalua dmphdsalua 25528 1988916574
module /stand/crashconfig/mod/rng rng 92744 4252986015
module /stand/crashconfig/mod/ipf ipf 583824 4051795845
module /stand/crashconfig/mod/pfil pfil 134592 3574228516
module /stand/crashconfig/mod/gvid_info gvid_info 30664 2844918749
image image.1.1 0x0000000000000000 0x0000000000401000 0x0000000000004400 0x00000000000047ff 130084519
image image.2.1 0x0000000000000000 0x0000000000401000 0x0000000000004800 0x0000000000004bff 3221218781
image image.3.1 0x0000000000000000 0x0000000000041000 0x0000000000004c80 0x0000000000004cbf 321619409
image image.4.1 0x0000000000000000 0x0000000000041000 0x0000000000004cc0 0x0000000000004cff 824665934
image image.5.1 0x0000000000000000 0x00000000081b0000 0x0000000000020000 0x000000000003bfff 2323278620
image image.6.1 0x0000000000000000 0x000000000ffe8000 0x0000000000400000 0x0000000000486e47 578983274
image image.6.2 0x0000000000000000 0x000000000a297000 0x0000000000486e48 0x00000000004ce087 2847225442
image image.7.1 0x0000000000000000 0x0000000000000000 0x00000000004ce08a 0x00000000004ce08a 4294967295
image image.8.1 0x0000000000000000 0x00000000054f0000 0x00000000004ce08c 0x00000000004eff50 788785196
image image.9.1 0x0000000000000000 0x000000000ffe5000 0x00000000004eff52 0x00000000005883d1 1800547018
image image.9.2 0x0000000000000000 0x000000000aebf000 0x00000000005883d2 0x00000000005fbfff 4105922677
image image.10.1 0x0000000000000000 0x000000000ffef000 0x0000000004040000 0x00000000040c07d7 1601354065
image image.10.2 0x0000000000000000 0x000000000b52a000 0x00000000040c07d8 0x00000000040ffad1 1138444461
image image.11.1 0x0000000000000000 0x0000000000001000 0x00000000040ffb0c 0x00000000040ffb35 3018728591
image image.12.1 0x0000000000000000 0x0000000000001000 0x00000000040ffb72 0x00000000040ffbd7 3018728591
image image.13.1 0x0000000000000000 0x0000000000001000 0x00000000040ffc80 0x00000000040ffdd9 3018728591
image image.14.1 0x0000000000000000 0x0000000000101000 0x00000000040ffe00 0x00000000040ffeff 2512814443
image image.15.1 0x0000000000000000 0x0000000000101000 0x00000000040fff00 0x00000000040fffff 3014269708

Re: HPUX Server crash

Adam Winebaugh — Mon, 21 Apr 2008 17:27:50 GMT

I am with Don on this one then. I am thinking someone hit something they shouldn't have, since you are operating normally. Hmmm I can't think of what else to check.