"Reboot after panic: , isr.ior = 0'10240034.0'dff7e124"

[SUDHEER M S]

Dear All,

I am facing issue with my K Class clustered Server.
Server Model: K Class 580
OS Version : HP-UX 11
Server Type : Cluster Server
Other Info : No messages in syslog /syslog.old
Problem: Server rebooted giving error in shutdownlog as

22:01 Sat Nov 05 2005. Reboot after panic: , isr.ior = 0'10240034.0'dff7e124
22:05 Mon Nov 07 2005. Reboot after panic: , isr.ior = 0'0.0'0

Can any one help me please, Is this error shows a hardware or software error ?.

Regards,
MS

p.S. This thread has been moved from HP-UX > System Administration to HP 9000. - Hp Forum Moderator

[RAC_1]

This COULD be MCSG or HPMC crash. HPMC crash is related to hardware problem. Did you look at crash files??

A Primary analysis of crash (aprt from Q4 analysis) can be done as follows.

adb -m /var/adm/crash/crash_dir/vmunix /var/adm/crash/crash.0

Once on adb prompt, do
msgbuf+8/s

for HPMC relate crash look at /var/tombstones/ts99 file.

[Arunvijai_4]

Go to GSP prompt and look at the log files. It will give more information on what went wrong.

-Arun

[AwadheshPandey]

check these threads,

http://unix.derkeiler.com/Mailing-Lists/HP-UX-Admin/2004-04/0073.html
http://unix.derkeiler.com/Mailing-Lists/HP-UX-Admin/2004-04/0075.html

Awadhesh

[Moderator edit: Removed the broken links.]

[SUDHEER M S]

hudhp01# cat /var/tombstones/ts99 |grep "HPMC Chassis"
HPMC Chassis Codes = 0xcbf0 0x5002 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5108 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5208 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5308 0x5408 0x5508 0x5608 0x5708
hudhp01# cat /var/tombstones/ts98 |grep "HPMC Chassis"
HPMC Chassis Codes = 0xcbf0 0x5002 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5108 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5208 0x5408 0x5508 0x5608 0x5708
HPMC Chassis Codes = 0xcbf0 0x5308 0x5408 0x5508 0x5608 0x5708

These are the messages I am getting for the grep of /var/tombstones/ts98 and 99. Can you please tell me , what it indicates.

[RAC_1]

Very few of us wil be able to decode the HPMC chassis codes.

Did you do following??

adb -m /var/adm/crash/crash_dir/vmunix /var/adm/crash/crash.0
Once on adb prompt, domsgbuf+8/s

[Ranjith_5]

Hi Sudheer,


grep -i "HPMC Chassis" /var/tombstones/ts99

See the output. Mostly you will need a crash dump analysis to find out the exact problem( HPMC/LPMC). Case can be escalated to HP. I strongly feel that this job is not coming under the scope of your support.



Also check for the patch level of the machine. It make sure that your syustem is updated with latest patch bundles released. This needs to be checked in priority.

See these threads as well.

http://www.darklab.net/resources/hpux-mailinglist/2933.html
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=207445
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=747375

[Moderator edit: The above links are no longer valid.]

Hope this helps,
Regards,
Syam

[Ranjith_5]

Hi,

Since this is a node in the SG cluster,it could be caused by MC/Servicguard losing connection between the nodes. Look at the OLDsyslog.log on the node that crashed or the syslog.log on the node that didn't crash for that time frame and see if you see messages related to losing contact with each other.

Also pls post the version of Serviceguard. Older versions have got lot of bugs which have been resolved in latest versions.

Regards,
Syam

[SUDHEER M S]

I Copied this file to another directory as there is no space on the said crash directory. Please see the below error message.

hudhp01# adb -m /var/opt/genfts/outfile/DHP_BW/test/vmunix /var/opt/genfts/outfile/DHP_BW/test
crash dump: No such file or directory
Error on open of corfil = /var/opt/genfts/outfile/DHP_BW/test.

Version is : A.11.09
Syam as you said I am able to see the following error in syslog of the other cluster server

Nov 7 21:38:06 hudhp02 cmcld: Timed out node hudhp01. It may have failed

Any new ideas.

Regards
MS

[Ranjith_5 · Accepted solution]

Hi Sudheer,

Posted the pre-processing of coredump.
==========================================

See checklist for the same.

http://67.176.164.101/ll/hpq4.html

=============================================================================
How do I use q4 to pre-process a dump that HP needs to read
DocId: OZBEKBRC00000611 Updated: 2/13/04 5:52:00 AM
=============================================================================
PROBLEM

Prior to Opening a Call with HP, I'd like to know the best way to pre-process
my crash dump so HP can troubleshoot it.

RESOLUTION

ITRC DOCUMENT ID: OZBEKBRC00000611

USING Q4 TO ANALYZE SYSTEM DUMP FILES
(For HPUX 10.10-11.20 systems)

[NOTE: These guidelines are for North American
customers. Other locales may use a different
procedure for preprocessing dump files]

==============================================================
===
"Edited to comply with ITRC Guidelines"

Regards,
Syam

[Moderator edit: The above link is no longer valid.]