K370 server crash

[Kamlesh Shete]

Hi,

We have a K370 running on 11.0, which has been crashing lately. Looks like its a system panic and all it does is reboot the server or just freeze dead.

I ran q4 . in the /var/adm/crash/crash.x directory and then ran a trace, the output of which is given below.

q4> crash event was an HPMC
q4> The Save State registers for this level are:
r0 /r1 /r2 0 0x6ce7d8 0x1
r3 /r4 /r5 0x8e0ec0 0x1 0xffffffff
r6 /r7 /r8 0x1b76dc0 0x1 0x1b48200
r9 /r10/r11 0x5c2b6100 0x4850580 0xa499000
r12/r13/r14 0xf2 0xfa 0xf2
r15/r16/r17 0xf2 0xf0001780 0xf0000124
r18/r19/r20 0xf000011c 0 0x8e151c
r21/r22/r23 0 0 0x4
r24/r25/r26 0 0 0
r27/r28/r29 0x6d0fd8 0 0x170deb3
r30/r31/r32 0x2e6a130 0
sr0 /sr1 /sr2 0 0x7298000 0x4ec5400
sr3 /sr4 /sr5 0 0 0xb9e5000
sr6 /sr7 /sr8 0xbd88000 0
LEVEL FUNC ARG0 ARG1 ARG2 ARG3
lev 0) idle+0x8b8 n/a n/a n/a n/a
lev 1) swidle_exit+0x0 n/a n/a n/a n/a

Any ideas/input greatly appreciated

[Patrick Wallek]

The crash was by an HPMC (High Priority Machine Check). What you need to find out now is what caused the HPMC. Check these files for more information: /etc/shutdownlog, /var/adm/syslog/OLDsyslog.log and see if they give any clues as to what was going on when it crashed. There should be something in those files.

Feel free to post contents of those files and we will help where we can.

[Steffi Jones_1]

Hello,

considering that an HPMC is most likely a hardware issue and you have to place a call to the support center to schedule a replacement ask them to help to determine which part went bad on you.

A few more things you can look into are:

- dmesg output
- syslog and OLDsyslog
- /var/tombstones/ts99 file

If you know how you can run stm and try to narrow the root cause.

But best thing will be to call HP and schedule time to have the box fixed.

Steffi Jones

[Kamlesh Shete]

Hi Patrick,

Here's one of the lines in the /etc/shutdownlog:

Reboot after Panic: , isr.ior = 0'240001.0'b3acfe70

There is no /var/tombstones directory.

Thanks and appreciate.

[Steffi Jones_1]

Hello again,

from past experience isr.ior panic strings point most likely to hardware.

I would highly recommend to have someone help you run the diagnostics and get the part replaced, so that you don't risk another crash and another unplanned downtime.

Steffi Jones

[paul courry]

If you will post the fault codes from either the LCD display on the computer or the console we can interpret them and narrow down the source of the problem. There may be several of them.

[Ajitkumar Rane]

If you do not have the /var/tombstones/ts99 for anlysing,Next time when the system reboots and if you r present or you can schedule downtime, you should halt the boot process and boot the system to the ISL prompt and than you can run PIM, which is what is written in the ts99 file.Thru the PIM report you would be able to narrow down on the problem also dont forget to check the time stamp.Hope this helps

Goodluck.

[Kamlesh Shete]

All,

I had the oppurtunity to reboot my server. Its sitting at the ISL prompt, but if I type in PIM it says invalid command.

I'm trying to get the ts99 file in the /var/tombstones directory.

Any help ASAP greatly appreciated.

[Alan Riggs]

/var/tombstones are set by online disagnostics. If you do not have online diagnostics installed, then you will need to do so. Then simply create the /var/tombstones directory. The ts?? files will be updated automatically at system startup.

To get into PIM, I'm afraid, you need to select it from teh menu BEFORE booting into isl mode (IIRC--I haven't gone that route for quite some time.) It accesses processor code, not the system loader.