topic Re: n4000 system crash - alert level:12 = software failure in Operating System - HP-UX

n4000 system crash - alert level:12 = software failure

KiddA — Sat, 12 Dec 2009 10:15:12 GMT

Hi people, server is not coming up after applying latest qpk.

We already checked gsp, interacted with isl, ma, tried to boot from pri and alt boot path, to see what might be the reason for this error but server still crash.

I would like to know if there's a way to save a crash dump at this point or how to troubleshoot this alert and have server up.

Screenshot attached.

Thanks all for time&help.

Re: n4000 system crash - alert level:12 = software failure

Michael Steele_2 — Sat, 12 Dec 2009 11:04:10 GMT

Hi

Boot off of the vmunic.prev kernel, and roll back the QPK with swremove. Do you need the steps? What do you do after you interrupt the boot up at the 10 second prompt?

Re: n4000 system crash - alert level:12 = software failure

lbseraph — Sat, 12 Dec 2009 12:36:17 GMT

If you still can't bootup, the crash dump file won't save to OS(/var/adm/crash). I am no sure if have other method to save crash dump files before OS up.

Could you bootup to single mode? Have better to remove the QPK firstly, if fail, may need to restore OS from your backup tape.

Re: n4000 system crash - alert level:12 = software failure

KiddA — Sat, 12 Dec 2009 13:58:55 GMT

Hi Michael ... if you could provide the steps, it would be more than appreciated.

lbseraph, we tried to boot into single user mode, but server crash before getting prompt.

Thanks again guys.

Re: n4000 system crash - alert level:12 = software failure

Michael Steele_2 — Sat, 12 Dec 2009 14:33:54 GMT

Hi

1) reboot
You will see a prompt that says "10 seconds to interrupt boot up - press any key "
2) Press space bar and interrupt
You will see another prompt that says "Do you want to interact with IPL?"
3) yes
4) ISL> hpux /stand/vmunix.prev

Note: Now, you said your box crashed before seeing 2), but you also said "...We already checked gsp, interacted with isl...." - If you can't get to the ISL then you're made.

However, crashing before seeing 2) indicates a HW problem, and this is not consistent with ALERT LEVEL 12 - Software. ALERT LEVEL 12 - Software is an O/S alarm. Not a HW alarm.

Write back with your results.

Re: n4000 system crash - alert level:12 = software failure

Michael Steele_2 — Sat, 12 Dec 2009 14:35:48 GMT

Ooops - Type -

"...If you CAN get to the ISL..."

:-)

Re: n4000 system crash - alert level:12 = software failure

KiddA — Wed, 16 Dec 2009 23:56:22 GMT

Hi Michael, as you mentioned, and since we weren't able to get prompt we started troubleshooting hardware.

First thing we did, we unplugged fibers that go to SAN, rebooted server and it came up normally.

I'll assign points and leave post open to see if this behavior is familiar to someone else.

Thanks Michael for your help.

Re: n4000 system crash - alert level:12 = software failure

Michael Steele_2 — Thu, 17 Dec 2009 00:01:26 GMT

Hi

Bad HBA card or, no HBA driver. The HBA cards fit into PCI I/O slots. The fibre attach to them on the server side.

Now that you are up, check these logs.

/etc/shutdownlog
/etc/opt/resmong/log/*
/var/adm/syslog/OLDsyslog.log
GSP Event Logs
/var/adm/crash/ - Find the dump from the first day

Get the HBA product numbers and verify the drivers installed and newest offered.