n4000 system crash - alert level:12 = software failure

[KiddA]

Hi people, server is not coming up after applying latest qpk.

We already checked gsp, interacted with isl, ma, tried to boot from pri and alt boot path, to see what might be the reason for this error but server still crash.

I would like to know if there's a way to save a crash dump at this point or how to troubleshoot this alert and have server up.

Screenshot attached.

Thanks all for time&help.

[Michael Steele_2 · Accepted solution]

Hi

Boot off of the vmunic.prev kernel, and roll back the QPK with swremove. Do you need the steps? What do you do after you interrupt the boot up at the 10 second prompt?

[lbseraph]

If you still can't bootup, the crash dump file won't save to OS(/var/adm/crash). I am no sure if have other method to save crash dump files before OS up.

Could you bootup to single mode? Have better to remove the QPK firstly, if fail, may need to restore OS from your backup tape.

[KiddA]

Hi Michael ... if you could provide the steps, it would be more than appreciated.

lbseraph, we tried to boot into single user mode, but server crash before getting prompt.

Thanks again guys.

[Michael Steele_2]

Hi

1) reboot
You will see a prompt that says "10 seconds to interrupt boot up - press any key "
2) Press space bar and interrupt
You will see another prompt that says "Do you want to interact with IPL?"
3) yes
4) ISL> hpux /stand/vmunix.prev

Note: Now, you said your box crashed before seeing 2), but you also said "...We already checked gsp, interacted with isl...." - If you can't get to the ISL then you're made.

However, crashing before seeing 2) indicates a HW problem, and this is not consistent with ALERT LEVEL 12 - Software. ALERT LEVEL 12 - Software is an O/S alarm. Not a HW alarm.

Write back with your results.

[Michael Steele_2]

Ooops - Type -

"...If you CAN get to the ISL..."

:-)

[KiddA]

Hi Michael, as you mentioned, and since we weren't able to get prompt we started troubleshooting hardware.

First thing we did, we unplugged fibers that go to SAN, rebooted server and it came up normally.

I'll assign points and leave post open to see if this behavior is familiar to someone else.

Thanks Michael for your help.

[Michael Steele_2]

Hi

Bad HBA card or, no HBA driver. The HBA cards fit into PCI I/O slots. The fibre attach to them on the server side.

Now that you are up, check these logs.

/etc/shutdownlog
/etc/opt/resmong/log/*
/var/adm/syslog/OLDsyslog.log
GSP Event Logs
/var/adm/crash/ - Find the dump from the first day

Get the HBA product numbers and verify the drivers installed and newest offered.