server crashed: Bad News in panic string

[Ra Jose]

I have IA64 running HP-UX 11iv2 that crashed.
Examined the crash dump using Q4 and here is what I found.

The crash occurred at 6:30:38 (Wed Jun 13) due to panic.

The panic string says "Bad News!". The message buffer examination shows "Bad News: Cannot use the Kernel Stack when interrupted on the ICS.".

Further examination revealed, processor 0 panicked.

Does anyone know what this ICS mean. It says Bad News ... any clues as to what it might suggests.

This server runs Oracle 10gR2 (RAC).

Thank you all for your resposes.

[Don Morris_1 · Accepted solution]

Interrupt Control Stack is ICS.

Bad News is the string you get when the trap handler doesn't know what to do with a trap [not sure the history of it... never bothered to track it down].

It sounds in this case like a bad pointer dereference within interrupt context (and since you are using the kernel stack in interrupt context... you aren't allowed to switch to a fresh kernel stack on a translation fault in interrupt context... hence the fault can't be handled, which is Bad News).

The panic stack trace would be helpful here (at the least we could search and see if there's a similar issue). Bad News is way too generic, I'm afraid. [In q4, "trace event 0" and post the result].

[Ra Jose]

Thank you Don.

here is event 0 trace.

q4> trace event 0
stack trace for event 0
crash event was a panic
Stack unwind reached bottom at depth, 1

panic_save_regs_switchstack+0x110
panic_save_regs_switchstack+0x110

Rajose.

[Don Morris_1]

Ugh... nothing good there (sometimes the stack unwinding doesn't work well). Try tracing event 1 to see if it gives anything more meaningful -- if not, do "examine &msgbuf+0x8 using s" and scan the message buffer to see if the bad news panic managed to dump the interrupt stack so we can have some idea of what this thing was doing.

An alternative to either of those (since its leaked out there anyway), grab crashinfo, run it -- and see what it has to say in the Panic section up front as to what happened.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1089090

[Ra Jose]

q4> trace event 1
stack trace for event 1
crash event was a INIT
Stack unwind reached bottom at depth, 1

The msg buf showed

Bad News: Cannot use the Kernel Stack when interrupted on the ICS.

******************************************************************************
******************************************************************************

reg_dump(): Displaying register values (in hex) from the save state at
ssp e0000001_017b7c00 return_status/reason/flags 0000/0056/00000001

Interruption type: Register NaT Consuption Fault

We have logged call with HP.

Rajose.

[Sameer_Nirmal]

ICS - Interrupt Control Stack
It is also referred/called as kernel stack. It stores the save state structure.

We had a panic with the same Bad News message on one of our rx8640s installed with Sep'06 11.23 version. However there was a mention of a tcp register in the stack trace and there is patch PHNE_35182 for its resolution.

Try by running crashinfo to get more information about panic in your case.

[Ra Jose]

We have put call to HP support. We have 4 servers which are exactly same in hardware/firmware/OS and all 4 are in Oracle 10gR2 RAC cluster. They have been doing fine for over 2 months. Nothing has changed since. All of sudden, just one server crashes. It crashed several times yesterday. The other 3 are fine. So it is tricky, what could be triggering this crash.
- Rajose

[Ra Jose]

Thank you Sameer and all. Looks like Sameer's suggestion worked. We applied the patch and for about 10 hrs now, no crashes yet. So good news. Thank you all for your input.
RaJose.

[Ra Jose]

Thank you.