Panic Message from 11i Running EVA+XP+VxVM - Anyone Have Any Similar Experience

[Alzhy]

On our nth month now of troubleshooting what could probably be the cause. Servers involved are rp84xx and Superdome. Happens erratically. The following is the consistent panic string - anyone have any inkling of what may be wrong? We run 11i (11.11), VxVM 3.5 U3(Full License), SecurePath Managed EVA5K storage alongside XP12000 storage (path managed by DMP):

The spinlock code has NOT failed! Instead, some spinlock
using code has failed to release a spinlock soon enough.
Address: 0x000000009f610000X ; owner 0x0000000000000000X ; lock 0x0000000000000000X ; flag 0x1
next_cpu 0x0
Milliseconds spent spinning =60001
Millseconds/sec = 1000

System Panic:

linkstamp: Sat Jun 18 16:31:41 PDT 2005
_release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
panic: Spinlock deadlock!

PC-Offset Stack Trace (read across, top of stack is 1st):
0x00253480 0x001bccac 0x001bd008
0x00034c1c 0x001d8524 0x001d8448
0x001e416c 0x001f2a2c 0x001f27d0
0x001a0724 0x0019c818 0x0039d52c
0x004aee68 0x0039d930 0x001792c4
0x00200de4
End Of Stack

[melvyn burnard]

Spinlock panics are not so easy to analyse. I would strongly recommend you log a call with your local HP Response Centre, and ask for these dumps to be analysed.

[Alzhy]

We've done that.

HP's finest have been looking at this issue for the past 3+ months with no definite cause and resolution.

That's why I am posting the panic message and stack trace in the hope that some poor souls out there have experienced this in the past.

[melvyn burnard]

AH, not good. One question to ask is whether you have requested that your problem be formally escalated within HP?

[Cheryl Griffin]

I searched for calls that you have opened and found none for this issue.

I would do as Melyn suggested and pursue the issue with the Response Center.

[Albert Smith_1]

In the past of getting spinlock dead lock panics they 100% time relate to CPU failure. 95% of the time it would be the monarch or CPU 0. I would check your cell board 0 there might be a faulty CPU there.

I would hope you are getting a full core if not they only way HP could not analyze it or have problems analyzing it is if the core is incomplete.

I assume that your call should be in the down system team if not I would get it there and escalate it to your local district manager and get the techs working on it.

Worse case if HP can't get a resolve I would demand a CE out there and replace all the Cell boards and CPU's. Then issue a TOC and see if you get the spinlock deadlock.

If not I would wait if it occurs again. If it does I would get the system board replaced.

[Kent Ostby]

Nelson --

Can you post the trace.out file (i.e. Q4 output from : trace event 0 ).

[Alzhy]

Melvyn, I was not the one who opened the tickets. Here's some of the tkts: 3211605341, 3210277549, 3210350572, 3210516104, 3210277549.

Albert - all of those you mentioned have already been done by HP.

Kent, if you could give me instructions on how to get the info you need and its not disruptive.. I will try.

[Kent Ostby]

Nelson, I'm sure I"ll be able to find what I need in the tickets you listed above since I work at the RC.

[Kent Ostby]

Nelson -- Sorry. I have not seen anything like that. Looking at the WFM cases, it looks like your in the correct process at this point.

Sorry that this hasn't gotten resolved already.

[Alzhy]

But as far as supportability, will you agree that we have a completely supported environment in the manner that we use it -- meaning 11i + VxVM 3.5 (Full License) + EVA (with SecurePath 3.0E path protecting it) + XP12000 (sans AutoPath since we use VxVM DMP)?

In our environment LVM still manages our local boot/swap disks. We have this setup since we no longer trust single array solutions - we mirror (using VxVM FlashSnap or fast mirror resynch) between the XP and the EVA. I've gotten assurances in the past that XP12K and EVA/SecurePath should work side by side. Even Veritas' latest HCL lists EVA/SecurePath coexisting with DMP enabled devices like the fully supported XP12000.

[Kent Ostby]

I'm not well versed in what is and what is not supported but knowing the guys who are working your case, I'm sure they would have raised that issue by now if they did not feel you were in a supported configuration.