panic: Spinlock deadlock!

[Rajesh SB]

Hi,

One of our Lab server "Model:9000/800/L2000-44"
installed with OS:UXCoreMedia - B.11.00.02,
Patch bundle:
XSWGR1100 B.11.00.50.5 HP-UX General Release Patches, September 2000
XSWHWCR1100 B.11.00.50.5 HP-UX Hardware Enablement and Critical Patches, September 2000

Suddenly this Server crashed yesterday.
After extracting log from crash dumps using Q$. I found following error

***************************************
vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (4 block extent)
Spinlock timeout failure:
The spinlock code has NOT failed! Instead, some spinlock
using code has failed to release a spinlock soon enough.
Address: 0x000000004007e240X ; owner 0xC8D2C8 ; lock 0x0 ; flag 0x1
next_cpu 0x1
Milliseconds spent spinning =60001
Millseconds/sec = 1000

linkstamp: Sat Mar 9 06:58:49 IST 2002
_release_version: @(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997
__kern_ci_revision: $Header: kern_sequence.c,v 1.6.106.512 97/11/05 18:01:46 msosa Exp $

panic: Spinlock deadlock!

PC-Offset Stack Trace (read across, top of stack is 1st):
0x002db134 0x00090d00 0x000e57b4
0x000e5c28 0x00136774 0x000ad16c
0x000bb204 0x0018917c 0x000d343c
0x00138120 0x000eff34 0x000f0ed4
0x000bc71c 0x000aea40 0x001344e8
0x00035944
End Of Stack

linkstamp: Sat Mar 9 06:58:49 IST 2002
_release_version: @(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997
__kern_ci_revision: $Header: kern_sequence.c,v 1.6.106.512 97/11/05 18:01:46 msosa Exp $

panic: hang detected in update()

*******************

By observing this error

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

I have checked the diskspace of /dev/vg00/lvol5 450MB space was free.
I am attaching the crash_dump log for your referrence.
What could be the reason for this crash.
Any pointer, yours valueable inputs are highly appreciated.

Thanks & Regds,
Rajesh SB

[James A. Donovan · Accepted solution]

1. Make sure you're up-to-date on your patches. There have been many, many, many kernel fixes since Sept. 2000

2. Check /etc/shutdownlog. If the last reboot lists a reboot caused by a panic and includes a hex string begun with "isr.ior" then 99 times out of 100, HP needs to come out and replace a processor.

3. Check /var/tombstones for a ts99 file. Open a call with HP and send them that file and your q4 analysis.

[melvyn burnard]

With a Spinlock Deadlock type panic, it is not so easy to get to the issue that caused it.
I would strongly advise you log a cll wit your local HP Response Centre and have the panic analysed properly.
Failing htat, ensure you have the latest patches installed.

[Pete Randall]

I would tend to suspect that the two symptoms are related. If lvol5 was being written to, it may have filled (temporarily) which would not show after the crash. If lvol5 was unwritable (because it was full), it seems to me that it's possible for that to cause your spinlock problems. What's on lvol5 - /opt, perhaps?

Pete

[Michael Duthie]

Spinlock panic could be anything, I think your error just means a resource was busy while panicing. lvol5 /opt ?

I would either send your dump to HP or apply a newer patch bundle, Sept 2000 is quite old.

[Bill Hassell]

A spinlock panic is not created by disk space issues. A spinlock is a requirement on multi-CPU systems where HP-UX activities are spread out on each processor. However, certain tasks require that all processors suspend current kernel activity while the Monarch processor (the first one) completes a specific task that may affect tthe other parts of the kernel being run in the other processors.

Without a spinlock, other processors may access incomplete or out-of-sync tables and crash the system. A spinlock is a kernel performance hit so they are not performed very often. Early designs for multi-CPU kernels did not scale well onto lots of CPUs due in part to spinlock delays.

If you search the patch database at itrc.hp.com for individual patches, you'll find a plethora of patches that mention spinlock issues. I would look at downloading the latest SupportPlus package of patches (both QPK and HWE) and get the kernel up to date.

[Rajesh SB]

Hi Experts,

First of all I am very thankful for your quick response on my quest.

Hi Jim Donovan,
Reply for your quests.
1. Make sure you're up-to-date on your patches. There have been many, many, many kernel fixes since Sept. 2000

> These installations were perfect.
This m/c is installed in "Mar 25, 2002".

I would like to take into your notice that
This m/c is loaded with OS core using latest media. The following are the components.
"
HPUXEng64RT B.11.00.01 English HP-UX 64-bit Runtime Environment
UXCoreMedia B.11.00.02 HP-UX Media Kit (Reference Only. See Description)"

But September 2000 Patch bundles installed Seperately. As this patch level was our project requirements.


2. Check /etc/shutdownlog. If the last reboot lists a reboot caused by a panic and includes a hex string begun with "isr.ior" then 99 times out of 100, HP needs to come out and replace a processor.

> I checked /etc/shutdownlog This was the shutdown message.

" 16:16 Tue Mar 04 2003. Reboot after panic: Spinlock deadlock! "

3. Check /var/tombstones for a ts99 file. Open a call with HP and send them that file and your q4 analysis.

> Tombstones is not installed on my m/c. Give me any pointer/link on this.

Hi Melvyn Bernard,

I cannot install latest OS patch bundle. As Project requirement is September 2000 patch bundle environment.

Hi Pete, Mike,

/dev/vg00/lvol5 is /var, not /opt.
Here is the current details of /var
/dev/vg00/lvol5 10240000 8930090 1273290 88% /var

Hi Bill Hassell,

Yes, This is dual processor system.
As you said, it could be patch bundle issue. This L-Class Server is not installed with Latest support plus and QPK bundle, as we are looking for OS environment with September 2000 Patch bundle.

Thanks & Regds,
Rajesh SB

[Bill Hassell]

Be very careful about application vendors that demand an obsolete patch level!! Many database vendors do NOT update their documentation, sometimes for years, and often their support people will require an old version, stating that anything newer is not supported. Since you are running a brand new L-2000 computer, the Sep 2000 bundle is incompatible (especially the HWE bundle) since there was no such thing as an L2000 in Sep 2000.

So if you must meet the requirement that only HP-UX with Sep 2000 patch bundle is acceptable, you'll have to trade in your new L2000 computer on an old K-series or maybe an older N-class. I would definitely pursue the obsolete patch bundle issue with your application vendor(s).

[Michael Steele_2]

This was your first problem:

btlan3: reset state is 550 at 0/0/0/0....
btlan3: WARNING: AUI Loopback Failed at 0/0/0/0....

Check your core I/O NIC which is internal to your L2000 at the 0/0/0/0 HW address. Is the LED power light on? Check the cable and port connector going into your switch, try another port.

Additionally, reboot and come up to single user mode:
ISL>hpux -is

Then run level 1, 2, 3 and so on.

># init 1

># init 2

># init 3

Observe the last startup daemon when failure.

[Rajesh SB]

Hi Bill Hassell,

I will agree this could be the problem. In deployment environment, we are having K-Class Server. Application vendor(s) supporting HPUX 11.00 with September 2000 Patch bundle.
In our test case, we are having L-Class server, I feel atleast latest"XSWHWCR1100 HP-UX Hardware Enablement and Critical Patches" should be installed.

One thing really strange, till one year this problem was not there. Recently, I faced this problem once. Now it is OK, still working fine.

I will watch for any repeatation of crash and I will go for upgrading Patch level.

Thanks a lot for yours valueable inputs.
Please can you give any link/pointer on tombstones.

Thanks & Regds,
Rajesh SB

[Michael Duthie]

/var/adm/crash probably filled your /var file system when the system paniced. If you have enough space capacity create a lvol for /var/adm/crash with enough space to dump into.

[Bill Hassell]

It will be hard to predict when there will be another crash. Spinlocks happen all the time but are very short, so a combination of swapping, LAN activity, context switching and perhaps an error retry on disk might trigger this timeout panic. If this is a production system where downtime due to a panic is intolerable, you had better return to the K-series.

The /var/tombstones directory contains a list of the CPU hardware codes when the system crashes. The file ts99 is the most recent one but is of little value unless the hardware failed. Then your choice is to repair the defective board. For software failures such as the spinlock timeout, only patches will prevent this.