Crash dump OpenVMS on lock manager

[Toine_1]

Hi,

Has anyone had this problem before?
During a reboot of a Alpha server with OpenVMS 8.3 I got this crash dump.
Is this a software problem or a hardware problem.

15:23:27 ** OpenVMS Alpha Operating System V8.3 - BUGCHECK ****
15:23:27
15:23:27 Bugcheck code = 000005A4: LOCKMGRERR, Error detected by Lock Manager
15:23:27 Crash CPU: 00000002 Primary CPU: 00000000 Node Name: NVE
15:23:27 Supported CPU count: 00000040
15:23:27 Active CPUs: 00000000.000000FF
15:23:27 Current Process: NULL
15:23:27 Current PSB ID: 00000001
15:23:27 Image Name:
15:23:41
15:23:41 Dumping error log buffers to HBVS unit 110
15:23:45

All help is welcome.

Regards,

/Toine

[Toine_1]

Extra info:
Crashdump Summary Information:
------------------------------
Crash Time: 7-AUG-2009 15:23:23.53
Bugcheck Type: LOCKMGRERR, Error detected by Lock Manager
Node: NVE (Cluster)
CPU Type: hp AlphaServer GS1280 7/1150
VMS Version: V8.3
Current Process: NULL
Current Image:
Failing PC: FFFFFFFF.804979D8 LCK$RCV_LOCKREQ_C+00CF8
Failing PS: 38000000.00000804
Module: SYS$CLUSTER (Link Date/Time: 29-NOV-2006 15:43:55.96)
Offset: 000299D8

Boot Time: 7-AUG-2009 15:11:27.00
System Uptime: 0 00:11:56.53
Crash/Primary CPU: 2./0.
System/CPU Type: 270F
Saved Processes: 216
Pagesize: 8 KByte (8192 bytes)
Physical Memory: 8192 MByte (268435456 PFNs, discontiguous memory)

Crashdump Summary Information:
------------------------------
Dumpfile Pagelets: 1814672 blocks
Dump Flags: olddump,writecomp,errlogcomp
Dump Type: compressed,selective,shared_mem
EXE$GL_FLAGS: poolpging,init,bugdump
Paging Files: 3 Pagefiles and 1 Swapfile installed

Stack Pointers:
KSP = FFFFFFFF.A6535DB8 ESP = FFFFFFFF.A6537000 SSP = FFFFFFFF.A6531000
USP = FFFFFFFF.A6531000

General Registers:
R0 = 00000000.00000124 R1 = 00000000.00000100 R2 = 00000000.00000000
R3 = FFFFFFFE.74A93D80 R4 = 00000000.00000000 R5 = FFFFFFFF.829FEE40
R6 = FFFFFFFE.74A93D80 R7 = FFFFFFFE.78544300 R8 = FFFFFFFE.785455C0
R9 = FFFFFFFF.81A5C0F0 R10 = FFFFFFFE.77C670C0 R11 = 00000000.00000124
R12 = 00000000.0000000A R13 = FFFFFFFF.819870A8 R14 = FFFFFFFF.81808000
R15 = FFFFFFFF.81989050 R16 = 00000000.000005A4 R17 = FFFFFFFF.FFFFFFF1
R18 = 00000000.00001439 R19 = 00000000.00000000 R20 = FFFFFFFF.8006AF9C
R21 = 00000000.0C010000 R22 = 00000000.00000000 R23 = 00000000.00000010

Crashdump Summary Information:
------------------------------
R24 = 00000000.00000001 AI = 00000000.00000001 RA = FFFFFFFF.801FA444
PV = 00000000.00000001 R28 = FFFFFFFF.804972D8 FP = FFFFFFFF.A6535FC0
PC = FFFFFFFF.804979DC PS = 38000000.00000804

System Registers:
Page Table Base Register (PTBR) 00000000.000007B5
Processor Base Register (PRBR) FFFFFFFF.829FDF80
Privileged Context Block Base (PCBB) 00000000.02DFE000
System Control Block Base (SCBB) 00000000.00006C00
Software Interrupt Summary Register (SISR) 00000000.00000100
Address Space Number (ASN) 00000000.00000000
AST Summary / AST Enable (ASTSR_ASTEN) 00000000.00000000
Floating-Point Enable (FEN) 00000000.00000000
Interrupt Priority Level (IPL) 00000000.00000008
Machine Check Error Summary (MCES) 00000000.00000000
Virtual Page Table Base Register (VPTB) FFFFFEFA.00000000


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Crashdump Summary Information:
------------------------------
Failing Instruction:
LCK$RCV_LOCKREQ_C+00CF8: BUGCHK

Instruction Stream (last 20 instructions):
LCK$RCV_LOCKREQ_C+00CA8: STQ R16,#X0008(SP)
LCK$RCV_LOCKREQ_C+00CAC: STQ R17,(SP)
LCK$RCV_LOCKREQ_C+00CB0: LDQ R17,#X0018(R15)
LCK$RCV_LOCKREQ_C+00CB4: BIS R17,#X04,R16
LCK$RCV_LOCKREQ_C+00CB8: LDQ R17,(SP)
LCK$RCV_LOCKREQ_C+00CBC: LDA SP,#X0008(SP)
LCK$RCV_LOCKREQ_C+00CC0: BUGCHK
LCK$RCV_LOCKREQ_C+00CC4: HALT
LCK$RCV_LOCKREQ_C+00CC8: LDQ_U R31,(SP)
LCK$RCV_LOCKREQ_C+00CCC: LDQ_U R31,(SP)
LCK$RCV_LOCKREQ_C+00CD0: LDL R21,#X006C(R9)
LCK$RCV_LOCKREQ_C+00CD4: SRL R21,#X1B,R27
LCK$RCV_LOCKREQ_C+00CD8: BLBC R27,#XFFFEA9
LCK$RCV_LOCKREQ_C+00CDC: LDA SP,#XFFF0(SP)
LCK$RCV_LOCKREQ_C+00CE0: STQ R16,#X0008(SP)


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Crashdump Summary Information:
------------------------------
LCK$RCV_LOCKREQ_C+00CE4: STQ R17,(SP)
LCK$RCV_LOCKREQ_C+00CE8: LDQ R17,#XF4D0(R15)
LCK$RCV_LOCKREQ_C+00CEC: BIS R17,#X04,R16
LCK$RCV_LOCKREQ_C+00CF0: LDQ R17,(SP)
LCK$RCV_LOCKREQ_C+00CF4: LDA SP,#X0008(SP)
LCK$RCV_LOCKREQ_C+00CF8: BUGCHK
LCK$RCV_LOCKREQ_C+00CFC: HALT
LCK$RCV_LOCKREQ_C+00D00: LDA SP,#XFFF0(SP)
LCK$RCV_LOCKREQ_C+00D04: STQ R16,#X0008(SP)
LCK$RCV_LOCKREQ_C+00D08: STQ R17,(SP)
SDA>
SDA> Clue stack
[1;1f [J
Stack Decoder:
--------------
System Stack (NULL Process):
Stack Pointer FFFFFFFF.A6535DB8
Stack Limits (low) FFFFFFFF.A6532000
(high) FFFFFFFF.A6536000


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Bugcheck Stack:
---------------
Stack Pointer SP => FFFFFFFF.A6535DB8

Stack (not decoded):
FFFFFFFF.A6535DB8 FFFFFFFE.74A8E000
FFFFFFFF.A6535DC0 81A5C010.0001004A SYS$K_VERSION_16+0000A
FFFFFFFF.A6535DC8 81A5C0F0.82D95640
FFFFFFFF.A6535DD0 00000000.00000000
FFFFFFFF.A6535DD8 FFFFFFFF.8048BA6C PROCESS_XFER_C+0029C
FFFFFFFF.A6535DE0 FFFFFFFF.81A5C1A8
FFFFFFFF.A6535DE8 FFFFFFFF.81A5CCB8
FFFFFFFF.A6535DF0 FFFFFFFF.81A6C210
FFFFFFFF.A6535DF8 FFFFFFFF.8387FAF8
FFFFFFFF.A6535E00 FFFFFFFF.82D95640
FFFFFFFF.A6535E08 FFFFFFFF.8180B1C8 SMP$GQ_DEBUG
FFFFFFFF.A6535E10 FFFFFFFF.81A5C0F0
FFFFFFFF.A6535E18 FFFFFFFF.81A5C010
FFFFFFFF.A6535E20 00000038.00000078
FFFFFFFF.A6535E28 000001F2.00000123
FFFFFFFF.A6535E30 00000000.00000008


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Bugcheck Stack:
---------------
FFFFFFFF.A6535E38 FFFFFFFF.8048B484 BLOCK_READ_DONE_C+000C4
FFFFFFFF.A6535E40 FFFFFFFF.81A5C0F0
FFFFFFFF.A6535E48 FFFFFFFF.82D95640
FFFFFFFF.A6535E50 FFFFFFFF.81A5C000
FFFFFFFF.A6535E58 FFFFFFFF.81A5C010
FFFFFFFF.A6535E60 FFFFFFFF.81A5C1A8
FFFFFFFF.A6535E68 00000000.0000003B
FFFFFFFF.A6535E70 FFFFFFFF.81A5C220
FFFFFFFF.A6535E78 FFFFFFFF.8387FAF8
FFFFFFFF.A6535E80 00000000.00000043
FFFFFFFF.A6535E88 00000000.00000000
Null Frame:
PV FFFFFFFF.A6535E90 FFFFFFFF.81987008 BLOCK_READ_DONE+00028
Entry Point FFFFFFFF.8048B410 BLOCK_READ_DONE_C+00050
FFFFFFFF.A6535E98 FFFFFFFF.81808000 EXE$GR_SYSTEM_DATA_CELLS
FFFFFFFF.A6535EA0 00000000.00000008
FFFFFFFF.A6535EA8 00000002.00000000
FFFFFFFF.A6535EB0 FFFFFFFF.80059E94 FORK_DISPATCH+00094
FFFFFFFF.A6535EB8 FFFFFFFF.81A5C010


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Bugcheck Stack:
---------------
FFFFFFFF.A6535EC0 FFFFFFFF.829FDFB8 CPUDB+00038
FFFFFFFF.A6535EC8 00000000.00000000
FFFFFFFF.A6535ED0 00000000.00000002
FFFFFFFF.A6535ED8 FFFFFFFF.81855818 EXE_STD$PRIMITIVE_FORK+00020
FFFFFFFF.A6535EE0 FFFFFFFF.81842910 EXE$TIMEDWAIT_SETUP
FFFFFFFF.A6535EE8 FFFFFFFF.8180ACC8 MMG$GL_RES_MEM_FLINK
FFFFFFFF.A6535EF0 00000000.00000050
FFFFFFFF.A6535EF8 00000000.00000000
FFFFFFFF.A6535F00 FFFFFFFF.818E51C8 SCH$GL_NULLPCB+00E00
FFFFFFFF.A6535F08 FFFFFFFF.818E7A60 SCH$IDLE
FFFFFFFF.A6535F10 00000000.00000000
FFFFFFFF.A6535F18 FFFFFFFF.81809E68 SCH$AR_NULLPCB
FFFFFFFF.A6535F20 00000000.00000002
FFFFFFFF.A6535F28 00000000.00000000
FFFFFFFF.A6535F30 FFFFFFFF.801409F8 SCH$IDLE_C+00098
FFFFFFFF.A6535F38 00000000.0000000F
FFFFFFFF.A6535F40 FFFFFFFF.8180ACC8 MMG$GL_RES_MEM_FLINK
FFFFFFFF.A6535F48 00000000.00000000
FFFFFFFF.A6535F50 00000000.00000002


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Bugcheck Stack:
---------------
FFFFFFFF.A6535F58 00000000.01BD5BA2
FFFFFFFF.A6535F60 FFFFFFFF.81812C20 MMG$GL_ZERO_LIST_HI_LIM
FFFFFFFF.A6535F68 FFFFFFFF.81808000 EXE$GR_SYSTEM_DATA_CELLS
FFFFFFFF.A6535F70 FFFFFFFF.81808000 EXE$GR_SYSTEM_DATA_CELLS
FFFFFFFF.A6535F78 FFFFFFFF.A6535FC0
FFFFFFFF.A6535F80 FFFFFFFF.8180A1C0 SCH$GQ_ACTIVE_PRIORITY
Null Frame:
PV FFFFFFFF.A6535F88 FFFFFFFF.818ECA60 SCH$WAIT_ANY_MODE
Entry Point FFFFFFFF.801721D0 SCH$WAIT_ANY_MODE_C
FFFFFFFF.A6535F90 FFFFFFFF.818E51C8 SCH$GL_NULLPCB+00E00
FFFFFFFF.A6535F98 00000000.00000004
FFFFFFFF.A6535FA0 FFFFFFFF.829FDF80 CPUDB
FFFFFFFF.A6535FA8 00000000.00000000
FFFFFFFF.A6535FB0 FFFFFFFF.80140C20 SCH$IDLE_C+002C0
FFFFFFFF.A6535FB8 00000000.00000300 SLOT$K_LENGTH

Stack Frame:
PV FFFFFFFF.A6535FC0 FFFFFFFF.818E7A60 SCH$IDLE
Entry Point FFFFFFFF.80140960 SCH$IDLE_C


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Bugcheck Stack:
---------------
FFFFFFFF.A6535FC8 00000000.00000000
FFFFFFFF.A6535FD0 FFFFFFFF.839731C0
return PC FFFFFFFF.A6535FD8 FFFFFFFF.80171F48 SCH$INTERRUPT+00C28
saved R3 FFFFFFFF.A6535FE0 FFFFFFFF.818ECA60 SCH$WAIT_ANY_MODE
saved R13 FFFFFFFF.A6535FE8 FFFFFFFF.818F45B0 SWP$MAIN_LOOP+000B0
saved R14 FFFFFFFF.A6535FF0 FFFFFFFF.81808000 EXE$GR_SYSTEM_DATA_CELLS
saved FP FFFFFFFF.A6535FF8 00000000.00000000
SDA>
SDA> clue reg
[1;1f [J
Current Registers: Process index: 0000 Process name: NULL PCB: 818E43C8 (CPU 2)
---------------------------------------------------------------------------------------
R0 = 00000000.00000124 %SYSTEM-F-INSFMEM, insufficient dynamic memory
R1 = 00000000.00000100
R2 = 00000000.00000000
R3 = FFFFFFFE.74A93D80 LKB
R4 = 00000000.00000000
R5 = FFFFFFFF.829FEE40 LCKCPU
R6 = FFFFFFFE.74A93D80 LKB
R7 = FFFFFFFE.78544300 RSB
R8 = FFFFFFFE.785455C0 RSB
R9 = FFFFFFFF.81A5C0F0
R10 = FFFFFFFE.77C670C0 RSB
R11 = 00000000.00000124
R12 = 00000000.0000000A
R13 = FFFFFFFF.819870A8 PROCESS_XFER
R14 = FFFFFFFF.81808000 EXE$GR_SYSTEM_DATA_CELLS
R15 = FFFFFFFF.81989050 LCK$BLOCK_RBLD
R16 = 00000000.000005A4
R17 = FFFFFFFF.FFFFFFF1
R18 = 00000000.00001439


[22;1f
Press RETURN for more.
SDA>
[1;1f [J
Current Registers: Process index: 0000 Process name: NULL PCB: 818E43C8 (CPU 2)
---------------------------------------------------------------------------------------
R19 = 00000000.00000000
R20 = FFFFFFFF.8006AF9C LDR_STD$DEALLOC_S2_VA_C+0063C
R21 = 00000000.0C010000
R22 = 00000000.00000000
R23 = 00000000.00000010
R24 = 00000000.00000001
AI = 00000000.00000001
RA = FFFFFFFF.801FA444 LCK$FILL_LKB_CACHE_C+00064
PV = 00000000.00000001
R28 = FFFFFFFF.804972D8 LCK$RCV_LOCKREQ_C+005F8
FP = FFFFFFFF.A6535FC0
PC = FFFFFFFF.804979DC LCK$RCV_LOCKREQ_C+00CFC
PS = 38000000.00000804 Kernel Mode, IPL 8, Interrupt
SDA>

[cdan]

Seems to be software , see here R0 = 00000000.00000124 %SYSTEM-F-INSFMEM, insufficient dynamic memory

Have you tried autogen to see if any parameters need adjustment? My best guesses are NPAGEDYN or LOCKIDTBL.

I would also do a SDA> show memory/pool

[Toine_1]

Extra info about the crash dump:

show mem/pool

System Memory Resources from Crashdump on 7-AUG-2009 15:23:23.53
-----------------------------------------------------------------

Dynamic Memory Usage: Total Free In Use Largest
Nonpaged Dynamic Memory (MB) 156.39 129.81 26.58 128.47
Bus Addressable Memory (MB) 4.12 4.10 0.01 4.10
Paged Dynamic Memory (MB) 9.53 5.24 4.28 5.24
Lock Manager Dyn Memory (MB) 73.36 26.01 47.35

sho mem/pool/full

System Memory Resources from Crashdump on 7-AUG-2009 15:23:23.53
-----------------------------------------------------------------

Nonpaged Dynamic Memory (Lists + Variable)
Current Size (MB) 156.39 Current Size (Pagelets) 320304
Initial Size (MB) 156.39 Initial Size (Pagelets) 320304
Maximum Size (MB) 520.80 Maximum Size (Pagelets) 1066608
Free Space (MB) 129.81 Space in Use (MB) 26.58
Largest Var Block (MB) 128.47 Smallest Var Block (MB) 128.47
Number of Free Blocks 1775 Free Blocks LEQU 64 bytes 0
Free Blocks on Lookasides 1774 Lookaside Space (MB) 1.34

Bus Addressable Memory (Lists + Variable)
Current Size (MB) 4.12 Current Size (Pagelets) 8448
Initial Size (MB) 4.12 Initial Size (Pagelets) 8448
Free Space (MB) 4.10 Space in Use (MB) 0.01
Largest Var Block (MB) 4.10 Smallest Var Block (KB) 6.87
Number of Free Blocks 2 Free Blocks LEQU 64 bytes 0
Free Blocks on Lookasides 0 Lookaside Space (By) 0.00

Paged Dynamic Memory
Current Size (MB) 9.53 Current Size (Pagelets) 19520
Free Space (MB) 5.24 Space in Use (MB) 4.28
Largest Var Block (MB) 5.24 Smallest Var Block (By) 16.00
Number of Free Blocks 262 Free Blocks LEQU 64 bytes 260

Lock Manager Dynamic Memory
Current Size (MB) 73.36 Current Size (Pages) 9391
Free Space (MB) 26.01 Hits 748899
Space in Use (MB) 47.35 Misses 161
Number of Empty Pages 3410 Expansions 9391
Number of Free Packets 85250

[Volker Halle · Accepted solution]

Toine,

you've hit some resource problem in the lock manager. The lock manager does not use nonpaged pool much. It has it's own pool zone in S2 space (for LKBs, RSBs etc.).

See a similar case in ITRC:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=997728

This is a software problem. It could occur during lock remastering, when a huge lock tree is being moved to the newly booted local node.

If you want a detailled analysis, explanation and guidance on how to prevent this from happening again, you owuld have to escalate this crash to HP.

Volker.