CRASH ANALYSIS of VAX/VMS server

[Sk Noorul Hassan]

Can any body help me analysing the reason for crash, of a VAX 4105 server/VMS. For that, what are the dumps or file u need, just let me know so that, I will post those dumps for u.

Please help...

[Jim_McKinney]

The output of (some or all of) the following, cut, pasted, and attached here as a .TXT file might provide a start (though a diagnosis may be impossible in a forum such as this)...

$ show system/noproc
$ diag/sinc=28-jan-2006/incl=(cpu,mem,bug,mach)
$ anal/cras sys$system:sysdump.dmp
SDA> clue crash
SDA> ^Z

[Volker Halle]

Hi,

this is a VAX, so there is no SDA> CLUE...

Let's start with:

$ ANAL/CRASH SYS$SYSTEM:SYSDUMP.DMP
SDA> READ/EXEC
SDA> SHOW CRASH
SDA> SHOW STACK

$ ANAL/ERR/SINCE=/BEFORE=

Volker.

[Ian Miller.]

What version of VMS?

On VAX/VMS CLUE is a seperate program so instead of using the CLUE command after ANAL/CRASH you have to

MCR CLUE SYS$SYSTEM:SYSDUMP.DMP

you may already have a file in
SYS$MANAGER:CLUE*.LIS
(check the dates).

[Sk Noorul Hassan]

Pls find the crash dump

[Sk Noorul Hassan]

Pls find the output of ana/err

[Sk Noorul Hassan]

Please find the clue file

[Sk Noorul Hassan]

Please find the clue file

[Volker Halle]

Hi,

it's the same old problem, isn't it ?!

See threads:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=938939

and

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=929654

An unexpected HALT on a VAX 4000-105 type of machine running V7.1, apparently running the same kind of application [AGMS.MCSYS...]*.EXE. Are these really different systems ? Or is it the same physical system booted with different node names from time to time ?

The crash (and therefore the CLUE file) does NOT contain the actual HALT PC and PSL. To capture these values, you need to record the console output from this system. Only the correct HALT-PC would allow a check, whether there really was a HALT instruction (or a binary NULL) in the instruction stream being executed.

What's unusual are the 3 non-fatal HALT bugchecks preceeding the fatal one (within a 4 minute timeframe).

The ICCS error bit is set.
CESR shows a CP1 Interrupt Vector Read Timeout

It may be hardware. See comments in the previous threads.

Volker.

[Sk Noorul Hassan]

Volker,

There 12 such servers(3 for each area) having same processes and same functionality with more or less same configuration. The only difference is no. of data points configured.

[Volker Halle]

Let's see, if we can find useful information in the RPB (Restart Parameter Block):

$ ANAL/CRASH SYS$SYSTEM:
SDA> READ SYS$SYSTEM:SYSDEF
SDA> FORM @EXE$GL_RPB/TYPE=RPB

Volker.

[Sk Noorul Hassan]

Volker,
Please find the RPB report

[Volker Halle]

The data in the RPB is inconclusive.

A short test with a program-incuded HALT restart crash under OpenVMS VAX V7.3 on a CHARON-VAX 4000-108A shows that the only reliable info about the real HALT PC is on the console terminal !

CHARON $ run halt

?06 HLT INST
PC = 00000218
Restarting system software.

**** Fatal BUG CHECK, version = V7.3 HALT, Halt instruction restart
...

The data in the crash and in the CLUE file produced afterwards do not provide the actual HALT PC. They look similar to your case, but at least the HALT code in AP (and on the INT stack) is correctly stored as 00000006 (HALT instruction executed in kernel mode).

SDA> SHOW CALL
SDA> SHOW CALL/NEXT
SDA> SHOW CALL/NEXT

at least allowed to traverse the call stack. IF the problem is indeed a software-induced HALT, this may also work in your case. Please provide the SHOW CALL data (repeat SHOW CALL/NEXT until the Return PC is in P0 space).

Consider to connect some hardcopy terminal or console manager type application to the console terminal to capture the real HALT pc and reason code.

Volker.

[Sk Noorul Hassan]

Volker, Please find the Application Error of the process which crashed the application & VMS.

[Volker Halle]

This error seems to correlate with the non-fatal HALT bugcheck immediately preceeding the system crash.

There should be another 2 such application error messages from processes DMEDIS and PPTTBF within the last 5 minutes before the crash (they could be seen in the ANAL/ERR output). Can you find them ?

This traceback data indicates, that the the instruction at PC=867C9EC0 tried to access some P1 space address (7FF89D70 - most likely a stack address, this address is also in R9) and failed. The failing PC is in MESSAGE_ROUTINES+22C0

Could this be some kind of stack overflow or underflow ?

The crash-PC reported in the crash (on the kernel stack) is 195FEB, quite near to 195D37 on top of the traceback call stack.

Can you do the following in SDA (in the dump):

SDA> READ/EXEC
SDA> EXA/INS 867C9EC0
SDA> EXA/INS 195D37
SDA> EXA/INS 195FEB

SDA> SHOW CALL
SDA> SHOW CALL/NEXT
SDA> SHOW CALL/NEXT
SDA> SHOW CALL/NEXT

You may also want to run your application processes with process dump enabled. Put a SET PROC/DUMP in their login procedures or run them with RUN/DUMP. This would cause process dump files to be written on unhandled exceptions.

Volker.

[Sk Noorul Hassan]

Hi,

Please find the SDA output asked by you

[Volker Halle]

The ACCVIO has happened at

EXE$PUTMSG+23: CMPZV #10,#0C,(R9),#01

with R9 pointing to an in-accessible address in P1 space (stack ?), which should contain the message code to be reported.
Your application seems to have called SYS$PUTMSG (from PC=005E8ED3). Saved R3 = 0001827A = %RMS-E-EOF - this could make sense.

The ACCVIO in EXE$PUTMSG seems to have triggered a call to an application condition handler (?).

00195D37: BLBS R0,00195D4D looks like a valid typical return address. Where does the call go ?

Please try:

SDA> EXA/INS 00195D37-n;20

Start with n=10 and vary n, until you get a valid instruction stream.

SDA> exa/ins 195feb
00195FEB: XFC - looks like the entry mask, please also show:

SDA> EXA/INS 195FEB;50

Volker.

[Volker Halle]

Could you also please provide the stack ranges:

SDA> EXA CTL$AL_STACK;10 ! bottom of stacks
SDA> EXA CTL$AL_STACKLIM;10 ! top of stacks

Volker.

[Sk Noorul Hassan]

Volker,
Pls find the report attached as aked by you.

[Volker Halle]

195D37 is a valid return PC from a CALLS:

SDA> exa/ins 00195d37-10;20
00195D27: BRW 00195DEF
00195D2A: PUSHL #10000002
00195D30: CALLS #01,@0019C250
00195D37: BLBS R0,00195D4D < Return PC

Where does that call go ?

SDA> EXA 19C250
SDA> EXA @19C250
SDA> EXA/INS @19C250;10

The following is the code stream from the interrupted PC found on the kernel stack:

SDA> exa/ins 195feb;50
%SDA-W-INSKIPPED, unreasonable instruction stream - 1 bytes skipped
00195FEC: HALT
00195FED: PUSHL R0
00195FEF: PUSHL R1
00195FF1: ADDL3 #10,08(AP),R1
00195FF6: JSB @#EXE$ALONONPAGED
00195FFC: BLBC R0,00196032

Please also provide the stack limits (see my previous entry).

Volker.

[Volker Halle]

This is the call frame of your application to SYS$PUTMSG:

SDA> sho call/next

Call Frame Information
----------------------
Call Frame Generated by CALLS Instruction

Condition Handler 7FF751D4 00000000
SP Align Bits = 00 7FF751D8 2FFC0000
Saved AP 7FF751DC 7FF75248
Saved FP 7FF751E0 7FF75224
Return PC 7FF751E4 005E8ED3 <
R2 7FF751E8 005D8E4C
R3 7FF751EC 0001827A
R4 7FF751F0 0000000F
R5 7FF751F4 7FF75298
R6 7FF751F8 005D8EE4
R7 7FF751FC 005D8E08
R8 7FF75200 005E8000
R9 7FF75204 00602484
R10 7FF75208 7FF75627
R11 7FF7520C 7FF7562D
Align Stack by 0 Bytes =>
Argument List 7FF75210 00000001
7FF75214 7FF75218

The MSGVEC argument starts at 7FF75218. Please examine:

SDA> EXA 7FF75218;50

Volker.

[Sk Noorul Hassan]

Volker,
Pls find the stack limit

[Sk Noorul Hassan]

Pls find the output of
SDA > exa 7FF75218;50

[Volker Halle]

The stack limits show, that the failing address in question is outside any of the user, supervisor, executive or kernel stacks.

But the MSGVEC argument to the SYS$PUTMSG call looks VERY suspicious:

The call frame to SYS$PUTMSG from your code (return address = 005E8ED3) presents ONE argument to SYS$PUTMSG - the argument list is on the stack:

Argument List: 7FF75210 00000001
______________ 7FF75214 7FF75218

The first argument to SYS$PUTMSG is the message vector MSGVEC passed by reference. So the MSGVEC starts at 7FF75218 and looks like this:

7FF75218: 7FF7562D <<< def msg options & arg count !
7FF7521C: 0001827A <<< message code = %RMS-E-EOF
7FF75220: 00000000 <<< STV value
7FF75224: 00000000
...

The argument count field, the low 16 bits of the first longword in the message vector is DEFINITELY WRONG ! It should have been 0001 ! EXE$PUTMSG 'walks' the argument vector based on the argument count. It may well walk into in-accessible address space based on that incorrect value of 562D.

Please use SDA> SHOW PROC/IMAGE to find out, which code is at address 005E8ED3, find the map and listings and the programmer (!) and check the construction of the MSGVEC for this call to SYS$PUTMSG.

Volker.

[Volker Halle]

Please also provide the data for:

SDA> EXA 19C250
SDA> EXA @19C250
SDA> EXA/INS @19C250;10

This could finally even lead us to determine the reason for the HALT

Volker.