VAx crash 3100

[vmsserbo]

We have a VAx 3100 runninmg OPenvms V6.2

I ran ANAL/CRASH_DUMP SYS$SYSTEM:SYSDUMP.DMP

Dump taken on 31-MAY-2006 15:04:05.24
MACHINECHK, Machine check while in kernel mode

SDA> show crash
System crash information
------------------------
Time of system crash: 31-MAY-2006 15:04:05.24


Version of system: OpenVMS (TM) VAX Version V6.2

System Version Major ID/Minor ID: 1/0


System type: MicroVAX 3100

Crash CPU ID/Primary CPU ID: 00/00

Bitmask of CPUs active/available: 00000001/00000001


CPU bugcheck codes:
CPU 00 -- MACHINECHK, Machine check while in kernel mode
CPU 00 Processor crash information
----------------------------------


CPU 00 reason for Bugcheck: MACHINECHK, Machine check while in kernel mode


Process currently executing on this CPU: SWAPPER


Current IPL: 31 (decimal)


CPU database address: 81D7C000
CPU 00 Processor crash information
----------------------------------

General registers:

R0 = 0A002F54 R1 = 809D1C00 R2 = 00003E00 R3 = 80AF3400
R4 = 00000000 R5 = 000000CB R6 = 00003E00 R7 = 00000000
R8 = FFFFFFFF R9 = 00000000 R10 = 00000000 R11 = 80BBA804
AP = 00000000 FP = 00000000 SP = 81D7DDE4 PC = 811540C0
PSL = 041F0008
CPU 00 Processor crash information
----------------------------------
Processor registers:


P0BR = 80AA4A00 SBR = 0137D600 ASTLVL = 00000004
P0LR = 00003000 SLR = 00019800 SISR = 00000000
P1BR = 801EDC00 PCBB = 00005620 ICCS = 00000040
P1LR = 001FFF7F SCBB = 0136C000 SID = 0A000006

TODR = 5D922A4B CADR = 000000FC
CACR = FFEBFF54 MSER = 000000E0

ISP = 81D7DDE4
KSP = 80AD1FE0
ESP = 00000000
SSP = 00000000
USP = 00000000
CPU 00 Processor crash information
----------------------------------

No spinlocks currently owned by CPU 00

How do I know what caused the crash???

Any other commands I need to perform to checj for crash??

Thanks!

[Volker Halle]

Miles,

a MACHINECHK is most likely a hardware problem.

First try to run ANAL/ERR/SIN=31-MAY-2006:14:00 - errlog entries from the dumpfile should have been added into ERRLOG.SYS during boot.

You could also extract the error log entries directly from the dumpfile itself with the following commands:

$ CLUE:==$CLUE
$ CLUE /ERROR_LOGS
$ ANAL/ERROR CLUE_ELOG.SYS
or
$ DIAGNOSE CLUE_ELOG.SYS ! using DECevent V3.4

Look for HW errors immediately preceeding the crash.

Consider to post the full CLUE file (to be found in CLUE$OUTPUT:CLUE$LAST_node.LIS) as a .TXT attachment in a further reply. Posting the output of an OpenVMS VAX SDA> SHOW CRASH command does not give lots of information needed for analysis.

Volker.

[vmsserbo]

Please give me the actual commands. I did a search for errlog.sys and found it in this directory

dir GREENS$DKB0:[000000...]errlog.sys

Directory GREENS$DKB0:[SYS0.SYSERR]

ERRLOG.SYS;1 1751/1752 20-NOV-2004 01:40:34.55

I also did a
ANAL/ERR/SIN=31-MAY-2006:14:00
Error Log Report Generator Version V6.1
******************************* ENTRY 10606. *******************************
ERROR SEQUENCE 1169. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 14:53:35.52 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:33:52
SCS NODE: GREENS VAX/VMS V6.2

TIME STAMP KA420 CPU FW REV# 7.
******************************* ENTRY 10607. *******************************
ERROR SEQUENCE 1170. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:03:35.53 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:43:52
SCS NODE: GREENS VAX/VMS V6.2

TIME STAMP KA420 CPU FW REV# 7.
******************************* ENTRY 10608. *******************************
ERROR SEQUENCE 1171. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.23 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

MACHINE CHECK KA420 CPU FW REV# 7.

MCHECK CODE 00000080
ERR DURING OPERAND OR P0/P1 PTE RD
VA 809D4B54
MOST RECENT VIRTUAL ADDRESS
INTERNAL STATE1 2CE0FCE0
MCHECK: CDAL PE
CDAL PE
1ST LEVEL CACHE MISS
1ST LEVEL CACHE STATUS:
_D STREAM ENABLED
_I STREAM ENABLED
_SET 1 ENABLED
_SET 2 ENABLED
HIGHEST SW INTER REQ = 0(X)
OPCODE = MOVC5
INTERNAL STATE2 54CBF10A
PC OFFSET = 0A(X)
ALU CONDITION CODES = 1(X)
VAX CANT RESTART FLAG = 1.
PC 8115BAEF
ERROR PSL 08080004
Z-BIT
INTERRUPT PRIORITY LEVEL = 08.
PREVIOUS MODE = KERNEL
CURRENT MODE = KERNEL
FIRST PART DONE SET
******************************* ENTRY 10609. *******************************
ERROR SEQUENCE 1172. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.23 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

CACHE/BUS ERROR KA420 CPU FW REV# 7.

MSER 000000E0
MCHECK: CDAL PE
CDAL PE
1ST LEVEL CACHE MISS
CADR 000000FC
1ST LEVEL CACHE STATUS:
_D STREAM ENABLED
_I STREAM ENABLED
_SET 1 ENABLED
_SET 2 ENABLED
CACR FFEBFF44
2ND LEVEL CACHE STATUS:
_DISABLED
CYCLE SPEED CODE = 1.
CACHE FLAGS 0000003F
1ST LEVEL CACHE STATUS:
_SET 1 ENABLED BEFORE ERROR
_SET 1 ENABLED AFTER ERROR
_SET 2 ENABLED BEFORE ERROR
_SET 2 ENABLED AFTER ERROR
2ND LEVEL CACHE STATUS:
_ENABLED BEFORE ERROR
_ENABLED AFTER ERROR
******************************* ENTRY 10610. *******************************
ERROR SEQUENCE 1173. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.24 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

MACHINE CHECK KA420 CPU FW REV# 7.

MCHECK CODE 00000080
ERR DURING OPERAND OR P0/P1 PTE RD
VA 809D4B54
MOST RECENT VIRTUAL ADDRESS
INTERNAL STATE1 2CE0FCE0
MCHECK: CDAL PE
CDAL PE
1ST LEVEL CACHE MISS
1ST LEVEL CACHE STATUS:
_D STREAM ENABLED
_I STREAM ENABLED
_SET 1 ENABLED
_SET 2 ENABLED
HIGHEST SW INTER REQ = 0(X)
OPCODE = MOVC5
INTERNAL STATE2 54CBF10A
PC OFFSET = 0A(X)
ALU CONDITION CODES = 1(X)
VAX CANT RESTART FLAG = 1.
PC 8115BAEF
ERROR PSL 08080004
Z-BIT
INTERRUPT PRIORITY LEVEL = 08.
PREVIOUS MODE = KERNEL
CURRENT MODE = KERNEL
FIRST PART DONE SET
******************************* ENTRY 10611. *******************************
ERROR SEQUENCE 1174. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.24 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

CACHE/BUS ERROR KA420 CPU FW REV# 7.

MSER 000000E0
MCHECK: CDAL PE
CDAL PE
1ST LEVEL CACHE MISS
CADR 000000FC
1ST LEVEL CACHE STATUS:
_D STREAM ENABLED
_I STREAM ENABLED
_SET 1 ENABLED
_SET 2 ENABLED
CACR FFEBFF44
2ND LEVEL CACHE STATUS:
_DISABLED
CYCLE SPEED CODE = 1.
CACHE FLAGS 0000003F
1ST LEVEL CACHE STATUS:
_SET 1 ENABLED BEFORE ERROR
_SET 1 ENABLED AFTER ERROR
_SET 2 ENABLED BEFORE ERROR
_SET 2 ENABLED AFTER ERROR
2ND LEVEL CACHE STATUS:
_ENABLED BEFORE ERROR
_ENABLED AFTER ERROR
******************************* ENTRY 10612. *******************************
ERROR SEQUENCE 1175. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.24 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

UNCORRECTABLE MEMORY ERROR KA420 CPU FW REV# 7.

MSER 000000E0
MCHECK: CDAL PE
CDAL PE
1ST LEVEL CACHE MISS
CADR 000000FC
1ST LEVEL CACHE STATUS:
_D STREAM ENABLED
_I STREAM ENABLED
_SET 1 ENABLED
_SET 2 ENABLED
CACR AFEBFF54
2ND LEVEL CACHE STATUS:
_ENABLED
CYCLE SPEED CODE = 1.
PHYSICAL ADDR 00D0C950
******************************* ENTRY 10613. *******************************
ERROR SEQUENCE 1176. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:05.24 SYS_TYPE 04010201
SYSTEM UPTIME: 8 DAYS 00:44:22
SCS NODE: GREENS VAX/VMS V6.2

FATAL BUGCHECK KA420 CPU FW REV# 7.

MACHINECHK, Machine check while in kernel mode

PROCESS NAME SWA

PROCESS ID 00010001

ERROR PC 811540C0
ERROR PSL 041F0008
N-BIT
INTERRUPT PRIORITY LEVEL = 31.
PREVIOUS MODE = KERNEL
CURRENT MODE = KERNEL
INTERRUPT STACK
FIRST PART DONE CLEAR

STACK POINTERS

KSP 80AD1FE0 ESP 00000000 SSP 00000000 USP 00000000 ISP 81D7DD68

GENERAL REGISTERS

R0 0A002F54 R1 809D1C00 R2 00003E00 R3 80AF3400 R4 00000000
R5 000000CB R6 00003E00 R7 00000000 R8 FFFFFFFF R9 00000000
R10 00000000 R11 80BBA804 AP 00000000 FP 00000000 SP 81D7DDDC

SYSTEM REGISTERS

P0BR 80AA4A00
P0 PTE BASE (VIRT ADDRS)
P0LR 00003000
TOTAL P0 PAGES
P1BR 801EDC00
P1 PTE BASE (VIRT ADDRS)
P1LR 001FFF7F
TOTAL NON-EXISTENT P1 PAGES
SBR 0137D600
SYSTEM PTE BASE (PHYS ADDRS)
SLR 00019800
TOTAL PAGES "SYSTEM" VIRT MEM
PCBB 00005620
PCB BASE (PHYS ADDRS)
SCBB 0136C000
SCB BASE (PHYS ADDRS)
ASTLVL 00000004
NO AST'S PENDING
SISR 00000000
INTERRUPT REQUEST ACTIVE = 0.
ICCS 00000040
INTERRUPT ENABLE
TODR 00000000
******************************* ENTRY 10614. *******************************
ERROR SEQUENCE 1177. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:54.23 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 00:00:02
SCS NODE: GREENS VAX/VMS V6.2

SYSTEM START-UP KA420 CPU FW REV# 7.

TIME OF DAY CLOCK 5D923DD3
******************************* ENTRY 10615. *******************************
ERROR SEQUENCE 1178. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:04:55.09 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 00:00:03
SCS NODE: GREENS VAX/VMS V6.2

MOUNT VOLUME KA420 CPU FW REV# 7.

UNIT _GREENS$DKB0:, VOLUME LABEL "VAXVMS62"

354. QIO OPERATIONS THIS UNIT, 0. ERRORS THIS UNIT
******************************* ENTRY 10616. *******************************
ERROR SEQUENCE 1179. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:06:08.86 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 00:01:17
SCS NODE: GREENS VAX/VMS V6.2

MOUNT VOLUME KA420 CPU FW REV# 7.

UNIT _GREENS$DKB100:, VOLUME LABEL "PLAT1"

20. QIO OPERATIONS THIS UNIT, 0. ERRORS THIS UNIT
******************************* ENTRY 10617. *******************************
ERROR SEQUENCE 1180. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:06:29.70 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 00:01:37
SCS NODE: GREENS VAX/VMS V6.2

MOUNT VOLUME KA420 CPU FW REV# 7.

UNIT _GREENS$DKB200:, VOLUME LABEL "PLAT2"

25. QIO OPERATIONS THIS UNIT, 0. ERRORS THIS UNIT
******************************* ENTRY 10618. *******************************
ERROR SEQUENCE 1181. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 15:06:50.28 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 00:01:58
SCS NODE: GREENS VAX/VMS V6.2

MOUNT VOLUME KA420 CPU FW REV# 7.

UNIT _GREENS$DKB300:, VOLUME LABEL "PLAT3"

25. QIO OPERATIONS THIS UNIT, 0. ERRORS THIS UNIT
******************************* ENTRY 10619. *******************************
ERROR SEQUENCE 1187. LOGGED ON: SID 0A000006
DATE/TIME 31-MAY-2006 16:07:43.72 SYS_TYPE 04010201
SYSTEM UPTIME: 0 DAYS 01:02:50
SCS NODE: GREENS VAX/VMS V6.2

TIME STAMP KA420 CPU FW REV# 7.
GREENS>


So what am I looking for??

[vmsserbo]

I did a anal/err/sinc=tod sys$errorlog

UNCORRECTABLE MEMORY ERROR KA420 CPU FW REV# 7.

What should I do, should I monitor it or place a call??

[vmsserbo]

Here is the error log in an attachment.

[Steven Schweda · Accepted solution]

> What should I do, should I monitor it or
> place a call??

Hard to say. What's a call cost? Does it
crash frequently? How much do you care?

It sure looks like a memory error, which
could be just bad luck, or a sign of things
to come.

I assume that someone with some knowledge
could translate the "PHYSICAL ADDR 00D0C950"
into a module location.

You could wait for the next crash, and while
it's down, jiggle all the memory.

[Steven Schweda]

If my unreliable memory may be trusted,
these systems used parity memory, not ECC,
so any detectable memory error would be an
"UNCORRECTABLE MEMORY ERROR".

If yours has ECC, then this error was worse
than a single-bit quirk.

[Allan Bowman]

Miles,

If you have a service contract, I would call them out to replace what is most likely a memory card going bad. If no contract, I would monitor the system and watch for any more errors on the same card. You can usually find replacement memory on the auction sites - the biggest issue would be matching exactly what you currently have. If you run into a bind down the road, let me know and I'll see what I can dig up in my basement full of MicroVax systems (3100, 3100-30, 3100-40, 3100-80, 3100-85).

Allan in Atlanta

[Volker Halle]

Miles,

CDAL PE errors can be caused by defects in either cache or memory (or contact/seating problems). You could search google groups for 'cdal pe vax memory error KA420' to see some old discussion about these type of errors.

You could watch these errors with $ SHOW ERROR and ANAL/ERR/SINCE each day. Note the PHYSICAL ADDR values reported, the VA, the PC and the instruction - compare these values from all the failures to try to establish a pattern. If these errors occur in KERNEL MODE, they will take down your system, if they occur in any other mode, they may just cause the process to be deleted.

Volker.

[vmsserbo]

Thanks everyone for all your support! Much appreciated!