machine crash vms 7.3.2

[otmani]

hello
i have alpha 2100 .i upgrade with vms7.3.2
i instal all new patchs but some times the machine crash .i make analyze of crash
see messages thank for your help

SDA> clue crash

Crashdump Summary Information:
------------------------------
Crash Time: 14-OCT-2005 16:41:03.70
Bugcheck Type: INVEXCEPTN, Exception while above ASTDEL
Node: THOROR (Standalone)
CPU Type: AlphaServer 2100 4/275
VMS Version: V7.3-2
Current Process: NULL
Current Image:
Failing PC: FFFFFFFF.800D9204 IOC_STD$FILSPT_C+00034
Failing PS: 10000000.00000804
Module: IO_ROUTINES (Link Date/Time: 10-AUG-2005 11:31:03.50)
Offset: 00007204

Boot Time: 6-OCT-2005 10:07:39.00
System Uptime: 8 06:33:24.70
Crash/Primary CPU: 00/00
System/CPU Type: 0902
Saved Processes: 27
Pagesize: 8 KByte (8192 bytes)
Physical Memory: 2048 MByte (262144 PFNs, contiguous memory)
Dumpfile Pagelets: 84140 blocks
Dump Flags: olddump,writecomp,errlogcomp
Dump Type: compressed,selective,shared_mem
EXE$GL_FLAGS: poolpging,init,bugdump
Paging Files: 2 Pagefiles and 1 Swapfile installed

Stack Pointers:
KSP = FFFFFFFF.83DABAD8 ESP = FFFFFFFF.83DAD000 SSP = FFFFFFFF.83DA7000
USP = FFFFFFFF.83DA7000

General Registers:
R0 = 00000000.00000000 R1 = 00000000.0000000C R2 = FFFFFFFF.83DABCD0
R3 = FFFFFFFF.810C45A0 R4 = FFFFFFFF.83DABB40 R5 = FFFFFFFF.83DABCB8
R6 = FFFFFFFF.83DABD00 R7 = 10000000.00000804 R8 = FFFFFFFF.8B9CB99A
R9 = 00000000.000000F8 R10 = 00000000.74636666 R11 = FFFFFFFF.81F31388
R12 = 00000000.7BCF79C8 R13 = FFFFFFFF.810CD7C0 R14 = 00000000.00000001
R15 = 00000000.00000000 R16 = 00000000.000001CC R17 = FFFFFFFF.83DABB40
R18 = FFFFFFFF.83DABD00 R19 = FFFFFFFF.83DABE28 R20 = FFFFFFFF.8004C33C
R21 = 00000000.00000002 R22 = 00000000.6E6F4320 R23 = 00000000.00000000
R24 = 00000000.00000001 AI = 00000000.00000001 RA = FFFFFFFF.800D8774
PV = FFFFFFFF.810CD7C0 R28 = 00000000.3A746E65 FP = FFFFFFFF.83DABD50
PC = FFFFFFFF.8009F1E8 PS = 18000000.00000804

Exception Frame:
R2 = 00000000.000000F8 R3 = FFFFFFFF.81F4CD80 R4 = FFFFFFFF.81F31140
R5 = FFFFFFFF.819C3880 R6 = 00000000.00000000 R7 = FFFFFFFF.81F31140
PC = FFFFFFFF.800D9204 PS = 10000000.00000804

Signal Array: 64-bit Signal Array:
Arg Count = 00000005 Arg Count = 00000005
Condition = 0000000C Condition = 00000000.0000000C
Argument #2 = 00000000 Argument #2 = 00000000.00000000
Argument #3 = 3A746E65 Argument #3 = 00000000.3A746E65
Argument #4 = 800D9204 Argument #4 = FFFFFFFF.800D9204
Argument #5 = 00000804 Argument #5 = 10000000.00000804

Mechanism Array:
Arguments = 0000002C Establisher FP = FFFFFFFF.83DABD50
Flags = 00000000 Exception FP = FFFFFFFF.83DABD00
Depth = FFFFFFFD Signal Array = FFFFFFFF.83DABCB8
Handler Data = FFFFFFFF.819C3F90 Signal64 Array = FFFFFFFF.83DABCD0
R0 = FFFFFFFF.81F31388 R1 = FFFFFFFF.81F31388 R16 = FFFFFFFF.819C3880
R17 = 00000000.000000F8 R18 = FFFFFFFF.819C3880 R19 = FFFFFFFF.83DABE28
R20 = FFFFFFFF.8004C33C R21 = 00000000.00000002 R22 = 00000000.6E6F4320
R23 = 00000000.00000000 R24 = 00000000.00000001 R25 = 00000000.00000001
R26 = FFFFFFFF.800D8774 R27 = FFFFFFFF.810CD7C0 R28 = 00000000.3A746E65

System Registers:
Page Table Base Register (PTBR) 00000000.0003FFF8
Processor Base Register (PRBR) FFFFFFFF.81806000
Privileged Context Block Base (PCBB) 00000000.01406080
System Control Block Base (SCBB) 00000000.00000C6E
Software Interrupt Summary Register (SISR) 00000000.00000100
Address Space Number (ASN) 00000000.00000000
AST Summary / AST Enable (ASTSR_ASTEN) 00000000.00000000
Floating-Point Enable (FEN) 00000000.00000000
Interrupt Priority Level (IPL) 00000000.00000008
Machine Check Error Summary (MCES) 00000000.00000000
Virtual Page Table Base Register (VPTB) FFFFFEFC.00000000

Failing Instruction:
IOC_STD$FILSPT_C+00034: LDQ R3,(R28)

Instruction Stream (last 20 instructions):
IOC$FILSPT_C+00014: BIS R31,#X01,R25
IOC$FILSPT_C+00018: ADDQ R13,#X10,R27
IOC$FILSPT_C+0001C: BSR R26,#X000004
IOC$FILSPT_C+00020: LDQ R28,(SP)
IOC$FILSPT_C+00024: LDQ R13,#X0008(SP)
IOC$FILSPT_C+00028: ADDQ SP,#X10,SP
IOC$FILSPT_C+0002C: RET R31,(R28)
IOC_STD$FILSPT_C: SUBQ SP,#X40,SP
IOC_STD$FILSPT_C+00004: STQ R27,(SP)
IOC_STD$FILSPT_C+00008: STQ R26,#X0010(SP)
IOC_STD$FILSPT_C+0000C: STQ R2,#X0018(SP)
IOC_STD$FILSPT_C+00010: STQ R3,#X0020(SP)
IOC_STD$FILSPT_C+00014: STQ R5,#X0028(SP)
IOC_STD$FILSPT_C+00018: STQ R13,#X0030(SP)
IOC_STD$FILSPT_C+0001C: STQ FP,#X0038(SP)
IOC_STD$FILSPT_C+00020: STQ R31,#X0008(SP)
IOC_STD$FILSPT_C+00024: BIS R31,R27,R13
IOC_STD$FILSPT_C+00028: BIS R31,SP,FP
IOC_STD$FILSPT_C+0002C: ADDL R31,R16,R5
IOC_STD$FILSPT_C+00030: LDL R28,#X00B0(R5)
IOC_STD$FILSPT_C+00034: LDQ R3,(R28)
IOC_STD$FILSPT_C+00038: BLBC R3,#X000039
IOC_STD$FILSPT_C+0003C: SRL R3,#X20,R3
IOC_STD$FILSPT_C+00040: LDQ R27,#XFE30(R13)
IOC_STD$FILSPT_C+00044: LDAH R25,#X0001(R31)

[Volker Halle]

Otmani,

the system crashed, because the address in R28 - as loaded from #X00B0(R5) - is not a valid address.

Use SDA to find out what R5 is pointing to:

$ ANAL/CRASH sys$system:sysdump.dmp
SDA> FORM 819C3880

Consider to post the output of SDA> CLUE REGISTER, as this will automatically determine the types of data structures the registers are pointing to. This would provide more context information about the crash.

Volker.

[Volker Halle]

R5 should point to a UCB (Unit Control Block)

#X00B0(R5) is UCB$L_SVAPTE(R5) and should contain a system space address of a PTE, but seems to contain ASCII text:

3A746E65 = 'ent:'

So it looks like the UCB pointed to by R5 or lat least the UCB$L_SVAPTE field has been corrupted/overwritten...

Try to look at the UCB with SDA> EXA 819C3880;200 and see, whether a corruption is 'visible'.

Next step would be to find out, if R5 really points to a (at least partially) valid UCB and what kind of device is involved. SDA> CLUE REGISTER would provide this information.

Volker.

[Arch_Muthiah]

Hi,

> IOC_STD$FILSPT_C+00030: LDL R28,#X00B0(R5)
> IOC_STD$FILSPT_C+00034: LDQ R3,(R28)

The LDL instn moves the contents of UCB (R5) to R28. So R28 will have value stored in 3A746E65 location. As Volker said, the value in R28 is 'ent'. I hope this is ok.

As my understanding when we call IOC_STD$FILSPT, it reads UCB$L_SVAPTE to obtain the system virtual address of PTE that maps the first page of the buffer.

This routine returns system virtual address of the first byte of the above mapped page that contains the buffer. This returned value usually be in R0. My suggestion would be checking the value in R0. It should contain Virtual address.

If it does not, then the IO routine ($QIO or $QIOW) which initiated the above IOC_STD$FILSPT call, still holding the lock (may be AST or event flag has not been released), so the instruction IOC_STD$FILSPT_C+00034: LDQ R3,(R28) has failed.

So I would suggest to check any IO routines (probably $QIO ro any fork) used in the programs should be checked for proper synchronization using $synch, $schdwkup rouitnes.

Archunan

[Volker Halle]

re: Archunan,

this crash cannot be caused by user-mode code. Even if a program would use bad synchronization or invalid parameters to a $QIO call, it's the job of the operating system to check and reject invalid parameters from user mode. There is absolutely no need to check user-mode code here...

The first key to the problem is finding the device pointed to by the UCB in R5. Then look at the type of corruption in the UCB. Is it just UCB$L_SVAPTE overwritten by 'ent:' or is more of the UCB corrupted ?

Volker.

[Arch_Muthiah]

Hi Volker,

We clearly knew that all language IOs (fopen-fread-fwrite of C, open-read-write of 4tran, cobol and all socket routines) intern calls either QIO or RMS routines, these routines finally stepped out of Executive mode to Kernel layer and intern access kernel IO_routines.

In the same way all user mode DCL commands (ex: f$getsyi, f$getjpi, f$getdev, f$getqui ...) are passing thru super, and executive mode (using equivalent sys$getsyi, sys$getjpi, sys$getdev, sys$getqui rouines) and finally using kernel routines and accessing either unit files and device files.

This is clear....

How?, we knew that a user process with CMEXEC, CMKCRNL, PHY_IO,....privileges can execute user executive iamges (with or without AST).

This way only we are usually allocating a channel to most of the devices using SYS$ASSIGN, or LIB$get_xxxx routines. And IO$_SENSE and IO$_SET_ting mode for the device, then IO$_Reading and IO$_Writing using $QIO routine. This intern access kernel IO_ routines.

So a user process can read, write, and lock the physical memory locations with proper privilleges set.

As I said in my earlier posting, I still doubt that the bugcheck INVEXCEPTN, Exception while above ASTDEL" may be occured by lock conflict.

How?. A user process can lock any phy mem (non-paged mem and system space - driver s/w, and UCB data structure can be loaded).

Now incase, if any process (ex:SYSMAN) or kernel tries to execute or load device driver image (driver.exe), which overlapes the locked area (32 or 64 bits), then there is chance of getting this bugcheck INVEXCEPTN. Becuase some of the device interface driver can not return ACCESS voilation or NOPRIV status, instead it will endup with bugcheck INVEXCEPTN crash.

otmani:
So as per Mr.Volker suggestion, better we can find what is the device driver involved in this bugcheck. If have DECEvent software installed in your OVMS, you can find the all the sequence event occured before the system crash.


Archunan

[otmani]

hello wolker
thank for your help see messages after commands on sda
SDA> FORM 819C3880
FFFFFFFF.819C3880 UCB$L_FQFL 8126DD20 X25$ACCESS+B7920
UCB$L_MB_MSGQFL
UCB$L_RQFL
UCB$W_MB_SEED
UCB$W_UNIT_SEED
FFFFFFFF.819C3884 UCB$L_FQBL 81806038 CPUDB+00038
UCB$L_MB_MSGQBL
UCB$L_RQBL
FFFFFFFF.819C3888 UCB$W_SIZE 0140
FFFFFFFF.819C388A UCB$B_TYPE 10
FFFFFFFF.819C388B UCB$B_FLCK 3A
FFFFFFFF.819C388C UCB$L_ASTQFL 812831D8 X25$NPDRIVER+045D8
UCB$L_FPC
UCB$L_MB_W_AST
UCB$T_PARTNER
FFFFFFFF.819C3890 UCB$L_ASTQBL 81F4CD80
UCB$L_FR3
UCB$L_MB_R_AST
UCB$Q_FR3
FFFFFFFF.819C3894 UCB$PS_DYNLCK FFFFFFFF
FFFFFFFF.819C3898 UCB$L_FIRST 819C3880
UCB$Q_FR4
UCB$W_MSGMAX
UCB$W_MSGCNT
FFFFFFFF.819C389C FFFFFFFF
FFFFFFFF.819C38A0 UCB$W_BUFQUO 0000
UCB$W_DSTADDR
FFFFFFFF.819C38A2 UCB$W_INIQUO 0000
UCB$W_SRCADDR
FFFFFFFF.819C38A4 UCB$L_ORB 819A2200
FFFFFFFF.819C38A8 UCB$L_CPID 00000000
UCB$L_LOCKID
FFFFFFFF.819C38AC UCB$PS_CRAM 00000000
FFFFFFFF.819C38B0 UCB$L_CRB 819A1E00
FFFFFFFF.819C38B4 UCB$L_DLCK 819A0CC0
FFFFFFFF.819C38B8 UCB$L_DDB 819A2B00
FFFFFFFF.819C38BC UCB$L_PID 00010017
FFFFFFFF.819C38C0 UCB$L_LINK 00000000
FFFFFFFF.819C38C4 UCB$L_VCB 00000000
FFFFFFFF.819C38C8 UCB$L_DEVCHAR 0C060001
UCB$Q_DEVCHAR
FFFFFFFF.819C38CC UCB$L_DEVCHAR2 00000000
FFFFFFFF.819C38D0 UCB$L_AFFINITY FFFFFFFF
FFFFFFFF.819C38D4 UCB$L_ALTIOWQ 00000000
UCB$L_XTRA
FFFFFFFF.819C38D8 UCB$B_DEVCLASS C8
FFFFFFFF.819C38D9 UCB$B_DEVTYPE 00
FFFFFFFF.819C38DA UCB$W_DEVBUFSIZ FA00
FFFFFFFF.819C38DC UCB$B_LOCSRV 00
UCB$B_SECTORS
UCB$L_DEVDEPEND
UCB$Q_DEVDEPEND
UCB$R_DEVDEPEND_Q_BLOCK
UCB$R_DISK_DEVDEPEND
UCB$R_NET_DEVDEPEND
UCB$R_TERM_DEVDEPEND
FFFFFFFF.819C38DD UCB$B_REMSRV 00
UCB$B_TRACKS
FFFFFFFF.819C38DE UCB$W_BYTESTOGO 0000
UCB$W_CYLINDERS
UCB$B_VERTSZ
FFFFFFFF.819C38E0 UCB$L_DEVDEPND2 00000000
UCB$L_TT_DEVDP1
UCB$W_TU_FORMENU
FFFFFFFF.819C38E4 UCB$L_DEVDEPND3 00000000
UCB$Q_DEVDEPEND2
UCB$R_DEVDEPEND2_Q_BLOCK
UCB$R_TMV_BCNT
UCB$W_TMV_BCNT1
UCB$W_TMV_BCNT2
FFFFFFFF.819C38E8 UCB$L_DEVDEPND4 00000000
UCB$W_TMV_BCNT3
UCB$W_TMV_BCNT4
FFFFFFFF.819C38EC UCB$L_IOQFL 819C38EC
FFFFFFFF.819C38F0 UCB$L_IOQBL 819C38EC
FFFFFFFF.819C38F4 UCB$W_UNIT 0000
FFFFFFFF.819C38F6 UCB$B_CM1 00
UCB$W_CHARGE
UCB$W_RWAITCNT
FFFFFFFF.819C38F7 UCB$B_CM2 00
FFFFFFFF.819C38F8 UCB$L_IRP 81F4CD80
FFFFFFFF.819C38FC UCB$L_REFC 00000001
FFFFFFFF.819C3900 UCB$B_DIPL 00
UCB$B_STATE
FFFFFFFF.819C3901 UCB$B_AMOD 00
FFFFFFFF.819C3902 UCB$W_FILL_0 0000
FFFFFFFF.819C3904 UCB$L_AMB 00000000
FFFFFFFF.819C3908 UCB$L_STS 00000110
FFFFFFFF.819C390C UCB$L_DEVSTS 00000000
FFFFFFFF.819C3910 UCB$L_QLEN 00000001
FFFFFFFF.819C3914 UCB$PS_START_AFF_QFL 00000000
FFFFFFFF.819C3918 UCB$PS_START_AFF_QBL 00000000
FFFFFFFF.819C391C UCB$L_PORT_CPUDB 00000000
FFFFFFFF.819C3920 UCB$PS_IO_COUNTERS 00000000
FFFFFFFF.819C3924 UCB$L_DUETIM 00000000
FFFFFFFF.819C3928 UCB$L_OPCNT 0000019D
FFFFFFFF.819C392C UCB$L_SVPN 00001EDB
FFFFFFFF.819C3930 UCB$L_SVAPTE 3A746E65
FFFFFFFF.819C3934 UCB$L_BCNT 00000E20
FFFFFFFF.819C3938 UCB$L_BOFF 7463656E
FFFFFFFF.819C393C UCB$L_SOFTERRCNT 00000000
FFFFFFFF.819C3940 UCB$L_ERTCNT 00000000
FFFFFFFF.819C3944 UCB$L_ERTMAX 00000000
FFFFFFFF.819C3948 UCB$L_ERRCNT 00000000
FFFFFFFF.819C394C UCB$L_PDT 00000000
FFFFFFFF.819C3950 UCB$L_DDT 81282DA0 X25$NPDRIVER+041A0
FFFFFFFF.819C3954 UCB$PS_ADP 00000000
FFFFFFFF.819C3958 UCB$PS_CRCTX 00000000
FFFFFFFF.819C395C UCB$L_MEDIA_ID 00000000
FFFFFFFF.819C3960 UCB$PS_DTN 00000000
FFFFFFFF.819C3964 UCB$PS_DTN_LINK 00000000
FFFFFFFF.819C3968 UCB$PS_TOUTROUT 00000000
FFFFFFFF.819C396C UCB$PS_SUD 8199B380
UCB$C_LENGTH

------------------------------------------------------------------------------------------------------------------------------------------------------------------

SDA> EXA 819C3880;200
FFFFFFFF 819C3880 FFFFFFFF 81F4CD80 812831D8 3A100140 81806038 8126DD20 à &.8`..@..:à 1(..à ô......8...... FFFFFFFF.819C3880
00010017 819A2B00 819A0CC0 819A1E00 00000000 00000000 819A2200 00000000 ....."..............Ã ....+...... FFFFFFFF.819C38A0
00000000 FA0000C8 00000000 FFFFFFFF 00000000 0C060001 00000000 00000000 ........................à ..ú.... FFFFFFFF.819C38C0
00000001 81F4CD80 00000000 819C38EC 819C38EC 00000000 00000000 00000000 ............ì8..ì8.......à ô..... FFFFFFFF.819C38E0
00000000 00000000 00000000 00000001 00000000 00000110 00000000 00000000 ................................ FFFFFFFF.819C3900
00000000 7463656E 00000E20 3A746E65 00001EDB 0000019D 00000000 00000000 ............Ã ...ent: ...nect.... FFFFFFFF.819C3920
00000000 00000000 00000000 81282DA0 00000000 00000000 00000000 00000000 .................-(............. FFFFFFFF.819C3940
00000000 00000000 00000000 00000000 8199B380 00000000 00000000 00000000 .............³.................. FFFFFFFF.819C3960
00000001 00000000 0000019D 0000019D 00000000 00000000 00000000 819AB740 @·.............................. FFFFFFFF.819C3980
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 ................................ FFFFFFFF.819C39A0
53454741 4D495F45 4C424144 414F4C24 53595324 0A620140 81901CC0 819C3740 @7..Ã ...@.b.$SYS$LOADABLE_IMAGES FFFFFFFF.819C39C0
00019E90 0000000D 00000000 00000045 58452E52 45564952 44564E24 3532583A :X25$NVDRIVER.EXE............... FFFFFFFF.819C39E0
0000FFFF 00000000 00000003 0000F400 806CC000 00000000 00A2F940 2881B7A0 .·.(@ù¢......à l..ô.............. FFFFFFFF.819C3A00
00000000 00000000 00000000 00013FFF 00010000 0000007D 00002C00 81284000 .@(..,..}........?.............. FFFFFFFF.819C3A20
00000000 0001BFFF 00018000 00000095 00000200 81286C00 FFFFFFFF FFFFFFFF .........l(..............¿...... FFFFFFFF.819C3A40
00014000 00000000 00000000 00000000 0001FFFF 0001C000 00000000 00000000 .........Ã ...................@.. FFFFFFFF.819C3A60
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

SDA> CLUE REGISTER
Current Registers: Process index: 0000 Process name: NULL PCB: 810D81C8 (CPU 0)
---------------------------------------------------------------------------------------
R0 = FFFFFFFF.81F31388
R1 = FFFFFFFF.81F31388
R2 = 00000000.000000F8
R3 = FFFFFFFF.81F4CD80 BUFIO
R4 = FFFFFFFF.81F31140 VCRP
R5 = FFFFFFFF.819C3880 UCB (Device NPA0:)
R6 = 00000000.00000000
R7 = FFFFFFFF.81F31140 VCRP
R8 = FFFFFFFF.8B9CB99A
R9 = 00000000.000000F8
R10 = 00000000.74636666
R11 = FFFFFFFF.81F31388
R12 = 00000000.7BCF79C8
R13 = FFFFFFFF.810CD7C0 IOC_STD$FILSPT
R14 = 00000000.00000001
R15 = 00000000.00000000
R16 = FFFFFFFF.819C3880 UCB (Device NPA0:)
R17 = 00000000.000000F8
R18 = FFFFFFFF.819C3880 UCB (Device NPA0:)
R19 = FFFFFFFF.83DABE28
R20 = FFFFFFFF.8004C33C EXE_STD$IOFORK_CPU_C+0047C
R21 = 00000000.00000002
R22 = 00000000.6E6F4320
R23 = 00000000.00000000
R24 = 00000000.00000001
AI = 00000000.00000001
RA = FFFFFFFF.800D8774 IOC_STD$INITBUFWIND_C+00044
PV = FFFFFFFF.810CD7C0 IOC_STD$FILSPT
R28 = 00000000.3A746E65
FP = FFFFFFFF.83DABD50
PC = FFFFFFFF.800D9204 IOC_STD$FILSPT_C+00034
PS = 10000000.00000804 Kernel Mode, IPL 8, Interrupt
SDA>

[Volker Halle]

Otmani,

at least 2 fields in the UCB looks suspicious:

FFFFFFFF.819C3930 UCB$L_SVAPTE 3A746E65
FFFFFFFF.819C3938 UCB$L_BOFF 7463656E

NPA0: seems to be a X25 pseudo device. Did you upgrade the X25 product together with the OpenVMS upgrade to V7.3-2 ?

Which version of X25 is installed ?

$ PROD SHOW HIST X25*

X.25 V1.6 ECO 2 seems to be the current available version.

Can you somehow correlate the crashes with any usage of X.25 ?

Volker.

[otmani]

hi volker

i verify the patch of x25

$ prod show hist x25*

dec axpvms x25 v1.6-2 install 12-may-2005

so i have x25 v1.6 eco2

thank for you help
regards

[Volker Halle]

Otmani,

then it's quite likely that there may be a software problem in the HP X.25 software. Did you have multiple crashes on this machine after the upgrade to V7.3-2 ?

$ TYPE CLUE$HISTORY

will print a 1 line summary per crash.

For each crash, there will be a CLUE file in CLUE$COLLECT:CLUE$THOROR_ddmmyy_hhmm.LIS with more detailled information from the crash. Are all crashes the same: bugcheck, module, offset ?

With the same basic information you've supplied for this crash, we may be able to find similarities with the other crashes. You could either mail me the CLUE files (look at my ITRC profile) or attach them in a .ZIP archive.

Volker.