INVEXCEPTN, Exception while above ASTDEL

[Vladimir Fabecic]

Hello
Yesterday I had a crash of ES47.
Output of SDA>CLUE CRASH is attached.
Did anyone have simular problem?
Can it be problem with TCPIP?
OS version is 8.2 with newest patches (included TCPIP).
Regards.

[Volker Halle]

Vladimir,

this is most likely a problem in TCPIP.

LDL R2,#X0030(R0)
BIS R31,#X01,R25
BEQ R2,#X00000B
LDQ R26,#X0008(R2)

R2 should have been loaded from #x30(R0) and contains an invalid S0 space address: A86A0A57 - access to #x08(r2) then fails with an ACCVIO.

To confirm this, have a look at

SDA> EXA 82430600+30
SDA> READ SYSDEF
SDA> FORMAT 82430600

This could also be generic pool corruption, but the output of the FORAMT command may tell more...

Volker.

[Vladimir Fabecic]

Thanks Valker
Here is the output:

Dump taken on 28-NOV-2006 14:55:21.42
INVEXCEPTN, Exception while above ASTDEL

SDA> EXA 82430600+30
FFFFFFFF.82430630: 00A5F685.A86A0A57 "W.j¨.ö¼."
SDA> read sysdef
SDA> format 82430600
FFFFFFFF.824305F4 WCB$L_PREVCOUNT 00000000
FFFFFFFF.824305F8 WCB$L_PREVLBN 00000000
FFFFFFFF.824305FC WCB$L_PREVRVN 00000000
FFFFFFFF.82430600 WCB$L_COUNT 827BF0D8
WCB$L_WLFL
FFFFFFFF.82430604 WCB$L_LBN 8229F780
WCB$L_WLBL
FFFFFFFF.82430608 WCB$L_RVN 61120060
WCB$W_SIZE
WCB$B_TYPE
WCB$B_ACCESS
WCB$C_MAP_PTR_LENGTH

FFFFFFFF.8243060C WCB$L_PID 001D00A8 TCPIP$A
CCESS_SHR+7E0A8
FFFFFFFF.82430610 WCB$L_REFCNT 00000000
FFFFFFFF.82430614 WCB$L_ORGUCB 81D7A8C0
FFFFFFFF.82430618 WCB$L_ACON 00000841
FFFFFFFF.8243061C 00000000
FFFFFFFF.82430620 WCB$L_NMAP 00000001
FFFFFFFF.82430624 WCB$L_FCB 827BF0C0
FFFFFFFF.82430628 WCB$L_RVT 00000000
FFFFFFFF.8243062C WCB$L_LINK 00000000
FFFFFFFF.82430630 WCB$Q_OPENTIME 00A5F685.A86A0A57
FFFFFFFF.82430638 WCB$L_READS 00000007
FFFFFFFF.8243063C WCB$L_WRITES 00000000
FFFFFFFF.82430640 WCB$L_SPLIT_IO 00000000
FFFFFFFF.82430644 WCB$L_ASSIST_IO 00000000
FFFFFFFF.82430648 00000000
FFFFFFFF.8243064C WCB$L_STVBN 00000001
WCB$C_LENGTH

FFFFFFFF.82430650 WCB$L_P1_COUNT 00000090
WCB$Q_DELIQ
WCB$W_P1_COUNT
FFFFFFFF.82430654 WCB$L_P1_LBN 048999E0
FFFFFFFF.82430658 WCB$L_P1_RVN 00000000
FFFFFFFF.8243065C WCB$L_P2_COUNT 00000000
FFFFFFFF.82430660 WCB$L_P2_LBN 00000000
FFFFFFFF.82430664 WCB$L_P2_RVN 00000000
SDA>

[Volker Halle]

Vladimir,

the code has fetched the low-order longword of WCB$Q_OPENTIME(r2) - an OpenVMS 64-bit time quadword - and used it as an address...

SDA> EXA/TIME 82430630

should produce a valid time value.

The data structure pointed to be R2 could be some TCPIP-internal WCB. Try to pin it to some recognizeable TCPIP structure by using some pointers from this 'WCB':

SDA> VALI QUE/LIS 82430600 ! WCB$L_WLFL
SDA> SHOW PROC/ID=001D00A8 ! WCB$L_PID
SDA> FORMAT 81D7A8C0 ! WCB$L_ORGUCB
SDA> FORMAT 827BF0C0 ! WCB$L_FCB

Also try to verify pool using:

SDA> CLUE MEM/FREE
SDA> CLUE MEM/LOOK
SDA> SHOW POOL/SUMM/NONPAGED

Volker.

[Vladimir Fabecic]

Hello Volker
See attached file

[Vladimir Fabecic]

Sorry, forgot this:
SDA> EXA/TIME 82430630
28-NOV-2006 14:55:07.38
SDA>

[Volker Halle · Accepted solution]

Vladimir,

nonpaged pool and lookaside lists look o.k.

We will not be able to 'solve' this problem here. We can only collect evidence and document symptoms, so if anybody else is seeing this crash, they can confirm if it is the same problem.

Let me summarize the symptoms:

OpenVMS Alpha V8.2 TCPIP V5.5 ECO 1

INVEXCEPTN crash at IPL 8 with ACCVIO
at TCPIP$INTERNET_SERVICES+208AC
Failing instruction: LDQ R26,#X0008(R2)
R2 pointing to invalid address
R2 fetched from #X0030(R0)
R0 pointing to WCB

Here are some more things to look at:

SDA> SHOW DEV/ADDR=81D7A8C0 ! WCB$L_ORGUCB
SDA> READ SYSDEF
SDA> FORMAT 827BF0C0 ! WCB$L_FCB

SDA> FORMAT 827BF0D8
SDA> FORMAT 8229F780

The question still is, is this WCB the correct type of data structure the code is supposed to handle here. Using WCB$Q_OPENTIME as an address seems to indicate, that R0 should be pointing to something else.

Consider to escalate this to TCPIP engineering.

Volker.

[Volker Halle]

Vladimir,

you can also use

SDA> TCPIP TAG ALL

to cause TCPIP to define SDA symbols for all of it's data structures.

Then try look at most of the data structures examined/formatted before with:

SDA> EXA address

Volker.

[Vladimir Fabecic]

Volker, thanks a lot for your time. I also did some examing and found no good explaination why server crashed. In hardware errorlog there is nothing indicating possible hardware failure.
Looks like a bug to me.
I escalated a call to HP support.

[Volker Halle]

Vladimir,

once you receive a solution or explanation, please don't forget to update this thread.

Thanks,

Volker.