XQPERR, Error detected by file system XQP

[mrityunjoy]

Hi,

One of server has crashed with XQPERR. I have analysed the clue$errlog.sys and got the below entry. Server model is Alpha server ES47 and openvms 7.3-2 is running. Can somebody help me to understand what was wrong and the reason for crash please.

Logging OS                        1. OpenVMS

System Architecture               2. Alpha

OS version                           V7.3-2  

Event sequence number         22165.

Timestamp of occurrence              21-AUG-2012 01:05:00

Time since reboot                    149 Day(s) 3:57:18

Host name                            RMSUA1  

System Model                         hp AlphaServer ES47 7/1150

Entry Type                       37. Crash Re-Start

Bugcheck Minor class              1. Crash Re-start

Bugcheck Msg                         XQPERR, Error detected by file system XQP

Process ID                x008401AE

Process Name                         AUTOMATION

KSP                       x000000007AF81028

ESP                       x000000007FF8C000

SSP                       x000000007FFAC000

USP                       x000000007AF6AD30

R0                        x0000000000000001

R1                        xFFFFFFFF86418030

R2                        xFFFFFFFF81A80700

R3                        xFFFFFFFF86FB4A00

R4                        xFFFFFFFF81A80700

R5                        xFFFFFFFF8641D840

R6                        xFFFFFFFF81A80700

R7                        xFFFFFFFF86FB4A00

R8                        x000000007AF8160C

R9                        x000000007AF81C8C

R10                       x000000007AF8160C

R11                       x000000007AF81584

R12                       x000000007AF81580

R13                       xFFFFFFFF86422650

R14                       xFFFFFFFF81A80700

R15                       x000000007AF81B84

R16                       x00000000000005B4

R17                       x0000000000000000

R18                       x00000000000005F0

R19                       x00000000000005B0

R20                       xFFFFFFFF8020BD48

R21                       x0000000000000002

R22                       x0000000000000000

R23                       x0000000100000000

R24                       x0000000000000000

R25                       x000000000000012F

R26                       xFFFFFFFF8020BD48

R27                       xFFFFFFFF86308000

R28                       xFFFFFFFF8020BD48

FP                        x000000007AF81028

SP                        x000000007AF81028

PC                        xFFFFFFFF8020BD84

PS                        x2800000000000800

PTBR                      x0000000000026FDD

Process Ctl Block Base Re x000000046A136080

PRBR                      xFFFFFFFF81642600

VPTB                      xFFFFFEFA00000000

System Ctl Block Base Reg x0000000000000943

Software Interrupt Summar x0000000000000000

ASN                       x00000000000000E5

ASTSR ASTEN               x000000000000000F

FEN                       x0000000000000001

IPL                       x0000000000000008

MCES                      x0000000000000000

[Bob Blunt]

The first questions are going to revolve around the level of your patching pretty much who you ask.  If you're not moderately current then the potential that you're exposed to an XQP crasher is greater.  V7.3-2 has a LOT of patches (I lost count when I my job included tracking them).  If you're not using one of the more recent UPDATE patches that would be where I'd start.  There have even been some recent additions that would go "on top" of the latest UPDATE that, while they're probably not absolutely required for XQP, certainly won't hurt if you're going to attempt to become more current.

If, however, you've got current patches then I'd drop any XQP crash like yours on HP in a New York second (but my bet is that you're in need of current patching).

bob

[mrityunjoy]

Hi Bob,

Thanks for your reply.

Last patch installed was

------------------------------------ ----------- ----------- --- ---------------------------------------------------------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- ---------------------------------------------------------
DEC AXPVMS VMS732_MONTOR V1.0        Patch       Install         18-MAR-2008
DEC AXPVMS VMS732_UPDATE V15.0       Patch       Install         18-MAR-2008
DEC AXPVMS VMS732_PCSI V4.0          Patch       Install         18-MAR-2008

DEC AXPVMS VMS732_VERIFY V2.0        Patch       Install         27-DEC-2007
DEC AXPVMS VMS732_UPDATE V14.0       Patch       Install         27-DEC-2007
DEC AXPVMS VMS732_SHADOWING V6.0     Patch       Install         27-NOV-2007

[Volker Halle]

Mrityunjoy,

this looks like the following XQP bugcheck in [F11X]INIFC2 routine UPDATE_FILESIZE

BUG_CHECK (XQPERR, FATAL, Unexpected filesize for INDEXF.SYS);

R2 should point to the FCB of INDEXF.SYS, @44(R2) should be FCB$_FILESIZE: verify with SDA> EXA @44(R2)

The new value for filesize is in R18 = 000005F0, which is VERY UNLIKELY to be a correct value for the file size of INDEXF.SYS ! If the new FILESIZE to be set is less than the FCB$L_FILESIZE, this crash is taken.

The most probably 'suspect' for a crash like this would be (third-party) disk management/cache/defragmentation tools running on ANY node in the cluster.

Your patches are not really 'up-to-date', but the symptoms point in another direction. Have you checked ($ TYPE CLUE$HISTORY), whether you've this specific bugcheck before on any node in the cluster ?

Volker.

[mrityunjoy]

Hi Volker,

Thanks for your reply.

I could not see any similar bugcheck happened earlier on any other nodes of the cluster from clue$history. In this case what do you suggest? How shall I proceed my investigation to get the root cause?

Mrityunjoy.

[Volker Halle]

Mrityunjoy,

please first verify my analysis with SDA:

$ ANALYZE/CRASH system-dump-file-name

SDA> READ SYSDEF

SDA> EXA @44(R2)  ! the value shown should be GREATER than 5F0

SDA> FORMAT @R2 ! format the FCB of INDEXF.SYS, look for FCB$W_FID and FCB$W_FID_SEQ

The File-ID and File Sequence Number should be 1,1 for INDEXF.SYS

SDA> FORMAT @(R2+FCB$L_WLFL) ! this should format the Windows Control Block (WCB)

The field WCB$L_ORGUCB should point to the disk device UCB

SDA> SHOW DEV/ADDR=<value of WCB$L_ORGUCB> should tell you the disk involved in this crash

Do you run any 3rd party (i.e. non-HP) tools on any node in the cluster, which could possibly manipulate the size of INDEXF.SYS (defragmenter, disk caching tools) ? If NOT and you're only running HP software, you might need to consider to install the most recent OpenVMS patches.

Volker.

[Craig A Berry]

It sure sounds like the problem Jim Duff reported recently at http://www.eight-cubed.com/blog/archives/001249.html#001249.  Apparently it's yet another bug caused by the symlink changes in v8.4.

[Craig A Berry]

Never mind, I just noticed you're at 7.3-2, not 8.4.

[Volker Halle · Accepted solution]

Craig,

on OpenVMS Alpha and I64, it's VERY EASY to compare crash footprints: just look at the first paragraph of the CLUE file (in CLUE$COLLECT:CLUE$node_ddmmyy_hhm.LIS) of the crash:

The XQPERR crash reported here is:

Bugcheck Type:     XQPERR, Error detected by file system XQP
VMS Version:          V7.3-2 
Current Process:   AUTOMATION
Current Image:       DSA4:[LCH$COMMON.IAM.PRODUCT.ALPHA_V7]RULE_ENGINE.EXE;1
Failing PC:              FFFFFFFF.8020BD80    UPDATE_FILESIZE_C+00080

It's a different module/routine name than in the V8.4 XQPERR seen by Jim. And OpenVMS Alpha and I64 are built from the SAME sources, so the routine names are identical. EVERY crash detected in and reported by the Files-11 XQP is called: 'XQPERR, Error detected by file system XQP' - and there are hundreds of them...

Volker.