topic Re: XQPERR, Error detected by file system XQP in Operating System - OpenVMS

XQPERR, Error detected by file system XQP

mrityunjoy — Tue, 21 Aug 2012 00:40:16 GMT

Hi,

One of server has crashed with XQPERR. I have analysed the clue$errlog.sys and got the below entry. Server model is Alpha server ES47 and openvms 7.3-2 is running. Can somebody help me to understand what was wrong and the reason for crash please.

Logging OS 1. OpenVMS

System Architecture 2. Alpha

OS version V7.3-2

Event sequence number 22165.

Timestamp of occurrence 21-AUG-2012 01:05:00

Time since reboot 149 Day(s) 3:57:18

Host name RMSUA1

System Model hp AlphaServer ES47 7/1150

Entry Type 37. Crash Re-Start

Bugcheck Minor class 1. Crash Re-start

Bugcheck Msg XQPERR, Error detected by file system XQP

Process ID x008401AE

Process Name AUTOMATION

KSP x000000007AF81028

ESP x000000007FF8C000

SSP x000000007FFAC000

USP x000000007AF6AD30

R0 x0000000000000001

R1 xFFFFFFFF86418030

R2 xFFFFFFFF81A80700

R3 xFFFFFFFF86FB4A00

R4 xFFFFFFFF81A80700

R5 xFFFFFFFF8641D840

R6 xFFFFFFFF81A80700

R7 xFFFFFFFF86FB4A00

R8 x000000007AF8160C

R9 x000000007AF81C8C

R10 x000000007AF8160C

R11 x000000007AF81584

R12 x000000007AF81580

R13 xFFFFFFFF86422650

R14 xFFFFFFFF81A80700

R15 x000000007AF81B84

R16 x00000000000005B4

R17 x0000000000000000

R18 x00000000000005F0

R19 x00000000000005B0

R20 xFFFFFFFF8020BD48

R21 x0000000000000002

R22 x0000000000000000

R23 x0000000100000000

R24 x0000000000000000

R25 x000000000000012F

R26 xFFFFFFFF8020BD48

R27 xFFFFFFFF86308000

R28 xFFFFFFFF8020BD48

FP x000000007AF81028

SP x000000007AF81028

PC xFFFFFFFF8020BD84

PS x2800000000000800

PTBR x0000000000026FDD

Process Ctl Block Base Re x000000046A136080

PRBR xFFFFFFFF81642600

VPTB xFFFFFEFA00000000

System Ctl Block Base Reg x0000000000000943

Software Interrupt Summar x0000000000000000

ASN x00000000000000E5

ASTSR ASTEN x000000000000000F

FEN x0000000000000001

IPL x0000000000000008

MCES x0000000000000000

Re: XQPERR, Error detected by file system XQP

Bob Blunt — Tue, 21 Aug 2012 05:17:34 GMT

The first questions are going to revolve around the level of your patching pretty much who you ask. If you're not moderately current then the potential that you're exposed to an XQP crasher is greater. V7.3-2 has a LOT of patches (I lost count when I my job included tracking them). If you're not using one of the more recent UPDATE patches that would be where I'd start. There have even been some recent additions that would go "on top" of the latest UPDATE that, while they're probably not absolutely required for XQP, certainly won't hurt if you're going to attempt to become more current.

If, however, you've got current patches then I'd drop any XQP crash like yours on HP in a New York second (but my bet is that you're in need of current patching).

bob

Re: XQPERR, Error detected by file system XQP

mrityunjoy — Tue, 21 Aug 2012 06:06:48 GMT

Hi Bob,

Thanks for your reply.

Last patch installed was

------------------------------------ ----------- ----------- --- ---------------------------------------------------------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- ---------------------------------------------------------
DEC AXPVMS VMS732_MONTOR V1.0        Patch       Install         18-MAR-2008
DEC AXPVMS VMS732_UPDATE V15.0       Patch       Install         18-MAR-2008
DEC AXPVMS VMS732_PCSI V4.0          Patch       Install         18-MAR-2008

DEC AXPVMS VMS732_VERIFY V2.0        Patch       Install         27-DEC-2007
DEC AXPVMS VMS732_UPDATE V14.0       Patch       Install         27-DEC-2007
DEC AXPVMS VMS732_SHADOWING V6.0     Patch       Install         27-NOV-2007

Re: XQPERR, Error detected by file system XQP

Volker Halle — Tue, 21 Aug 2012 06:12:49 GMT

Mrityunjoy,

this looks like the following XQP bugcheck in [F11X]INIFC2 routine UPDATE_FILESIZE

BUG_CHECK (XQPERR, FATAL, Unexpected filesize for INDEXF.SYS);

R2 should point to the FCB of INDEXF.SYS, @44(R2) should be FCB$_FILESIZE: verify with SDA> EXA @44(R2)

The new value for filesize is in R18 = 000005F0, which is VERY UNLIKELY to be a correct value for the file size of INDEXF.SYS ! If the new FILESIZE to be set is less than the FCB$L_FILESIZE, this crash is taken.

The most probably 'suspect' for a crash like this would be (third-party) disk management/cache/defragmentation tools running on ANY node in the cluster.

Your patches are not really 'up-to-date', but the symptoms point in another direction. Have you checked ($ TYPE CLUE$HISTORY), whether you've this specific bugcheck before on any node in the cluster ?

Volker.

Re: XQPERR, Error detected by file system XQP

mrityunjoy — Tue, 21 Aug 2012 08:13:01 GMT

Hi Volker,

Thanks for your reply.

I could not see any similar bugcheck happened earlier on any other nodes of the cluster from clue$history. In this case what do you suggest? How shall I proceed my investigation to get the root cause?

Mrityunjoy.

Re: XQPERR, Error detected by file system XQP

Volker Halle — Tue, 21 Aug 2012 09:17:31 GMT

Mrityunjoy,

please first verify my analysis with SDA:

$ ANALYZE/CRASH system-dump-file-name

SDA> READ SYSDEF

SDA> EXA @44(R2) ! the value shown should be GREATER than 5F0

SDA> FORMAT @R2 ! format the FCB of INDEXF.SYS, look for FCB$W_FID and FCB$W_FID_SEQ

The File-ID and File Sequence Number should be 1,1 for INDEXF.SYS

SDA> FORMAT @(R2+FCB$L_WLFL) ! this should format the Windows Control Block (WCB)

The field WCB$L_ORGUCB should point to the disk device UCB

SDA> SHOW DEV/ADDR=<value of WCB$L_ORGUCB> should tell you the disk involved in this crash

Do you run any 3rd party (i.e. non-HP) tools on any node in the cluster, which could possibly manipulate the size of INDEXF.SYS (defragmenter, disk caching tools) ? If NOT and you're only running HP software, you might need to consider to install the most recent OpenVMS patches.

Volker.

Re: XQPERR, Error detected by file system XQP

Craig A Berry — Tue, 21 Aug 2012 11:45:55 GMT

It sure sounds like the problem Jim Duff reported recently at http://www.eight-cubed.com/blog/archives/001249.html#001249. Apparently it's yet another bug caused by the symlink changes in v8.4.

Re: XQPERR, Error detected by file system XQP

Craig A Berry — Tue, 21 Aug 2012 11:47:45 GMT

Never mind, I just noticed you're at 7.3-2, not 8.4.

Re: XQPERR, Error detected by file system XQP

Volker Halle — Tue, 21 Aug 2012 11:58:43 GMT

Craig,

on OpenVMS Alpha and I64, it's VERY EASY to compare crash footprints: just look at the first paragraph of the CLUE file (in CLUE$COLLECT:CLUE$node_ddmmyy_hhm.LIS) of the crash:

The XQPERR crash reported here is:

Bugcheck Type:     XQPERR, Error detected by file system XQP
VMS Version:          V7.3-2
Current Process:   AUTOMATION
Current Image:       DSA4:[LCH$COMMON.IAM.PRODUCT.ALPHA_V7]RULE_ENGINE.EXE;1
Failing PC:              FFFFFFFF.8020BD80    UPDATE_FILESIZE_C+00080

It's a different module/routine name than in the V8.4 XQPERR seen by Jim. And OpenVMS Alpha and I64 are built from the SAME sources, so the routine names are identical. EVERY crash detected in and reported by the Files-11 XQP is called: 'XQPERR, Error detected by file system XQP' - and there are hundreds of them...

Volker.