topic Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS in Operating System - OpenVMS

DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

EdgarZamora_1 — Tue, 25 Nov 2008 19:27:40 GMT

Any idea of what this error means...

|%DFU-I-MOUNTING, Busy mounting disk $1$DGA31:... |
|%DFU-I-ANALDISK, Analyzing INDEXF and BITMAP... |
|%DFU-I-TOTAL, Maparea maps 8654880 blocks in 35 fragments (86% used) |
|%DFU-I-FINDLBN, Largest free contiguous space 205743584 blocks at LBN 18075223|
|36 |
|%DFU-I-MOVE, 8650752 blocks can be defragmented (31 fragments) |
|Continue to modify INDEXF.SYS ? (Y/N) [N] : y |
|%DFU-I-MOUNTFOR, Busy remounting disk $1$DGA31: /FOREIGN... |
|%DFU-I-STARTDFR, Now copying fragments to new location... |
|%DFU-S-COPIED, 4718592 blocks copied (fragment 5) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 6) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 7) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 8) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 9) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 10) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 11) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 12) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 13) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 14) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 15) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 16) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 17) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 18) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 19) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 20) |
[---------------------------------< DFU V3.2 >---------------------------------]
|%DFU-S-COPIED, 131072 blocks copied (fragment 23) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 24) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 25) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 26) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 27) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 28) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 29) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 30) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 31) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 32) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 33) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 34) |
|%DFU-S-COPIED, 131072 blocks copied (fragment 35) |
|%DFU-E-MISMATCH, Error in new mapping pointers |
|%DFU-I-DISMNT, Volume dismounted |

The command I issued was DFU INDEXF/DEFRAG $1$DGA31:

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Richard W Hunt — Wed, 26 Nov 2008 19:20:18 GMT

I wonder if the problem was that something updated the INDEXF.SYS on that volume, or at least tried to?

I haven't used DFU but other defraggers abort the process if you change the disk's equivalent of an index file.

I see that it was doing a MOUNT/FOREIGN so you shouldn't have had that problem. I note also that you have a huge file being defragged at that point, and the last file name I saw was INDEXF.SYS - can it be that your index file is that large? INDEXF.SYS has special requirements and really doesn't like to get very big.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jess Goodman — Wed, 26 Nov 2008 20:18:55 GMT

Richard,
The command Edgar used was DFU/INDEXF/DEGRAG which tries to defragment one specific file: INDEXF.SYS! I think DFU might be the only software that has this feature.

Edgar,
I have a volume I was tempted to try this command on, but I was worried that a DFU bug (I have found a couple before) might corrupt the volume, since I doubt very many sites have tried this command. Is your volume still mountable, or was it corrupted?

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Hoff — Wed, 26 Nov 2008 21:06:47 GMT

Drop some email along to Jur; he doesn't manifest here very often.

The DFU web site has been down for a day or two now; not sure what's up with that.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Volker Halle — Thu, 27 Nov 2008 07:46:20 GMT

Edgar,

I would assume, that this might be an internal problem in DFU. So only Jur could help.

Did you consider to try the same operation on a scratch volume ? Or on a LD: device ? Just to test, if the operation works in general ?

Volker.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Volker Halle — Thu, 27 Nov 2008 08:00:58 GMT

Edgar,

testing on a test device does only seem to work, if the existing INDEXF.SYS is sufficiently fragmented. So not a straightforward task...

Volker.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jur van der Burg — Thu, 27 Nov 2008 09:14:37 GMT

www.digiater.nl is down, and I just redirected my email to another server. There's a communication failure between my home and the isp and they are working on it. I don't know how long it's gonna take though, I hope it will be back today or tomorrow but I got not guarantee. Keep on trying.

I will look into the DFU issue when I got some time.

Jur.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jur van der Burg — Thu, 27 Nov 2008 11:04:55 GMT

Can you post the output of this?

$ define dfu$nosmg 1
$ mc dfu indexf/ana/show $1$dga31:
$ mou/over=id $1$dga31:
$ dump/head/bl=co=0 $1$dga31:[000000]indexf.sys
$ dump/head/bl=co=0 $1$dga31:[000000]indexf.sys/nofor

Thanks,

Jur.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jur van der Burg — Thu, 27 Nov 2008 11:41:51 GMT

One more thing, what does ANALYZE/DISK say about this disk?

Jur.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Hein van den Heuvel — Thu, 27 Nov 2008 17:49:52 GMT

8 Million+ headers / files... that is a lot.
How many actually in use? I guess a lot as INDEXF.SYS has expanded that often.
Those 86% used is a little scary, but it is not clear it is worth fixing.

It might be easier to just pre-extend INDEXF.SYS in one fell swoop. Now there is no standard tool for this, but I wrote a little utility to do just that. You'll find it attached.

Example usage:

$ mcr sys$login:extend -c -e 70000 mda1:[000000]indexf.sys
Old: 96 blocks in MDA1:[000000]INDEXF.SYS;1
New: 70096 blocks. Extended by 70000. Contiguous.

Jur>> " dump/head/bl=co=0 $1$dga31:[000000]indexf.sys/nofor"

See... that's why I keep reading this stuff.
I have desired, a raw header dump but never noticed the /NOFORMAT sub-qualifier to /HEADER. I just directly dumped the block from INDEXF.SYS but this is easier.

btw... the HELP for DUMP /FORM reads:
" dumps the file header in octal format."
Fortunately that's not correct.
It is a hex dump. It does suggest the switch was there for a while!

Cheers,
Hein.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

EdgarZamora_1 — Mon, 01 Dec 2008 13:43:33 GMT

Sorry for the delayed response... long holiday weekend.

I was able to use HP's DFG 3.0 to defragment the INDEXF.SYS successfully.

Jess, the volume did not get corrupted by the DFU error... it remained the same (thank God).

Volker, I had actually used DFU on this same volume (and a couple of others) about a year ago to defragment the INDEXF.SYS successfully, so I know DFU does successfully defragment the INDEXF.SYS in most cases. I guess this volume is just getting bigger and bigger (thousands of archive files are being dumped into it) and I may have hit some limit on DFU.

Hein, thanks for that great tool you attached! I'm looking at it as it looks to be most useful.

Jur, I'm sorry I didn't see these responses before I ran DFG on the disk. The indexf is defragmented now. I won't be able to supply the debug info you requested.

Thank you all for the responses!

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jur van der Burg — Mon, 01 Dec 2008 15:58:24 GMT

>Jess, the volume did not get corrupted by the >DFU error... it remained the same (thank God).

DFU defrgaments indexf.sys by copying it to a new contiguous location found on disk. After the copy it found some issues, and did not update the rest, so only a piece of free space was overwritten. Call it the 'safe approach'.

Jur.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

EdgarZamora_1 — Mon, 01 Dec 2008 19:14:17 GMT

Jur,

As it turns out, I found a similar volume that exhibited the same problem when I tried using DFU to defrag the indexf.sys. I collected the info you asked for and I'm attaching to this reply. I will hold off defragging the indexf with HP DFG in case you want some other info. Thanks!

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jon Pinkley — Wed, 03 Dec 2008 00:34:39 GMT

Edgar,

I can't see anything obviously wrong, and I don't understand why this problem is happening. I have used this feature on volumes that have "used up" the mapping area, so it is something other than the number of fragments that is causing the problem. I have never defragged an indexf.sys file that had such large extents, so that may be a contributing factor.

Can you also provide the output of the following:

$ define dfu$nosmg 1
$ mcr dfu report/graph $1$dga925: ! more volume related info
$ show device/full $1$dga925: ! this will show total blocks and logical volume size
$ dump/count=1 $1$dga925:[000000]bitmap.sys ! Storage Control Block (SCB)

-----

To determine why DFU is reporting

%DFU-E-MISMATCH, Error in new mapping pointers

is probably going to require a change to the source code of DFU so it gives a better hint as to why it thought there was an error, i.e. what it was expecting and what it found.

Jon

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jon Pinkley — Wed, 03 Dec 2008 00:46:55 GMT

The last command to dump the SCB should be:

$ dump/block=count=1 $1$dga925:[000000]bitmap.sys

not

$ dump/count=1 $1$dga925:[000000]bitmap.sys

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

EdgarZamora_1 — Wed, 03 Dec 2008 13:20:28 GMT

Jon, thanks for spending some time on this. Here's the info you asked for.

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jon Pinkley — Thu, 04 Dec 2008 12:46:20 GMT

Edgar,

I see an unrelated bug in the output of DFU's /graph output, where it is evidently overflowing in the calculation of starting LBN. This works correctly on my 120GB disk (and I expect it to work correctly on any disk with logical volume size <= 153391689 blocks). In addition to the calculation being incorrect, the output field for LBN is too small for large disks. That may get fixed if another version ever gets released.

However, that is cosmetic and not related to this problem.

It would be interesting to see if it is possible to use the DFU INDEXF /EXTEND=xxx command, or if that results in the same problem. Since you have used about half the disk, you could just double what you currently have and use only one extent. For example:

$ mcr dfu indexf /extend=4700000 $1$DGA925:

Jon

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

EdgarZamora_1 — Thu, 04 Dec 2008 14:42:34 GMT

Jon, the /EXTEND worked...

$ mcr dfu indexf /extend=4700000 $1$DGA925:

Disk and File Utilities for OpenVMS V3.2
%DFU-I-MOUNTING, Busy mounting disk $1$DGA925:...
%DFU-I-ANALDISK, Analyzing INDEXF and BITMAP...
%DFU-I-TOTAL, Maparea maps 4760560 blocks in 38 fragments (94% used)
%DFU-I-FINDLBN, Largest free contiguous space 617758680 blocks at LBN 577326120
%DFU-I-EXTEND, INDEXF.SYS can be extended with 4700000 blocks
Continue to modify INDEXF.SYS ? (Y/N) [N] : y
%DFU-I-MOUNTFOR, Busy remounting disk $1$DGA925: /FOREIGN...
%DFU-I-NEWTOTAL, New Maparea maps 9460560 blocks in 39 fragments
%DFU-S-REWRTIF, INDEXF.SYS File header rewritten !
%DFU-I-RBDBITMAP, Updating BITMAP.SYS...
%DFU-S-DONE, all operations succesfully completed

%DFU-I-DISMNT, Volume dismounted

%DFU-I-READY, INDEXF command ready
$ mount/over=id $1$dga925:
%MOUNT-I-MOUNTED, ARCHIVE3 mounted on _$1$DGA925: (CLCC)
$ def dfu$nosmg 1
$ mcr dfu indexf /anal $1$dga925:

Disk and File Utilities for OpenVMS V3.2
%DFU-I-ANALDISK, Analyzing INDEXF and BITMAP...
%DFU-I-MAPPTR, Retrieval ptr ( 1) Size : 16 , LBN : 0
%DFU-I-MAPPTR, Retrieval ptr ( 2) Size : 8 , LBN : 1032
%DFU-I-MAPPTR, Retrieval ptr ( 3) Size : 8 , LBN : 370656
%DFU-I-MAPPTR, Retrieval ptr ( 4) Size : 304080 , LBN : 66576
%DFU-I-MAPPTR, Retrieval ptr ( 5) Size : 131072 , LBN :38346104
%DFU-I-MAPPTR, Retrieval ptr ( 6) Size : 131072 , LBN :63176120
%DFU-I-MAPPTR, Retrieval ptr ( 7) Size : 131072 , LBN :79053280
%DFU-I-MAPPTR, Retrieval ptr ( 8) Size : 131072 , LBN :107558728
%DFU-I-MAPPTR, Retrieval ptr ( 9) Size : 131072 , LBN :120191128
%DFU-I-MAPPTR, Retrieval ptr (10) Size : 131072 , LBN :133670072
%DFU-I-MAPPTR, Retrieval ptr (11) Size : 131072 , LBN :146541056
%DFU-I-MAPPTR, Retrieval ptr (12) Size : 131072 , LBN :159403536
%DFU-I-MAPPTR, Retrieval ptr (13) Size : 131072 , LBN :173544616
%DFU-I-MAPPTR, Retrieval ptr (14) Size : 131072 , LBN :207464576
%DFU-I-MAPPTR, Retrieval ptr (15) Size : 131072 , LBN :241281928
%DFU-I-MAPPTR, Retrieval ptr (16) Size : 131072 , LBN :258080104
%DFU-I-MAPPTR, Retrieval ptr (17) Size : 131072 , LBN :274775344
%DFU-I-MAPPTR, Retrieval ptr (18) Size : 131072 , LBN :214126040
%DFU-I-MAPPTR, Retrieval ptr (19) Size : 131072 , LBN :291149168
%DFU-I-MAPPTR, Retrieval ptr (20) Size : 131072 , LBN :307298640
%DFU-I-MAPPTR, Retrieval ptr (21) Size : 131072 , LBN :324895312
%DFU-I-MAPPTR, Retrieval ptr (22) Size : 131072 , LBN :341005752
%DFU-I-MAPPTR, Retrieval ptr (23) Size : 131072 , LBN :357158328
%DFU-I-MAPPTR, Retrieval ptr (24) Size : 131072 , LBN :372325304
%DFU-I-MAPPTR, Retrieval ptr (25) Size : 131072 , LBN :386814152
%DFU-I-MAPPTR, Retrieval ptr (26) Size : 131072 , LBN :403266872
%DFU-I-MAPPTR, Retrieval ptr (27) Size : 131072 , LBN :419210080
%DFU-I-MAPPTR, Retrieval ptr (28) Size : 131072 , LBN :493705864
%DFU-I-MAPPTR, Retrieval ptr (29) Size : 131072 , LBN :509799296
%DFU-I-MAPPTR, Retrieval ptr (30) Size : 131072 , LBN :525271840
%DFU-I-MAPPTR, Retrieval ptr (31) Size : 131072 , LBN :437381128
%DFU-I-MAPPTR, Retrieval ptr (32) Size : 131072 , LBN :454103432
%DFU-I-MAPPTR, Retrieval ptr (33) Size : 131072 , LBN :468584160
%DFU-I-MAPPTR, Retrieval ptr (34) Size : 131072 , LBN :545502912
%DFU-I-MAPPTR, Retrieval ptr (35) Size : 131072 , LBN :559213368
%DFU-I-MAPPTR, Retrieval ptr (36) Size : 131072 , LBN :570764696
%DFU-I-MAPPTR, Retrieval ptr (37) Size : 131072 , LBN :477588064
%DFU-I-MAPPTR, Retrieval ptr (38) Size : 131072 , LBN :487445880
%DFU-I-MAPPTR, Retrieval ptr (39) Size : 4700000 , LBN :577326120
%DFU-I-TOTAL, Maparea maps 9460560 blocks in 39 fragments (96% used)
%DFU-I-FINDLBN, Largest free contiguous space 613058680 blocks at LBN 582026120
%DFU-I-MOVE, 9156448 blocks can be defragmented (35 fragments)

%DFU-I-READY, INDEXF command ready
$

I will probably use HP DFG to defragment the INDEXF.SYS today anyway. It's good to know about the extend though. Thanks!

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jon Pinkley — Sun, 07 Dec 2008 09:53:53 GMT

Edgar,

I haven't determined what the cause is, but I was able to reproduce the problem. It seems that DFU doesn't like mapping pointers that map a multiple of 65536 (2^16) blocks. 131072=2*65536=2^17. DFU will not let me extend by either 65536 or 131072. The disk I was playing with was a snapclone of a 120GB drive, and currently is somewhat fragmented, so there wasn't enough space to extend by 3*65536.

There is an additional switch to DFU that can be used with the INDEXF operations, /show_pointers. The help suggests that it works only with /analyze, but it appears to work with /extend and /defrag. When I try to extend by a multiple of 65536, the reported pointer size is 65536 more than what was requested, so the sanity check fails. Following are examples for 65536 and 131072. Note that the new mapping pointer is 65536 blocks larger than it should be.

$ dfu indexf /exte=65536 /show $1$dga2401

Disk and File Utilities for OpenVMS V3.2
%DFU-I-MOUNTING, Busy mounting disk $1$DGA2401:...
%DFU-I-ANALDISK, Analyzing INDEXF and BITMAP...
%DFU-I-MAPPTR, Retrieval ptr ( 1) Size : 32 , LBN : 0
%DFU-I-MAPPTR, Retrieval ptr ( 2) Size : 16 , LBN : 1024
%DFU-I-MAPPTR, Retrieval ptr ( 3) Size : 16 , LBN : 38912
%DFU-I-MAPPTR, Retrieval ptr ( 4) Size : 5088 , LBN : 33824
%DFU-I-MAPPTR, Retrieval ptr ( 5) Size : 131152 , LBN :198826400
%DFU-I-TOTAL, Maparea maps 136304 blocks in 5 fragments (8% used)
%DFU-I-FINDLBN, Largest free contiguous space 132896 blocks at LBN 188392224
%DFU-I-EXTEND, INDEXF.SYS can be extended with 65536 blocks
Continue to modify INDEXF.SYS ? (Y/N) [N] : y
%DFU-I-MOUNTFOR, Busy remounting disk $1$DGA2401: /FOREIGN...
%DFU-I-MAPPTR, Retrieval ptr size 32 LBN 0
%DFU-I-MAPPTR, Retrieval ptr size 16 LBN 1024
%DFU-I-MAPPTR, Retrieval ptr size 16 LBN 38912
%DFU-I-MAPPTR, Retrieval ptr size 5088 LBN 33824
%DFU-I-MAPPTR, Retrieval ptr size 131152 LBN 198826400
%DFU-I-MAPPTR, Retrieval ptr size 131072 LBN 188392224
%DFU-E-MISMATCH, Error in new mapping pointers
%DFU-I-DISMNT, Volume dismounted

%DFU-I-READY, INDEXF command ready
$ dfu indexf /exte=131072/show $1$dga2401

Disk and File Utilities for OpenVMS V3.2
%DFU-I-MOUNTING, Busy mounting disk $1$DGA2401:...
%DFU-I-ANALDISK, Analyzing INDEXF and BITMAP...
%DFU-I-MAPPTR, Retrieval ptr ( 1) Size : 32 , LBN : 0
%DFU-I-MAPPTR, Retrieval ptr ( 2) Size : 16 , LBN : 1024
%DFU-I-MAPPTR, Retrieval ptr ( 3) Size : 16 , LBN : 38912
%DFU-I-MAPPTR, Retrieval ptr ( 4) Size : 5088 , LBN : 33824
%DFU-I-MAPPTR, Retrieval ptr ( 5) Size : 131152 , LBN :198826400
%DFU-I-TOTAL, Maparea maps 136304 blocks in 5 fragments (8% used)
%DFU-I-FINDLBN, Largest free contiguous space 132896 blocks at LBN 188392224
%DFU-I-EXTEND, INDEXF.SYS can be extended with 131072 blocks
Continue to modify INDEXF.SYS ? (Y/N) [N] : y
%DFU-I-MOUNTFOR, Busy remounting disk $1$DGA2401: /FOREIGN...
%DFU-I-MAPPTR, Retrieval ptr size 32 LBN 0
%DFU-I-MAPPTR, Retrieval ptr size 16 LBN 1024
%DFU-I-MAPPTR, Retrieval ptr size 16 LBN 38912
%DFU-I-MAPPTR, Retrieval ptr size 5088 LBN 33824
%DFU-I-MAPPTR, Retrieval ptr size 131152 LBN 198826400
%DFU-I-MAPPTR, Retrieval ptr size 196608 LBN 188392224
%DFU-E-MISMATCH, Error in new mapping pointers
%DFU-I-DISMNT, Volume dismounted

%DFU-I-READY, INDEXF command ready
$

Since the XQP likes to extend by 2^17 blocks, this probably affects a large number of disks that need to have their indexf.sys file defragmented.

Can you possibly try defragmenting again while using the /show_pointers qualifier? The defrag will not work, but it hopefully will provide some additional info that will help narrow the problem.

$ define dfu$nosmg 1
$ mcr dfu /defrag /show_pointers $1$DGA925:

Jon

Re: DFU-E-MISMATCH error in trying to defrag INDEXF.SYS

Jon Pinkley — Sun, 07 Dec 2008 10:39:45 GMT

Edgar,

What I menat was

$ define dfu$nosmg 1
$ mcr dfu INDEXF /defrag /show_pointers $1$DGA925:

Jon