VMS Puzzle #2: Using HBVS to allow read-only snapshot to be used system wide

[Wim Van den Wyngaert]

Mmmm. So the disk got mounted under WIM. But then I put it back into the shadow set with label DISK1 witout any options. It was accepted !!! The disk had not been mounted write since the dismount but still a full copy was started (no mini). But again, on my old 7.3 node.

Detail : the shadow_server doing the copy was on the node that did the mount.

Wim

[Ian Miller.]

The VTJ editor is always looking for material so do contact her about this possible article.

[Jon Pinkley]

Uwe,

Thanks for the info, I am not the Windows admin. I know it was a problem when we first started using the EVA, and we had to find a work around. We now have Windows Server 2003, so our admin is probably aware of the new functionality, but I will make sure he is.

Wim,

No time for details now, but here is my current understanding (I need to do more testing and research to know for sure, so be warned that this is my guess).

The label is changed in the HOMEBLOCK when the set volume/label is executed. The VCB (Volume Control Block) is an in memory copy of volume related info, and it is created at mount time and the values in it come from the HOMEBLOCK (and probably the SCB). My guess is that the VCB is the source of info that $GETDVI uses (and where show device gets its info). When you do a set volume, the VCB on the node where the command is executed is updated with the new value, but evidently there is no "cluster" updating done. The same is true for things like volume retention times, and logical volume size.

That does bring up a question about where the shadow server gets the volume label for its comparison with the write bitmaps, since the copy operation doesn't have to occur on the node that originally mastered the minicopy bitmap.

For the tests I did, I was using a test cluster with a single node. I will put another node in and do some additional tests, but I have to do some other things first, since I was at Nashua for two weeks.

I also used mount /policy=minicopy to ensure the copy was in fact a minicopy. That is supposed to ensure that the minicopy is used, or the mount will fail. And that's how I determined that the labels have to match for the minicopy to happen.

Jon

[Wim Van den Wyngaert]

Jon,

Could you do on your test system :
dsa0=disk1, disk2 label "D1"
dsa1=disk3 label "D2"

set vol dsa0 /lab=WIM
dism/pol=min=opt disk2
set vol dsa0 /lab=D1

set vol dsa1 /lab=WIM
mount dsa1/sys /shad=disk2

I wonder if it will get accepted and if it will do the minicopy but can't test that on my system due to lack of disks.

Wim

[Wim Van den Wyngaert]

I did the test myself and found that the disk is accepted witout any /over. Without minicopy.

I dismounted it again and could add it to the original dsa witout any /over. Without minicopy.

Wim

[Jon Pinkley]

Wim,

The scenario you proposed and the results you saw were normal and expected. Any time a new member is added to an existing shadowset a copy will occur, regardless of the label, contents of SCB of added member, etc. The only question is whether a FULL copy or a minicopy will happen. For a minicopy to happen, there must be a minicopy bitmap that the shadowing software believes represents the portions of the shadowset VU that have been written to since the SSM was remove (these portions are possibly different from the removed member).

Experiments show at least the following conditions must be met for shadowing to use a bitmap for a minicopy operation:

1. The bitmap name must specify the volume lock name in the SCB of the SSM to be added, and the SCB of the VU.
2. The bitmap name must specify the Generation number recorded in the SCB of the SSM to be added.
3. The volume labels of the VU and the SSM to be added must match.

In the example Wim proposed, conditions 1 and 2 were not met. Therefore a full copy was performed. However, it does show a way to get two volumes mounted systemwide that with the same label. However, they do not have the same volume lock name.

The added member will always be made consistent with the currently mounted shadowset, so any time a member is mounted, data on that member should be considered disposable, since any differences from the currently mounted shadowset will be overwritten (assuming no trickery such as modifying an SCB with Logical I/O is involved). Consider that this is exactly what happens when you add a new member that is erased to an existing VU.

"Black Box" experiments (i.e. looking at behavior without looking at the listings) seem to indicate that for a minicopy to occur, the following must be true: The HOMEBLOCK and SCB are read from the candidate for minicopy (the removed SSM). The Volume Lock Name from the SCB and the Generation number are used to compose the BITMAP name. For example, an SCB containing "ITRCSHAD " as the Volume Lock Name and "00A7A14D1B9D794B" as the generation number will have a bitmap with the name that is displayed by SDA> show device /bitmap as "SHAD$ITRCSHAD 000000A7A14D1B9D794B". I am not sure what the 16 zero bits before the generation number are. The Volume label specified in the mount and the volume label from the shadowset VU (from LD show/trace it appears each member of the VU is read when the new member is added), if these don't agree (and the /noassist qualifier was present), the error message %MOUNT-F-INCVOLLABEL, incorrect volume label is generated. If /assist was not present, then it prompts you to mount the correct volume for the VU. (This seems like an odd message, as the VU is already mounted...) Here's an example:

OT$ sho dev dsa9998

Device Device Error Volume Free Trans Mnt
Name Status Count Label Blocks Count Cnt
DSA9998: Mounted 0 ITRC_SCB 1269 1 1
$4$LDA7: (OMEGA) ShadowSetMember 0 (member of DSA9998:)
$4$LDA8: (OMEGA) ShadowSetMember 0 (member of DSA9998:)
$4$LDA9: (OMEGA) ShadowSetMember 0 (member of DSA9998:)
OT$ dism $4$lda8 /policy=minicopy
OT$ sho dev/bit/ful dsa9998
Device BitMap Size Percent Type of Master Active Creation Cluster Local Delete Bitmap
Name ID (Bytes) Populated Bitmap Node Date/Time Size Set Pending Name
DSA9998: 0006015D 12 0.01% Mini Copy OMEGA Yes 29-MAY-2008 18:19:52.17 127 0.01% No SHAD$ITRC_SCB ...9.K8¥§.


OT$ set vol/lab=itrc_scb1 dsa9998:
OT$ sho dev dsa9998

Device Device Error Volume Free Trans Mnt
Name Status Count Label Blocks Count Cnt
DSA9998: Mounted 0 ITRC_SCB1 1269 1 1
$4$LDA7: (OMEGA) ShadowSetMember 0 (member of DSA9998:)
$4$LDA9: (OMEGA) ShadowSetMember 0 (member of DSA9998:)
OT$ mou/sys dsa9998: /shad=$4$lda8 /policy=minicopy itrc_scb
%MOUNT-I-OPRQST, incorrect volume label
%MOUNT-I-OPRQST, Please mount volume ITRC_SCB in device _DSA9998:
Interrupt

OT$ ! That seems to be odd advice to me. It seems it should just report that the specified label was inconsistent with the currently mounted VU
OT$ mou/sys dsa9998: /shad=$4$lda8 /policy=minicopy itrc_scb/noassist
%MOUNT-I-OPRQSTCAN, operator request canceled
%MOUNT-F-INCVOLLABEL, incorrect volume label
OT$

In the above case the $4$LDA8 SSM was removed /policy=minicopy when the DSA9998: VU volume label was ITRC_SCB, then the label of the DSA9998: VU was changed to ITRC_SCB1. Therefore the label of the SSM is ITRC_SCB and the label of the VU is ITRC_SCB1. The label specified in the mount command must match the VU, not the label of the SSM being added. In the above case we tried using the label of the SSM and the mount request was rejected.

Now if we use the VU's label, and there is a bitmap matching the SSM's Volume Lock Name and Generation number, but the labels of the SSM and the VU do not match, then the bitmap is not considered valid, and if the mount command includes the /policy=minicopy, the mount will fail. If /policy=minicopy is not specified, the mount will succeed, but the bitmap will not be used, and a FULL copy will occur. This is something to be aware of if you have outstanding minicopy bitmaps and you decide to change the VU's volume label. To allow a minicopy to occur, you must temporarily change the VU's label to match the SSM member's label. Here's a demonstration continuing from the previous example:

OT$ mou/sys dsa9998: /shad=$4$lda8 /policy=minicopy itrc_scb1/noassist
%MOUNT-I-MOUNTED, ITRC_SCB1 mounted on _DSA9998:
%MOUNT-I-SHDWMEMFAIL, _$4$LDA8: (OMEGA) failed as a member of the shadow set
-MOUNT-F-REQ_WBM_FAIL, unable to add member using minicopy
%MOUNT-I-ISAMBR, _$4$LDA9: (OMEGA) is a member of the shadow set
%MOUNT-I-ISAMBR, _$4$LDA7: (OMEGA) is a member of the shadow set
OT$ set vol/lab=itrc_scb dsa9998
OT$ mou/sys dsa9998: /shad=$4$lda8 /policy=minicopy itrc_scb1/noassist
%MOUNT-F-INCVOLLABEL, incorrect volume label
OT$ mou/sys dsa9998: /shad=$4$lda8 /policy=minicopy itrc_scb /noassist
%MOUNT-I-MOUNTED, ITRC_SCB mounted on _DSA9998:
%MOUNT-I-SHDWMEMMCPY, _$4$LDA8: (OMEGA) added to the shadow set with a minicopy operation
%MOUNT-I-ISAMBR, _$4$LDA9: (OMEGA) is a member of the shadow set
%MOUNT-I-ISAMBR, _$4$LDA7: (OMEGA) is a member of the shadow set
OT$ set vol/lab=itrc_scb1 dsa9998

Given the above info, I always use mount /policy=minicopy if I think there is a valid bitmap. If the mount fails, you may still be able to achive a minicopy by changing the volume label of the VU. If you leave the /policy=minicopy off, the effect is the same as /policy=minicopy=optional, and in that case, the bitmap may be considered invalid, and thus a full copy may start. If you have a large disk, that can be a costly operation.

Jon

[Hoff]

{{{In the example Wim proposed, conditions 1 and 2 were not met. Therefore a full copy was performed. However, it does show a way to get two volumes mounted systemwide that with the same label. However, they do not have the same volume lock name.}}}

That brings back (bad) memories. That nailed a Freeware distro a while back, where the volume labels were changed to reflect the individual volumes and (unbenownst to me) the lock names were not. You couldn't mount both disks system-wide. Never having thought to test multiple parallel mounts and not having had multiple CD drives on the test system, that bug escaped out into the wild.

[Jon Pinkley]

Hoff,

Yes, I remember that problem.

Was the mount code changed since then? It seems that currently if a volume is mounted /nowrite, the Volume Lock Name in the SCB is irrelevant, the volume lock name that is created by Mount and stored in the VCB is taken from the label in the HOMEBLOCK. In fact, that is a necessary behavior for the removed member to be mounted /system. I.e. if you change the volume label of a shadowset VU, the volume lock name in the SCB is not changed, therefore in the removed member, the volume label in the HOMEBLOCK is different than the volume lock name in the SCB. However, you are still able to mount the removed member /system as long as you have returned the VU's label to something different than the removed member's.

Jon

[Hoff]

Was MOUNT changed? I believe it was tweaked for the /NOWRITE case, yes, though the entire original design was -- as implemented with SET VOLUME /LABEL and the largely inaccessible lock resource name in the SCB, and MOUNT and INITIALIZE -- arguably broken.

There were discussions around changes to either MOUNT or (better, IMHO, if you're going to use the lock resource name over in MOUNT) to SET VOLUME /LABEL. I don't have a way to check the resolution (if any) of those discussions anymore, however.

Jeff F. or Andy G. might (will?) know the answer or the plans.