topic Re: Shadow set merge after crash in Operating System - OpenVMS

Shadow set merge after crash

Kevin Carter_3 — Thu, 13 Jan 2011 19:49:53 GMT

Running OpenVMS V8.3 on an AlphaServer DS20 with a shadowset on two 36Gig drives formatted as ODS-5 volume. Only 7 gigs are in use but the merge after reboot took about 1 hour and 45 minutes on a system with very little use (no users logged in).

$ show shadow dsa30
DSA30: Volume Label: ORACLE
Virtual Unit State: Steady State
Enhanced Shadowing Features in use:
Host-Based Minimerge (HBMM)

VU Timeout Value 3600 VU Site Value 0
Copy/Merge Priority 5000 Mini Merge Enabled
Recovery Delay Per Served Member 30
Merge Delay Factor 200 Delay Threshold 200

HBMM Policy
HBMM Reset Threshold: 1000000
HBMM Master lists:
Up to any 2 nodes in the cluster - Multiuse: 0
Modified blocks since bitmap creation: 13081

Device $1$DKC0 Master Member
Read Cost 2 Site 0
Member Timeout 120

Device $1$DKB300
Read Cost 2 Site 0
Member Timeout 120

What can I do to improve (reduce) the merge time?

Re: Shadow set merge after crash

abrsvc — Thu, 13 Jan 2011 20:36:48 GMT

Because merges are rather I/O intensive, they are somewhat throttled to prevent any slowing down of the entire system. If I recall the algorithms correctly (its been a while), there is intensive verification to prevent any possible data overwrites. It is the data checking that takes the time. Others will likely add details, but that timeframe is typical for the shadow merges I have seen recently.

Dan

Re: Shadow set merge after crash

Robert Gezelter — Thu, 13 Jan 2011 22:04:12 GMT

SAVDBA,

Any messages in the log?

Since shadow copies are direct block-for-block copies, the number of blocks in use is not directly relevant to how long the copy takes.

If the volumes do not need to be fully available to 36GB, it may pay to reorganize storage so the entire volume is not shadowed (and then use SET VOLUME) to increase the size of the volumes as needed. This is somewhat different spin on the techniques I spoke of in "Migrating OpenVMS Storage Environments without Interruption or Disruption" at the 2007 HP Enterprise Technology Symposium (see http://www.rlgsc.com/hptechnologyforum/2007/1512.html).

- Bob Gezelter, http://www.rlgsc.com

Re: Shadow set merge after crash

Kevin Carter_3 — Thu, 13 Jan 2011 22:29:27 GMT

Bob,

From the operator.log
%%%%%%%%%%% OPCOM 13-JAN-2011 09:54:30.79 %%%%%%%%%%% Message from user SYSTEM on SADS20 %SHADOW_SERVER-I-SSRVBEGMRG, BEGINNING merge operation on _DSA30: at LBN: 0, I/O size: 127 blocks,

%%%%%%%%%%% OPCOM 13-JAN-2011 09:54:30.79 %%%%%%%%%%% Message from user SYSTEM on SADS20 -SHADOW_SERVER-I-SSRVINIMRG, ID number: 38000045, Threshold 200, Factor 200.

Then of course the notice of the completion.

Interesting idea, only use part of the disk and expand as needed...

Kevin

Re: Shadow set merge after crash

John Gillings — Thu, 13 Jan 2011 22:39:08 GMT

Kevin,

Full merges take a long time. Typically much longer than a straight copy. I notice that you have Mini Merge enabled, and wonder why it wasn't used. How many nodes crashed, and how did you recover?

Please check your configuration to make sure you're allowing all optimisations.

Re: Shadow set merge after crash

Kevin Carter_3 — Thu, 13 Jan 2011 23:08:55 GMT

John,

The system is a stand alone server, the drives are on separate controllers.

I'm not sure what you mean by "make sure you're allowing all optimisations".

Kevin

Re: Shadow set merge after crash

John Gillings — Thu, 13 Jan 2011 23:43:14 GMT

$ HELP SET SHADOW/ENABLE

SET

SHADOW

/ENABLE

/ENABLE=HBMM

Note: This qualifier applies to HBMM operations only. If you specify any non-HBMM qualifiers with this one, the command will fail.

Enables host-based minimerge (HBMM) on the specified shadow set or across the entire cluster if an applicable HBMM policy exists. To learn more about HBMM policies and their application, see the HP Volume Shadowing for OpenVMS manual.

HBMM is the only supported value for /ENABLE, and it must be included.

------------------------------------------

Host Based Mini Merge is an optimisation which bypasses the need for a full merge under some circumstances. If it's used it can drammatically reduce merge times (ie: your objective). However, there are quite a few conditions which need to be met before HBMM can be used.

See the Shadowing manual, and in particular, the section in Chapter 1 labelled "HBMM Configuration Requirements" and all of Chapter 8 "Host-Based Minimerge (HBMM)".

Re: Shadow set merge after crash

Kevin Carter_3 — Thu, 13 Jan 2011 23:52:07 GMT

John,

HBMM is enabled, the formating was lost on the original post above.

Enhanced Shadowing Features in use:
Host-Based Minimerge (HBMM)

Thanks
Kevin

Re: Shadow set merge after crash

Robert Brooks_1 — Fri, 14 Jan 2011 00:10:26 GMT

The system is a stand alone server, the drives are on separate controllers.

---

HBMM requires the use of at least a two-node cluster. In other words, while you may have HBMM enabled, the bitmap that's being created to track writes is useless. If you have a spare Alpha or IA64 system gathering dust (and a cluster license), I'd boot it up, form a two-node cluster, and give HBMM a shot. In general, HBMM merges will complete in seconds, or a few minutes at most (depending on various parameters, but HBMM merges are quite quick).

This assumes some storage that is directly-attached to both systems; if the MSCP-serving node crashes, then the remaining node can't access the device(s), and can't merge.

-- Rob

Re: Shadow set merge after crash

Kevin Carter_3 — Fri, 14 Jan 2011 00:14:47 GMT

HBMM requires the use of at least a two-node cluster

Re: Shadow set merge after crash

Martin Vorlaender — Fri, 14 Jan 2011 03:12:47 GMT

> Merge Delay Factor 200 Delay Threshold 200

> What can I do to improve (reduce) the merge time?

Give more I/Os to the merge operation (which of course adversely affects user operation). The above two settings are the knobs to do that; see the section titled "Improving Performance of Unassisted MergeOperations" at http://h71000.www7.hp.com/doc/732final/aa-pvxmj-te/00/00/75-con.html .

HTH
Martin