How to remove shadowset member to be used as backup

[Cor Mom]

Anton,
That's a good point, because it reduces the risks. In fact we do have a structure like you described and I think I am gonna use it.

Ian,
I did not forget, did not have time yet.

Regards,
Cor

[John Gillings]

Cor,
(and everyone else)

HOLD ON! This is not a good plan. The fundamental purpose of shadowing is to protect you from bad blocks. As soon as you reduce a shadowset to less than TWO members you are actually DOUBLING your exposure to bad blocks.

Remember shadowing is NOT a tool for making backups easy, it's NOT a tool for making I/O faster. You paid the big bucks for shadowing licenses to eliminate bad blocks. So, to use it in a way that *increases* your exposure makes no sense.

A simple rule to follow... don't ever reduce a shadowset to less than TWO members.

In this case, take a spare disk, add it to the shadowset and let it copy, then remove the newest disk as your backup.

As long as you're on V7.3 or higher, and there are no open files on the disk, a member dismounted from a shadowset will be in a consistent state.

As of V7.3-2, if you really can't find a 3rd member, then you can minimise your exposure to bad blocks by forcing a full merge on the shadowset:

$ SET SHADOW/DEMAND_MERGE DSA1

Note that this doesn't *eliminate* the risk, but it makes sure that all existing bad blocks are detected and corrected.

[Robert_Boyd]

John,

I agree with you about one of the purposes of using shadow sets being bad blocks. However the primary purpose that they have served in my experience is redundancy. The probability of having more than a single drive failure in a shadowset is much much lower than that for a single drive. This makes it very unlikely that the users would ever be subjected to a loss of access to the drive due to single drive failures. The primary risk for any period of time where the shadowset is reduced to 1 member is that any failure of that drive will result in loss of access and possible loss of data that has not been backed up.

I agree completely with the preferability of adding in a 3rd member to copy the disk rather than reducing the shadowset to 1 member if it is going to be a prolonged situation. As always, these risk management decisions involve tradeoffs: time, money, exposure to loss, efficiency, etc....

Robert

[Jan van den Ende]

John,

thanks for pointing out the thing I completely overlooked, and which others may therefore overlook as well, although from the other side.

In _MY_ perspective, I did not even think in reducing to single member. We reduce 3-member sets to 2-member.
At the current prices of diskdrives, any single issue is sooooo much more expensive, in repair time alone already, and if you factor in the production hours lost...

We are on record of requesting the 3 member limit to be expanded. We would much prefer to have 2 members at each site, plus the availability to ADD the backup member, instead of having to take it out!

Proost.

Have one on me.

Jan

[Andy Bustamante]

One more important point. You need to know that your application has a consistent database on the the shadowsets. Either a way to hold or force outstanding I/O or shut down. A "flying dismount" may lead to problems with data consistency. The shadowing manual has many cautions about ensuring data consistancy when dismounting a shadow set member, mostly saying "you should know your application and take appropriate steps."

[John Gillings]

Robert,

>The primary risk for any period of time >where the shadowset is reduced to 1 member >is that any failure of that drive will >result in loss of access and possible loss >of data that has not been backed up.

No so!

At any time there is a small, but finite probability that any individual block on a disk will go bad. We won't know it's bad until we read it. With very large numbers of blocks, even very small probabilities start to turn into certainties. In any case the probability of ANY block on the disk going bad within a specific time period is many orders of magnitude higher than that of the whole disk failing.

So we need to accept bad blocks WILL happen, guaranteed! However, provided the corresponding block on all other shadow members doesn't also go bad before we read the bad block, shadowing will be able to repair the damage from known, good data. (note that bad blocks behave like Schrodinger's cat in the classic physics thought experiment).

A shadow set with two members may have a number of bad blocks on one member, and a (hopefully different) set on the other member. As long as those sets don't intersect, then we don't care - they effectively don't exist. But, if we break the set, we've exposed all the bad blocks on both members.

Before minicopy, breaking the set was the equivalent of doubling the probability of exposure to bad blocks because a latent bad block on the surviving member would be realised when it was read, and the ones on the removed member would be propagated to the backup and realised when it was restored.

Mini copy helps a lot, because we might get the shadow set back together before reading one of the bad blocks, so our exposure is reduced to the set of blocks read while the shadow set is reduced.

Forcing a merge before breaking the set will make sure every block on every member is read, thereby finding any bad blocks and fixing them from a good member.

Even with mini copy, reducing a shadow set to a single member represents an easily avoided risk, especially considering the low cost of storage.

[Lawrence Czlapinski]

John, thanks for the bad block info. In practice, two member shadow sets are broken a lot to keep one as a fallback for various reasons (upgrades, standalone backup and restore to other member, etc.). Thanks for the tip about merging before breaking the shadow sets. I know we haven't been doing that. So many of us do care how many bad blocks are being generated per some unit of time and on which disks. If bad blocks are detected, they should be reported in the error log perhaps on a selectable time frame (daily,hourly). It would be useful to have a utility that could be started by a batch file off-peak for reading all the blocks of a physical disk and reporting the number of bad blocks detected. It could reduce the time till the bad blocks are detected significantly.
Lawrence

[David B Sneddon]

With regard to forcing merges etc. Would ANALYZE/DISK/SHADOW
do a similar task given that it reads all blocks on all
members and compares them? Obviously it won't do any
writing, which a merge will do.

Dave

[Ian Miller.]

Lawrence, bad blocks are reported to the error log when they are detected. The problem is that they are not detected until VMS tries to read or write the block. The commands mentioned could be run periodically to ensure all blocks on a disk are accessed and therefore bad blocks are detected sooner. The error count on the disk could be checked before and after if if it increases then check the error log.