Cross-site shadow sets, are they really necessary?

[Jeremy Begg]

Hi,

I'm (still) working on a project to replace a number of AlphaServers by the same number of BL860c i2 blades.  There is a C7000 chassis with 7 blades at one site and another chassis with 7 blades at the other site.  One blade from each site is paired with a blade at the other site to form a two-node VMScluster; there are seven such clusters.

Storage for the AlphaServer clusters is provided by EVAs at each site, which are being phased out as the migration progresses. Storage for the Itanium clusters is provided by Hitachi Data Systems (the corporate SAN here, aka "HDS").

However we've found the HDS storage doesn't perform as well as it should.  Work is on-going to investigate and remedy that problem - a problem which is made worse by having cross-site shadow sets and the long latencies they introduce into disk write I/Os.

Unlike the EVA setup here, the HDS system is configured to do mirroring across sites.  When an application writes to an HDS SAN disk, the write is automatically replicated (by the HDS hardware) to the other site.

Which gets us wondering, do we need to continue to use VMS volume shadowing to maintain our data at both sites?

I should add that the AlphaServers were monitored by DTCS and the blades will be also.  Does DTCS have a requirement for shadow sets?

When the HDS performance was being investigated the HDS people recorded all the relevant SAN traffic and noticed something interesting.  When VMS was writing to a shadow set, it would do a burst of I/O then wait 200ms before starting the next burst of I/O.  When VMS was writing to a non-shadowed disk, there was no 200ms delay between I/Os.  What is the purpose of this 200ms delay, and can it be shortened?

Thanks,

Jeremy Begg

[John Gillings]

Hi Jeremy,

I can't explain your 200ms bursts. Can't imagine that OpenVMS would do that. Are you sure it's not an artefact of the SAN? Is the HDS qualified for VMS?

>> do we need to continue to use VMS volume shadowing to maintain our data at both sites?

Choosing a level at which to do your cross site replication depends on why you're using multiple sites, and how you want to be able to recover your system state depending on different scenarios. The controller knows nothing about what's happening at application level (or even OS level), so, regardless of how "good" the blind replication is, if you consider points in time there will always be time windows during which the remote replica will be inconsistent from an application perspective. So, if the primary site is lost, what are the chances the DR copy of the data will be usable? How much processing will be required to be sure your system is in a consistent state from which you can recover?

For some applications, this might not be an issue. For example, if the data is basically just a linear time sequence (say collecting satellite remote sensing data), you should be able to look at the data and draw a line saying "it's good up to here". On the other hand, if you're transactional, with potentially many transactions in flight, it may be all mush. For complex files, like RMS indexed files, or data bases, they may even be corrupt because the replication layer had no idea how to correctly sequence multiple updates. For some workloads, I think it's likely that the remote replica will ALWAYS be inconsistent because of frequent, multiple overlapping updates. Storage folk sometimes don't seem to get that data is always changing, so replication (or, for that matter, backup) isn't just a matter of a big pipe and a fast shovel.

The benefit of host based shadowing is the sequence of writes is under direct control of the operating system, which has a good idea what's going on. That means the integrity of the remote disk can be strictly maintained at the file system level, so it doesn't matter when you pull the plug, you can be confident you can boot from and operate with whatever has reached the other side. Yes, this is slower, but do you want slow and safe or fast and risky?

You need to look at your system and keep asking yourself "what if it breaks right now", keeping in mind that failures have a habit of finding worst case timing ;-)

[fhsjvl]

Jeremy,

We have been using HDS + Brocade switches SAN with all kind of entry-level Itanium systems for years. However our data is so critical that DR is not an option (database + OLTP). We are using OpenVMS shadows sets, with all disks in both sites locally available to all VMS nodes. This means that each write to a volume is performed by two "local" I/Os issued by the VMS system itself. That is our way to keep it as simple as possible. After all, we have more confidence in the Shadow Server handling things than any other switch/controller/network/blackbox.

The 200ms delay is unknown here and I agree with John Gillings that it is not likely this comes from VMS.

We have had performance problems as well. But they came from the fact that the switches and the disks in the SAN are shared with multiple other architectures and applications which I cannot control. Heavy SAN activity by other systems can slow down our own systems. This is very unfortunate and very hard to detect and investigate (that is the price one pays for consolidation/virtualisation/cloudification).

I also agree with John that the choice between shadow sets, DR and any other scheme depends primarily on how much data corruption you can tolerate in case of failures.

[Jeremy Begg]

Thanks for the update.  I had to ask the question.

Some testing here has thrown up an interesting observation.  Running a BACKUP job from one HDS (non-shadowed) disk to another HDS (non-shadowed) disk took about 1.5 minutes on average.  The disks in question were then reconfigured as single-member shadow sets and the job run again.  This changed caused a signficant increase the in average runtime for this test, from 1.5 minutes to over 6!  I don't have more details about this available to me at present, but am I correct in assuming that shadowing shouldn't add that much overhead for single-member shadow sets?

Thanks,

Jeremy Begg

[abrsvc]

The problem here is that with the configuration you have, you have all of the overhead of shadowing with none of the benefits. Add a member to the "read" side and see the difference. Shadowing will allw the use of both disks for reading. This should speed up the read side. The write side will require both disks to be writetn so that may slow down the write sie a bit (not measureable in most cases).

For most applications that I have been involved with, the read benefits were clearly seen and the write overhead was not. Caching has an effect on this as well. Be careful what you are measureing relative to your actual application environment.

Dan

[Jeremy Begg]

It still seems a little odd that a single-member shadow set should incur that much overhead, compared to a single non-shadowed disk.

[abrsvc]

The overhead of shadowing will be dependent upon the I/O size more than anythnig. Realize that you have injected another processing layer into the I/O stream with HBSS. The overhead amount will be roughly fixed per I/O. If I get a chance, I'll try a few experiments to measure the overhead with differing I/O activity and report it here.

Dan