Shadowing with IP Cluster

Allan Large — Tue, 10 May 2011 17:15:51 GMT

Has anyone tried enabling disk shadowing with Clustering over IP under VMS 8.4 ?

So far our experience is that it is way too slow to be functional.

Re: Shadowing with IP Cluster

Robert Brooks_1 — Tue, 10 May 2011 18:04:17 GMT

You might generate more useful discussion if you mention the relevant hardware involved, and other useful information (using a separate network for IPCI?, jumbo frames enabled?, MSCP-serving disks?, etc...).

Are your complaints with the speed of steady-state operation, or with copies/merges?

In general, for direct-attached storage on every cluster member, there will be little cluster traffic generated by HBVS for steady-state use, other than lock manager stuff.

As I've read somewhere before, terse questions beget terse answers.

-- Rob

Re: Shadowing with IP Cluster

Allan Large — Tue, 10 May 2011 18:17:25 GMT

Rob:

Actually I was just looking for someone that has tried it and was just looking for a general response.

Re: Shadowing with IP Cluster

Colin Butcher — Wed, 11 May 2011 05:23:18 GMT

Think about what's happening here.

HBVS does synchronous writes, ie all writes will complete before the IO operation returns completion to the user.

The limit to maximum achievable IO rate is IO latency.

Take a local EVA on direct path from a rx6600. You can typically achieve a raw disc write latency of just under 1 millisecnd - that's around 1000-1500 IOs / second, maximum, to that disc.

Add HBVS to that with multiple local direct path EVAs (say 3 way) and given how HBVS work, you'll probably achieve around 1.5 milliseconds at best, say 2 milliseconds. That's around 500 IOs/second, maximum, to that DSA device.

Now pull the EVAs apart to say 100 mile separation and add distance latency - it's easy to drop to around 100 IOs/sec max, with direct path fibre.

MSCP serving over IPCI is not direct path. You have the extra overhead of MSCP transactions and you have the extra overhead of the IP stack and the fact that an IP / ethernet network behaves completely differently to fibrechannel direct path. It's easy to double or triple the IO latency.

So, it's really easy to end up with a quite small maximum achievable IO write rate. Sometimes that's too small for your applications to cope with.

MSCP serving over IPCI will work, but, as you've found, it will have worse IO write latency than direct path fibrechannel.

More overhead = worse latency = what you think of as poor performance under your specific circumstances relative to your expectations.

Now think how that affects normal operations (which are generally small block count individual IO writes) and shadow copy / merge operations (which sensibly try to group up IOs into large block count IO writes) - and the effect hat has on both the latency and throughput. Latency governs IO response and thus IO rate, bandwidth (and things like transmit window size in the network) govern IO throughput.

This stuff is complex under the hood with all the different effects at the different levels of the assortment of protocol stacks and drivers - a lot more complex than most people consider.

The laws of physics are inconvenient, but real. That's why testing in your specific circumstances matters and it's why good design matters.

Cheers, Colin (http://www.xdelta.co.uk).

topic Re: Shadowing with IP Cluster in Operating System - OpenVMS

Shadowing with IP Cluster

Re: Shadowing with IP Cluster

Re: Shadowing with IP Cluster

Re: Shadowing with IP Cluster