EVA, VMS Blade and preferred paths

[Steve Reece_3]

Hiya,

I should be asking this of my HP Storage Expert, but he's already home for the evening and I have a nagging doubt in my head.

config:
Three EVA6400s with eight disk shelves each.
BL860c blade servers - two enclosures with three and one with eight.
SAN in between the hosts and the storage managed by others. Storage will be configured using Command View and Continuous Access will be doing replication between two of the EVAs.
OpenVMS 8.3-1H1 on all of the blades.

Question:
With VMS being able to do path switching, do I need/want to include a preferred path in the EVA config or should I just rely on VMS to pick a path and, if I don't like it, choose a different path?

With redundant paths between the controllers and no fibre connection between them, I'm suspecting that we should actually just leave VMS to manage it.

Any ideas please???

Steve

[The Brit · Accepted solution]

Steve,
As you can see from the Help for "set preferred_path", this is usually used for MSCP served disks.
You could use this at boot time to mount your disks units using a specific path, you would need to have specific knowledge about the IO characteristics of the volume for your path choices to be even close to optimal.
The alternative would be to let the system select a path at boot time and then, if necessary, modify it later using the "set device /path= /switch". As far as I recall, if a path switch is required, (because the current path in not available) the system will switch to the "next available", not necessarily the best.
Unless you have an extremely IO intensive system with specific "HOT" disks, I would recommend letting the system take care of it.
If you are using Virtual Connect (VC) in your enclosures, then you need to consider your VC SAN configuration to ensure the maximum number of redundant paths.

I would also comment that I am surprised that you are using CA for data replication when Volume Shadowing is both available and superior (assuming that you have the EOE license.) CA on EVA is not really synchronous, even when you specify synchronous. (it is basically asynchronous with a large journal file)

but that is just my opinion, it is of course a personal choice.

HTH

Dave.

[Steve Reece_3]

Thanks Dave.
Host based shadowing would be great, but I can't do that to DR systems that are a couple of hundred miles away and not clustered (the latency of the disks would be too high I fear.)
Steve

[The Brit]

Agreed!

Just be sure that you understand the limitations of CA on the EVA's! Dont get me wrong, I am a big fan of EVA storage, but I work at a location where there have been "issues".

Dave.

[Hoff]

You're probably planning for this, but I'd leave it alone and start rolling up some T4 data under normal load.

I have a basic SAN load-balancing discussion posted at:

http://labs.hoffmanlabs.com/node/1451

And generic DT advice: test the full CA path before you need the recovery.

[Steve Reece_3]

Hi Hoff,

I'm planning on collecting it when this new environment goes live, but the ancient VAX systems that we're running the apps on at the moment doesn't have anything like T4 on it and we're fairly constrained on what we can and can't install.

The SAN management contact that we have has experience of EVA, but none of VMS yet is telling me that he must have preferred controllers configured on the EVA for every LUN.

Steve

[Hoff]

T4 is built on MONITOR /RECORD, and that exists on VAX. So if you can't get to use T4, you can at least get a baseline.

[Steve Reece_3]

Good point! Ta!

[Robert Gezelter]

Steve,

T4 itself is effectively a reporting tool. The CPU-related information can be run directly.

I suggest that you install T4 on one of your Alpha/Itanium systems, and take a careful look at the batch job that collects statistics. It is straightforward to implement the MONTIOR/RECORD, even if the system is significantly behind on revisions.

The resulting data file can then be extracted and used with directly with the T4 analysis tools.

- Bob Gezelter, http://www.rlgsc.com

[Robert Brooks_1]

As far as I recall, if a path switch is required, (because the current path in not available) the system will switch to the "next available", not necessarily the best.

---

Right -- there is no definition of "best".

There is, however, a definition of "worst". An MSCP path will only be automatically switched to if there are no other local paths.
In that case, as soon as a local path becomes available, multipath will automatically "fail back" to that newly-available local path.

Multipath will first try paths that are "connected" before any "non-connected" paths.
This is an older HSG/HSZ artifact from the dual controller setup, where a LUN would only be "connected" to one of the two controllers at a time. For the EVA's, with
active/active support, it's less of an issue, but I believe that there is still a distinction for a dual-controller setup, with a performance penalty on reads.

-- Rob

[Robert Atkinson]

Hi Steve.

We've recently moved to async-CA between Essex and Lincs. Up until then, we'd come to the conclusion of NOT using SET DEV/PATH, as they get pushed around by VMS anyway.

However, because DR disks in a particular group all have to be delivered from the same controller, we've decided to add SET DEV/PATH from VMS for those disks (don't ask me why - I can't remember!).

Another piece of information you may not be aware of is that although the path may be switch from controller A to B, controller B will not necessarily pick up the load. Instead, controller B passes all requests to controller A, and it continues to do the work. If your EVA is heavily loaded, this can start to show.

Cheers, Rob.

[Uwe Zessin]

> active/active support, it's less of an issue, but I believe that there is still a distinction for a dual-controller setup, with a performance penalty on reads.

Right, because on mid-range arrays into which group the EVA belongs (despite the E-nterprise in its name) a single controller is responsible for all physical disk I/O of a particular virtual disk.

Does that mean that VMS still does not understand ALUA (asymmetric logical unit access)?
If it did, you should set up preferred paths (in reality it means you assign management of a virtual disk to a controller...) on the EVA and all VMS hosts should detect this and limit I/O to one of the 'performance paths'.

[JBF]

Steve,

Since V8.3 OpenVMS understands and uses Asymmetric Logical Unit Access (ALUA) with the EVA storage arrays.

OpenVMS will always choose a path based on which controller manages the vdisk. It also uses this information during a mount verification event. For example, if a path fails (such as when a fibre cable is removed) then OpenVMS will (1) first retry the current path, then (b) search for another Active Optimized (AO) path (on the same controller) to the LUN, and then if necessary choose an Active Not Optimized (ANO) path on any other controller that can access the LUN. If OpenVMS must move to an ANO path, then OpenVMS requests the new controller assume control of the LUN and make it an Active Optimized path.

Due to the coordination between nodes in a VMScluster, I do NOT reocmmend using preferred pathing on the EVA storage array. I have worked several cases where this led to problems. Instead, I recommend allowing OpenVMS to pick a path.

However, on top of that, I suggest you build a DCL command procedure to periodically perform a SET DEVICE/SWICH/PATH=. Ideally you will take the top "n" devices (in terms of the I/O request rate) and spread those across the available paths. This only needs to be done on one node in the VMScluster if you run OpenVMS V8.3 or later. And you probably only need to do this once a day.

Why do this? It helps avoid problems where all the LUNs are switched to one controller (such as when a switch or controller is rebooted). This will at least daily be certain your top workload is balanced across the paths.

Hope that helps!
=jbf=

John B. Fisher

[Robert Atkinson]

Interesting reading John. I'd heard that VMS had become more 'aware' of the controllers, but this is the first information I've seen to back that up.

Could you expand a little more on "Due to the coordination between nodes in a VMScluster, I do NOT reocmmend using preferred pathing on the EVA storage array"

Why is this important, and how would it effect the use of DR disk groups?

Thanks, Rob.

[Ian Miller.]

will multipath polling switch to a AO path from a ANO path or does the polling just check that the path works?

[Robert Brooks_1]

will multipath polling switch to a AO path from a ANO path or does the polling just check that the path works?

--

The multipath poller is very lightweight. The entity that's relevant is the path, not any one device. If *any* device is reachable on a path, that path is deemed "working".

The details for those with the sources:

Take a look at the mpdev_ppb structure in [lib]
and the code in [sys]mpdev_poller.c. You'll see that the mpdev_ppb structure maintains a list of all devices on a path; thus, there is an mpdev_ppb structure for each path on a system.

The poller walks this list of devices, sending an io$_path_verify I/O. If the first device responds correctly, the poller for that path terminates, and waits for the next interval (usually 60 seconds, if a path is OK). If the first device has "issues", then we'll try the next device. Only after *every* device on a path has failed do we declare the path "bad", and we emit an OPCOM message to that effect.

The poller does not affect current paths for any device *except* for the case of "failback" from an MSCP path. If we've failed over to an MSCP path because all fibre paths are down, it's the poller's job to notice that a device that automatically switched to the MSCP path can now be "failed back" to a fibre path. In that case, the poller will initiate a path switch away from the MSCP path. If, for some reason, the MSCP path was manually selected, the poller will leave it on the MSCP path.

-- Rob

[Ian Miller.]

There does not appear to be a way of telling VMS to re-evaluate the paths and choose the 'best'

[EdgarZamora_1]

In reading this thread, John (Fisher) mentions that OpenVMS 8.3 is ALUA-aware. I was wondering where I could find documentation on this. Unless I missed it, I didn't find it anywhere in the OpenVMS docs.

I guess I will do some actual experimentation and see what happens (although I may have to stop short of doing actual controller failovers... users will probably not take too kindly to that).

[Robert Brooks_1]

There does not appear to be a way of telling VMS to re-evaluate the paths and choose the 'best'

--
Whose definition of best path? Fewest devices using it as a "current path"? Fewest I/O's per unit time? Fewest number of bytes transferred per unit time?

Among the things we wanted to explore with multipath was "concurrent multipath", where I/O could be sent down more than one path at a time.

-- Rob

[EdgarZamora_1]

I've gotten the answer I needed from the storage vendor...

Hi Edgar,

The answer to your question is a resounding NO to ALUA!
Do not enable ALUA. OpenVMS does not really support ALUA. (The feeble attempt by HP to do this on the EVA doesn't count).
Since OpenVMS likes static (stable) paths, switching manually is the best approach. Hosts like OpenVMS (at system boot) like to use paths in the order they are presented from the fabric.
The dance goes something like this (on both Alphas and I64):
Host Adapter FGA0 - Who do I see out there??? Give me some target names!!!
Host Adapter FGB0 - Who do I see out there??? Give me some target names!!!
And so on and so on...

Since the OpenVMS sees the same target names over and over (isn't multipathing beautiful?), it selects the first path it sees and says "you're my primary". It then saves the others as alternate/backup paths.
OpenVMS is very fair minded in that it doesn't discriminate on race, color, creed, national origin, or optimal path. So It'll take the prize behind door # 1.

In the real world, this is not always the correct choice. I would suggest that you determine the best (optimal) paths to use, and then place a small script in your systartup_vms chain to set the paths at system boot. Unless there is something else going on in your environment, these paths should remain stable. (Older versions of OpenVMS like 7.2-1 had awful FC polling algorithms and did some path switching on their own. Versions 7.3 and above seem very stable.)

Hope this helps.

[Robert Brooks_1]

I've gotten the answer I needed from the storage vendor...

--

Who is your storage vendor?

-- Rob

[EdgarZamora_1]

This was NetApp. I asked them if I should enable ALUA on the VMS luns and they responded that the host has to support ALUA.

[Robert Brooks_1]

NetApp's understanding of how VMS implements multipath is not quite accurate.

I've never used any NetApp stuff, and I have no direct knowledge of how well it "plays" with VMS; my experience is pretty much limited to using DEC/Compaq/HP storage subsystems with multipath, which is how we tested our code.

Pretty much anything that John Fisher says regarding storage can be taken as gospel.

For what it's worth, the multipath code hasn't been touched in a few years -- certainly not since V8.3.

-- Rob (who wrote some of the multipath code in question)