MPW List in a SAN environment

[Richard W Hunt]

My configuration is an OpenVMS cluster (pair of Alpha ES40) with a non-HP (Symmetrix) Storage Area Network server connected through fiber channel and using the more or less standard "class" disk driver. The Symm presents the disks as though they were on HSG80-class controllers. This is on OpenVMS 7.2, and I am reasonably familiar with OpenVMS internals from my VAX days. But I'm not as comfortable with Alpha/VMS internals.

We are using point-in-time replication, so that means that I have to take advantage of a quiet system moment to take a data snapshot of my disks. The good news is that through system performance monitoring and an analysis of login times from my accounting files, I found a suitable time. The bad news is, my MPW list is populated (as is true for all OpenVMS unless they are badly mis-tuned). Those MPW pages are dirty. When I take my snapshot, the disks do not EXACTLY reflect what has been recently done to them because some of the changes are still in an MPW page somewhere. So my snapshots are "blurred."

Short of forcing a dismount of all disks just before the SAN operation, is there a way to force a complete flush of the Modified list just before I am about to take final action on the disk snapshots? Not all of the MPW parameters are dynamic, and permanently setting MPW_LOLIMIT to zero is not highly recommended. But that is what I would have to do if I wanted to reset MPW_HILIMIT to a low number to force the flush that way.

If I have to write some code to call a system routine to just trigger a flush, I'll try it. (I'm silly enough to try nearly anything.) But I don't think I can afford the side effects if I try to dismount and remount the disks. There is always that "stray" job that might hold the disk in the marked-for-dismount state and stall everything for me.

[John Gillings]

Good question...

It illustrates the near impossibility of taking a reliable on-line backup without the active cooperation of all application software.

You could try flushing the MPL, but there would still be timing windows in which a page might be modified between the flush and the snapshot point. Also remember that VMS goes to great pains to prevent unnecessary I/O, so you'll definitely be working against the grain.

Things to think about. In working out your backup policy, START by thinking about how you want to RESTORE and work backwards to figure out what needs to be saved, and THEN work out how to achieve that. Using knowledge of your data you can often reduce what needs to be saved to a small fraction of the total storage.

In this case, you're starting with a mechanism for saving data and then trying to "brute force" it into saving everything, rather than just the stuff you KNOW you need. Typically only the application knows when data is in a consistent state.

Remember that there will be no dirty pages for processes that have been deleted or images that have been run down. Of course, for global sections you need ALL accessors to have closed the section.

Regardless of flushing MPL and "quiet time", unless files have been closed and disks dismounted, you cannot be certain to capture all files in application level consistent states at a known point in time. This applies to all low level "snapshot" technologies, removing shadow members from shadow sets and BACKUP/IGNORE=INTERLOCK. The best you can hope for using this type of application blind backup is "crash consistency" (ie: the backup is as good as the state of the system after an unplanned crash or power failure).

Unfortunately there are no universal "magic wands" for backing up data. Look beyond the sales pitches - it's the RESTORE side that matters most! The best solution is full integration in the application.

(Note that post V7.3 shadowing will guarantee the integrity of a removed shadow member at the file system level, but cannot guarantee the consistency within and between files which were open at the time the physical unit is removed).

[Keith Parris · Accepted solution]

I wonder if there is an implicit assumption here that file-system data is on the modified page list, similar to the buffer cache in Unix.

The modified page list on VMS is intended to hold program data pages (including global sections) that have been modified; RMS and the VMS file system do their caching and buffering in such a way that data integrity is not dependent on the MPL getting written to disk. RMS and the VMS file system use a technique called Careful Write Ordering, in which changes are written to disk (and then the code waits for the I/O to complete) in such an order that the file system (or internal file) metadata on disk will always be in a consistent, readable state at any given point in time. If your EMC takes a snapshot as of an instant in time (effectively not allowing any additional I/Os through to the snapshot once the snapshot operating begins), your snapshot should be a readable copy of the file data as of the instant in time that the snapshot is taken.

Let's look at this from another angle: Say you were able to force VMS to flush the MPL to disk. All the data from the modified page list will go into page and swap files or into RMS files that are the backing store for writeable global sections. Pages put into page and swap files are of no use except while a node is running, so it is only any writeable global sections that would be of concern. Does your application keep volatile data in writeable global sections, and depend on that data getting flushed to disk to be saved? The VMS file system itself and RMS file-handling code do not do this. If your application depended on flushing writeable global sections to disk, you'd have to code in a method to trigger it to perform that flush before doing a snapshot.

[Ian Miller.]

an interesting article on backups can be found in the OpenVMS Technical Journal V1 here.
http://h71000.www7.hp.com/openvms/journal/v1/backup.html

Basically it makes the point that application co-operation is required and you have to think carefully about why you are backing up.

[Richard W Hunt]

Thanks to all who replied. To add a little bit more information, I have an ORACLE database set up with four instances. Three of these represent test/developer instances. The fourth is the production instance.

My replication script "cooperates" with ORACLE by asking it to shut down the test instances during my quiet time. Since those disks support folks who don't come to work until three or four hours later, they aren't affected by this procedure. The Symmetrix disks are set into groups that segregate the instances. As each instance comes down, I wait for ORACLE to say "OK" then force the snapshot for the group related to that instance.

For the production instance, I have to drop it into HOT BACKUP mode, take the snapshot, release the hot backup, and then separately take the snapshot of the disks that contain the archive-log and redo-log files. Two data groups are involved here, an instance container group and a separate logging group.

My problem is that this version of ORACLE has a big system global area in memory. This part bothers me a lot because I have no control and no concept of what is in it at any given moment. This is the part that might have components still in the MPL.

For the closed instances, your replies suggest I'll be OK because ORACLE definitely closes all files for those instances. For the HOT BACKUP instance, we are trying to capture the change-logs AFTER we capture the instance itself, so maybe that will be good enough.

We are still jockeying to run a test at our remote site, but so far that hasn't happened successfully. This new script, because it synchronizes with ORACLE, might make the difference. As you folks noted, cooperation among tasks is paramount to the success of this operation. We came to that very same conclusion when our prior tests (using Symmetrix-driven replication) just flat wouldn't run. Almost certainly due to serious non-cooperation between ORACLE and the Symm.