Need to speed up expansion of very large savesets

[Kenneth Toler]

I have sets of savesets that need to be expanded simultaneously. 8 savesets are 130 GB. It took 5 full 24 hour days for these savesets to fully exspand. Is there a way to speed up this process? Someone said to use the "mount" command with the following qualifiers "/blocksize & /cache" along with the "mount" command. Is this the best solution?

I am using SCSI drives for my source and destination. The source drive contains the savesets that range from 14 to 50 GB each. I am using OpenVMS 7.2

Does anyone have any ideas?

[Robert Gezelter]

Ken,

By expand, do you mean restore?

Have you measured what the bottleneck really is?

If you want to talk about this privately, please contact me via email.

The BLOCKSIZE parameter on disk mount is unlikely to have an effect.

Caching will also likely not help, as the data is not being reused.

I would consider increasing the RMS parameters, and possibly using a local DECnet connection, HOWEVER, I would not recommend random experimentation. It is far easier to see what the bottleneck actually is and address the problem directly.

- Bob Gezelter, http://www.rlgsc.com

[Kenneth Toler]

Yes, I mean restoring savesets. One problem is the tri-link SCSI connector. But there is nothing we can do about that. What I need is a bonefied way to harness the true power of OpenVMS to accelerate the restoration process. What parameters can be set for RMS to fix this? I have backups to my system disk, so, experimentaion is not a problem.

[Kenneth Toler]

C.2.1 Sequential File Access
To make t he best use of DECdfs's quick file access, most applications benefit from default RMS multibuffer and multiblock values of 3 and 16, respectively, when accessing sequential files.
Set the number of buffers to 3 for the most efficient multibuffering of file operations. Use the following DCL command:

$ SET RMS_DEFAULT/BUFFER_COUNT=3 /DISK
Next, set the size of each buffer to sixteen 512-byte blocks:

$ SET RMS_DEFAULT/BLOCK_COUNT=16
To set these values for just your user process, you can include the commands in your LOGIN.COM file. To set them on a systemwide basis, you can add the /SYSTEM qualifier a nd include the commands in the DFS$SYSTARTUP file.
RMS multibuffer and multiblock values that are larger than the default values can slow performance by allowing the application to exceed the DECnet pipeline quota. However, these values are recommendations that may not be optimal for every application. If your application opens many files or if it has a small working set size, you may find these values are too large.
Note
________________________________________
If you prefer, you can set the RMS default multibuffer value by using the SYSGEN parameter RMS_DFMBF. You can set the RMS default multiblock value by using the SYSGEN parameter RMS_DFMBC.

Is this what you were talking about?

[Robert Gezelter]

Kenneth,

Actually, I suspect that the SCSI cabling is not the problem.

What does the MONITOR show on the disks involved in the operation?

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Kenneth,

The BUFFER and BLOCK factors do help, if BACKUP actually honors them in your context. When I do backups, various versions of BACKUP do not honor the RMS defaults.

To force the use of the RMS defaults, I use local DECnet to access the file via transparent file access. The File Access Listener (FAL) DOES honor the RMS settings (although to get them set for the FAL process requires that they be set in the LOGIN.COM file (I generally condition them on F$MODE() .eqs. "NETWORK").

The RMS parameters are not only for the use of DECdfs (which I presume you are not using).

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Kenneth,

A small note: I will be leaving my terminal in a few minutes. While I will not post the number for my mobile in the forum, I will be happy to give it to you if you send me an email.

Please remember to remove the "REMOVETHIS" from the email address on my contact page.

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

>> $ SET RMS_DEFAULT/BUFFER_COUNT=3 /DISK
>> $ SET RMS_DEFAULT/BLOCK_COUNT=16

Not good enough. 16 blocks had been the defautl for decades. The new default is actually 32. So 16 would be a step backward.
I would suggest 64 as a general default ( /SYSTEM ) and 120 for the process doing the restore.

A multi buffer count of 2 is normally enough to make the sequential file write behind work. But for a I'd err on the save side and set it to 4.

And while you are there... why not SET RMS/SYSTEM/EXTEN=1000 (or more). Better safe than sorry for this one. Overallocated sequential file will be truncated back to the minimum clusters needed on close.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Jon Pinkley]

Kenneth Toler,

In summary I would recommend using LDDRIVER and transferring the container files: details follow.

Can we assume this is related to your previous questions?

PKZIP for VMS vs backup/log http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1114625

"The savesets are now in excess of 500,000 files with more than 1,000,000 close at hand."

If these 500,000 files total to 130 GB, then the average file size is around 250 KB. or about 500 blocks each.

Total Number of Files and Blocks inside savesets http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1121642

If so, then it seems you are restoring a large number of files. And possibly to a directory that has many thousands of files. These are both operations that VMS is not optimized for

You don't specify what the target disk(s) are (other than SCSI), is it on an HSZ controller? (from reference to tri-link adapter). If it is, then writeback caching will help, but not as much as the solution that follows. Specifically the use of LDDRIVER and container files.

First, if you were restoring multiple savesets to a single "spindle", then I would expect you to get better performance by single threading the restores. This seems counterintuitive, but by increasing the number of files you are restoring concurrently, you will cause the heads to do additional seeking, and that will kill your performance. If all your files are 1 block in size, it probably won't make much difference, you are going to get poor performance if you try restoring on a file-by-file basis no matter what you do.

Here is an option. Use physical backups to sequential savesets on the portable drive and physical restores on the destination machine. If you don't have identical drives on both sides, use the LDDRIVER (on the freeware or directly from Jur van der Burg's site, see thread http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1100345 LDDRIVER allows you to make a single file appear like a disk, which can be mounted. This would
allow you to create identically sized virtual disk LDAxx units on both systems, a requirement for backup/physical in older versions of VMS.

If both the customer and you are willing to load the LDDRIVER, then you have another even easier option.

Use an LD device to create you files on. When it is time to provide a new "snapshot", dismount the LD device and backup/ignorie=(nobackup) or copy the file to the SCSI drive to be taken to the customer site. If you ever upgrade, you will need to make sure you disable caching on the LD container file, as is mentioned in the release notes. Take the SCSI drive to the customer site and copy the container files off the SCSI drive. THIS WILL BE MUCH FASTER than file-by-file restores.

When the container files are copied to the customer's disk, use LD connect to create a new LD devices, specifying the new container files.

Mount the LD devices. You are done.

The LDDRIVER is a very thin (low overhead) intercept driver, so you can use it for production use.

[Hoff]

Please identify the VAX or Alpha host here, the SCSI controller(s), and the details of the I/O path out to the target device(s).

The OpenVMS BACKUP performance (as was measured within OpenVMS Engineering, and discussed in various presentations), when using the recommended process quota settings and baseline tuning, has been found to be within 90% of the underlying hardware bandwidth; within 90% of the slowest component from source to target.

The OpenVMS releases after V7.2 can and do have a number of performance optimizations (including various changes relevant to BACKUP and to I/O), but I'd (also) be looking carefully at the performance of the hardware involved. The host, the SCSI controller, bus contention, and the device.

Extent settings and such and process quotas and such can and certainly do help with performance, but I'd also be looking really, really, really hard at things like I/O bandwidth all the way out to the target, and disk queue depths. And at the host VAX or Alpha processor here.

Operator process quotas are listed in the tables available at: http://64.223.189.234/node/49

I'd also be looking at the specific processing requirements here, to are see if there are process changes that might be relevant.

And a reversal of interpretation on why you really want to look at hardware bandwidth: if you're achieving within approximately 90% of the bandwidth of the slowest I/O giblet, I'd be looking at hardware upgrades, and not at performance tuning.

Stephen Hoffman
HoffmanLabs LLC

[Jon Pinkley]

Let's look at the numbers provided:

130GB restored in 5 days.

Assuming larger interpretation of 130 GB

130 * 2^30 = 130 * 1073741824 = 139586437120 bytes

5 days = 5 * 24 * 60 * 60 = 432000 seconds

So average throughput is 139586437120 byte/432000 seconds or just under 325000 bytes/second.

So you do need to find out what the bottleneck is.

Since you already have some large files on the SCSI disk, can you measure the time it takes to copy the 14GB saveset to a disk.

You may be able to increase the performance some by adjusting the process RMS block and buffer counts, but both oth those utilities are smart enough to allocate the space in one go, and then copying the blocks, so setting the RMS extend size probably will not affect copy performance. It definitely helps when creating a file that is going to be large.

One you determine the speed that you can copy a large file, you will have a baseline for whoat you could expect if using the LDDRIVER method.

Note that by using LDDRIVER the performance will be independent of the size or number of files contained in the container file. The smaller the files are, the more benefit you will get from using LDDRIVER to treat a single file as a disk.

[Kenneth Toler]

Actually the OpenVMS version is 7.3 not 7.2. We are about to go to 8.3, but am not sure when.

The system is alpha based now, not VAX.

There are 50,000 to over 500,000 files per saveset. The 500,000 is the 50 GB saveset.

Any other ideas?

[Jon Pinkley]

Maybe I wasn't specific enough. By copy the saveset I mean copy the single file, not use backup to restore the individual files.

for example:

$ copy dkb5:[savesets]disk1.bck dka3:[somedir]

or

$ backup dkb5:[savesets]disk1.bck dka3:[somedir]/interchange/own=par ! this copies the file disk1.bck, it does not expand to individual files

[Kenneth Toler]

I am not neccesarily needing to copy the saveset. I need to greatly improve the performance of expanding large savesets with 100s of thousands of files each.

[Robert Gezelter]

Kenneth,

As I noted earlier, it is very far from clear, without performance data, where the bottleneck actually is.

Looking at the followup postings, it would not be surprising if the issue is not the access to the savesets themselves, but the actual output files themselves. Information about the actual performance of the system is needed to understand what the bottleneck is, and then how the problem can be addressed.

- Bob Gezelter, http://www.rlgsc.com

[Jon Pinkley]

My point is that by using LD container files, you will not need to recreate millions of files. That is a very slow operation on VMS. By using LDDRIVER you can move the whole disk filesystem as a single file.

If a single spindle is sufficient, then they could just access the files directly on the disk you provide. If they have HBVS licensed, you could even use that to make the files available while they were being copied (via the shadowing software), although while it is copying, the performance is going to be poor for other users, and the amount of I/O to get the files moved will be higher (I think at least double). That would be good if you needed availitly instead of raw performance.

The purpose of copying the saveset is just to measure the "best case" speed. Copying a single large file is going to be faster than copying a whole bunch of small ones. It doesn't have to be a saveset, just some large file. If the time it takes to copy the data as a single file is too long, then you need to look at other options, like upgrading the system.

You haven't provided any metrics for what type of performance they are expecting once the files are loaded on the customers system, or even what you consider to be acceptable time to restore. If you are expecting to be able to restore 130 GB in 1 hour, that's nearly 40 MB/sec, and I doubt your are going to be able to achive that coming from a single spindle (even if you were restoring to a RAM disk). Hein can probably provide soem good guidelines for what hardware is needed to support sustained raw transfer ratess.

Without you giving us more info, we really can't provide an optimaal solution. I don't think you are going to find a magic bullet that will be best for any arbitrary case.

If you are interested in considering the LD driver approach, I can give a few more details, otherwise, I'll let others give you their favorite tuning tweaks.

If you think you may be interested, please visit http://www.digiater.nl/lddriver.html for a good description of what it is.

[Hoff]

>>> My point is that by using LD container files, you will not need to recreate millions of files. That is a very slow operation on VMS. By using LDDRIVER you can move the whole disk filesystem as a single file.<<<

I don't understand this. You still have to create the files inside the LD device, right? And LD still has to write through to the underlying disk, right? Yes, LD has a single file system, but if you're restoring a saveset into individual files you're still working with a file system inside the LD device. (And LD isn't a RAM-based disk; it does write back to disk for the blocks written into the LD device.)

[Kenneth Toler]

What is LDDRIVER?

If it is not part of the as-delivered OpenVMS 7.3, I can't use it. If it is part of OpenVMS 7.3, please tell me how to access and use it.

[Jon Pinkley]

Re: Hoff's question.

Reading between the lines, this appears to be an ongoing requirement.

If he used an LD device on his system, the files would be created in the container file. This it would be easy to transport the "disk" to the other system as a single file.

So Hoff is corrrect, it doesn't solve the problem of getting from a saveset to the disk. However, it avoids the need in the future.

Tim: How can I make my car go faster?
Tom: Why does it need to go faster?
Tim: I need to drive 2000 miles, and it takes too long.
Tom: Did you consider taking a plane?

[Hoff]

LDDRIVER is part of OpenVMS Alpha V7.3-1 and later (with some wrinkles), and fully part of V8.2 and later. LDDRIVER is also available via the Freeware, and (among other uses) central to recording optical media on OpenVMS.

Information on LD is available in the V8.2 and later OpenVMS manuals, and details on configuring and using LDDRIVER are also embedded in the recording details at: http://64.223.189.234/node/28

LD would work as a way to pass over a block of data -- for this volume of data, do ensure you have the current LD ECO, as older drivers had a bugcheck with volumes in this range. As for approaches similar to LD, I'd be tempted to perform the BACKUP or gzip locally, then transfer the saveset or the archive as a unit. Or work on compressing or reducing the quantity of data involved. Or both.

Though pending the information on the system and I/O hardware and on the particular limitation or bottleneck here between the source and the target, I'm only really left to speculate.

[Steven Schweda]

> What is LDDRIVER?

http://h71000.www7.hp.com/freeware/freeware80/ld/

Also:

http://h71000.www7.hp.com/faq/vmsfaq_stmlf.txt

Get the most recent LDDRIVER available on the
Freeware, or activate and use the LD version latent
in OpenVMS Alpha V7.3-1 and V7.3-2 by loading the LD
command verb (look within SYS$MANAGER:CDRECORD.COM
for related details), or use the integrated LD found
in OpenVMS V8.2 and later.

So, assuming that V7.3 really means V7.3,
and if you insist on it being built-in,
rather than merely widely accepted and very
useful, then I suppose that you'll need to
wait for an OS upgrade.

[Jon Pinkley]

http://www.digiater.nl/lddriver.html for a good description of what LDDRIVER is.

[Hoff]

>>>The system is alpha based now, not VAX.<<<

There are about two thousand models of Alpha systems. Give or take. There are yet larger numbers of I/O widgets. There are yet larger numbers of disks and tapes.

>>>There are 50,000 to over 500,000 files per saveset. The 500,000 is the 50 GB saveset.<<<

750 GB drive spindles are common in various servers, and the current-generation disk drives available in the market are terabyte devices. One PC I deal with on occasion has 2.4 terabytes configured, and a particular 1U server I've worked with has 2.25 TB (in 1U). 50 GB just isn't all that much storage.

That written, the quantity of data and the aggregate transfer time doesn't uniquely identify the bottleneck. Factors including aggregate distances and processing windows and the data links are all centrally involved.

Would this data transfer be related to the following thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1114625

The numbers certainly look similar.

[Robert Gezelter]

Kenneth,

In reviewing the thread, I see that information on the systems themselves is missing. Identifying the CPUs and memory capacities would be extremely helpful.

As can be seen from the thread, a variety of approaches can be attempted. The success of each of them will, in the end, depend upon precisely what is bottlenecking performance.

Some of the potential solutions are:
- expand the use of caching for the target disk
- use LD to create an in-memory copy of the target disk; and using host based volume shadowing to then "copy" the disk image to the actual disk
- quota changes to the account being used for the restore (if they are not in conformance with the values recommended for use with BACKUP)
- access to the save sets via local DECnet (to improve RMS blocking/buffering)
- since the disk subsystem is not identified; changes in its configuration could also be possible bottlenecks
- changes in system parameters could be the issue

And there are likely other changes that would improve performance.

What is needed is data about where the actual bottleneck is. Without that, all of us are merely guessing (admittedly, not completely randomly) about what the problem actually is. I have seen cases where the actual problem was none of the above, but was actually a failing drive that was taking large amounts of time to retry and correct soft errors.

What is needed is accurate measurements of the actual situation. Then a prescription can be targeted at what is the actual performance bottleneck.

And, yes, as a disclaimer, we do provide consulting services in this area (as do Hoff, Hein, and several of my other colleagues who regularly comment in ITRC).

- Bob Gezelter, http://www.rlgsc.com

[Sebastian Bazley]

Not sure anyone has mentioned this:

If the disk is set up with HIGH WATER MARKING enabled, this can slow down the creation of files. Likewise ERASE ON DELETE will slow down file deletion.

I'm not sure how much these would affect backup, but it would be worth checking if these are set, and disabling them if they are not needed (which will depend on your site security policy).