topic Re: Need to speed up expansion of very large savesets in Operating System - OpenVMS

Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 11:17:13 GMT

I have sets of savesets that need to be expanded simultaneously. 8 savesets are 130 GB. It took 5 full 24 hour days for these savesets to fully exspand. Is there a way to speed up this process? Someone said to use the "mount" command with the following qualifiers "/blocksize & /cache" along with the "mount" command. Is this the best solution?

I am using SCSI drives for my source and destination. The source drive contains the savesets that range from 14 to 50 GB each. I am using OpenVMS 7.2

Does anyone have any ideas?

Re: Need to speed up expansion of very large savesets

Robert Gezelter — Sat, 28 Apr 2007 11:43:46 GMT

Ken,

By expand, do you mean restore?

Have you measured what the bottleneck really is?

If you want to talk about this privately, please contact me via email.

The BLOCKSIZE parameter on disk mount is unlikely to have an effect.

Caching will also likely not help, as the data is not being reused.

I would consider increasing the RMS parameters, and possibly using a local DECnet connection, HOWEVER, I would not recommend random experimentation. It is far easier to see what the bottleneck actually is and address the problem directly.

- Bob Gezelter, http://www.rlgsc.com

Re: Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 11:59:50 GMT

Yes, I mean restoring savesets. One problem is the tri-link SCSI connector. But there is nothing we can do about that. What I need is a bonefied way to harness the true power of OpenVMS to accelerate the restoration process. What parameters can be set for RMS to fix this? I have backups to my system disk, so, experimentaion is not a problem.

Re: Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 12:10:51 GMT

C.2.1 Sequential File Access
To make t he best use of DECdfs's quick file access, most applications benefit from default RMS multibuffer and multiblock values of 3 and 16, respectively, when accessing sequential files.
Set the number of buffers to 3 for the most efficient multibuffering of file operations. Use the following DCL command:

$ SET RMS_DEFAULT/BUFFER_COUNT=3 /DISK
Next, set the size of each buffer to sixteen 512-byte blocks:

$ SET RMS_DEFAULT/BLOCK_COUNT=16
To set these values for just your user process, you can include the commands in your LOGIN.COM file. To set them on a systemwide basis, you can add the /SYSTEM qualifier a nd include the commands in the DFS$SYSTARTUP file.
RMS multibuffer and multiblock values that are larger than the default values can slow performance by allowing the application to exceed the DECnet pipeline quota. However, these values are recommendations that may not be optimal for every application. If your application opens many files or if it has a small working set size, you may find these values are too large.
Note
________________________________________
If you prefer, you can set the RMS default multibuffer value by using the SYSGEN parameter RMS_DFMBF. You can set the RMS default multiblock value by using the SYSGEN parameter RMS_DFMBC.

Is this what you were talking about?

Re: Need to speed up expansion of very large savesets

Robert Gezelter — Sat, 28 Apr 2007 12:11:20 GMT

Kenneth,

Actually, I suspect that the SCSI cabling is not the problem.

What does the MONITOR show on the disks involved in the operation?

- Bob Gezelter, http://www.rlgsc.com

Re: Need to speed up expansion of very large savesets

Robert Gezelter — Sat, 28 Apr 2007 12:16:19 GMT

Kenneth,

The BUFFER and BLOCK factors do help, if BACKUP actually honors them in your context. When I do backups, various versions of BACKUP do not honor the RMS defaults.

To force the use of the RMS defaults, I use local DECnet to access the file via transparent file access. The File Access Listener (FAL) DOES honor the RMS settings (although to get them set for the FAL process requires that they be set in the LOGIN.COM file (I generally condition them on F$MODE() .eqs. "NETWORK").

The RMS parameters are not only for the use of DECdfs (which I presume you are not using).

- Bob Gezelter, http://www.rlgsc.com

Re: Need to speed up expansion of very large savesets

Robert Gezelter — Sat, 28 Apr 2007 12:18:22 GMT

Kenneth,

A small note: I will be leaving my terminal in a few minutes. While I will not post the number for my mobile in the forum, I will be happy to give it to you if you send me an email.

Please remember to remove the "REMOVETHIS" from the email address on my contact page.

- Bob Gezelter, http://www.rlgsc.com

Re: Need to speed up expansion of very large savesets

Hein van den Heuvel — Sat, 28 Apr 2007 12:56:41 GMT

>> $ SET RMS_DEFAULT/BUFFER_COUNT=3 /DISK
>> $ SET RMS_DEFAULT/BLOCK_COUNT=16

Not good enough. 16 blocks had been the defautl for decades. The new default is actually 32. So 16 would be a step backward.
I would suggest 64 as a general default ( /SYSTEM ) and 120 for the process doing the restore.

A multi buffer count of 2 is normally enough to make the sequential file write behind work. But for a I'd err on the save side and set it to 4.

And while you are there... why not SET RMS/SYSTEM/EXTEN=1000 (or more). Better safe than sorry for this one. Overallocated sequential file will be truncated back to the minimum clusters needed on close.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

Re: Need to speed up expansion of very large savesets

Jon Pinkley — Sat, 28 Apr 2007 13:19:54 GMT

Kenneth Toler,

In summary I would recommend using LDDRIVER and transferring the container files: details follow.

Can we assume this is related to your previous questions?

PKZIP for VMS vs backup/log http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1114625

"The savesets are now in excess of 500,000 files with more than 1,000,000 close at hand."

If these 500,000 files total to 130 GB, then the average file size is around 250 KB. or about 500 blocks each.

Total Number of Files and Blocks inside savesets http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1121642

If so, then it seems you are restoring a large number of files. And possibly to a directory that has many thousands of files. These are both operations that VMS is not optimized for

You don't specify what the target disk(s) are (other than SCSI), is it on an HSZ controller? (from reference to tri-link adapter). If it is, then writeback caching will help, but not as much as the solution that follows. Specifically the use of LDDRIVER and container files.

First, if you were restoring multiple savesets to a single "spindle", then I would expect you to get better performance by single threading the restores. This seems counterintuitive, but by increasing the number of files you are restoring concurrently, you will cause the heads to do additional seeking, and that will kill your performance. If all your files are 1 block in size, it probably won't make much difference, you are going to get poor performance if you try restoring on a file-by-file basis no matter what you do.

Here is an option. Use physical backups to sequential savesets on the portable drive and physical restores on the destination machine. If you don't have identical drives on both sides, use the LDDRIVER (on the freeware or directly from Jur van der Burg's site, see thread http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1100345 LDDRIVER allows you to make a single file appear like a disk, which can be mounted. This would
allow you to create identically sized virtual disk LDAxx units on both systems, a requirement for backup/physical in older versions of VMS.

If both the customer and you are willing to load the LDDRIVER, then you have another even easier option.

Use an LD device to create you files on. When it is time to provide a new "snapshot", dismount the LD device and backup/ignorie=(nobackup) or copy the file to the SCSI drive to be taken to the customer site. If you ever upgrade, you will need to make sure you disable caching on the LD container file, as is mentioned in the release notes. Take the SCSI drive to the customer site and copy the container files off the SCSI drive. THIS WILL BE MUCH FASTER than file-by-file restores.

When the container files are copied to the customer's disk, use LD connect to create a new LD devices, specifying the new container files.

Mount the LD devices. You are done.

The LDDRIVER is a very thin (low overhead) intercept driver, so you can use it for production use.

Re: Need to speed up expansion of very large savesets

Hoff — Sat, 28 Apr 2007 14:01:10 GMT

Please identify the VAX or Alpha host here, the SCSI controller(s), and the details of the I/O path out to the target device(s).

The OpenVMS BACKUP performance (as was measured within OpenVMS Engineering, and discussed in various presentations), when using the recommended process quota settings and baseline tuning, has been found to be within 90% of the underlying hardware bandwidth; within 90% of the slowest component from source to target.

The OpenVMS releases after V7.2 can and do have a number of performance optimizations (including various changes relevant to BACKUP and to I/O), but I'd (also) be looking carefully at the performance of the hardware involved. The host, the SCSI controller, bus contention, and the device.

Extent settings and such and process quotas and such can and certainly do help with performance, but I'd also be looking really, really, really hard at things like I/O bandwidth all the way out to the target, and disk queue depths. And at the host VAX or Alpha processor here.

Operator process quotas are listed in the tables available at: http://64.223.189.234/node/49

I'd also be looking at the specific processing requirements here, to are see if there are process changes that might be relevant.

And a reversal of interpretation on why you really want to look at hardware bandwidth: if you're achieving within approximately 90% of the bandwidth of the slowest I/O giblet, I'd be looking at hardware upgrades, and not at performance tuning.

Stephen Hoffman
HoffmanLabs LLC

Re: Need to speed up expansion of very large savesets

Jon Pinkley — Sat, 28 Apr 2007 15:12:07 GMT

Let's look at the numbers provided:

130GB restored in 5 days.

Assuming larger interpretation of 130 GB

130 * 2^30 = 130 * 1073741824 = 139586437120 bytes

5 days = 5 * 24 * 60 * 60 = 432000 seconds

So average throughput is 139586437120 byte/432000 seconds or just under 325000 bytes/second.

So you do need to find out what the bottleneck is.

Since you already have some large files on the SCSI disk, can you measure the time it takes to copy the 14GB saveset to a disk.

You may be able to increase the performance some by adjusting the process RMS block and buffer counts, but both oth those utilities are smart enough to allocate the space in one go, and then copying the blocks, so setting the RMS extend size probably will not affect copy performance. It definitely helps when creating a file that is going to be large.

One you determine the speed that you can copy a large file, you will have a baseline for whoat you could expect if using the LDDRIVER method.

Note that by using LDDRIVER the performance will be independent of the size or number of files contained in the container file. The smaller the files are, the more benefit you will get from using LDDRIVER to treat a single file as a disk.

Re: Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 15:22:46 GMT

Actually the OpenVMS version is 7.3 not 7.2. We are about to go to 8.3, but am not sure when.

The system is alpha based now, not VAX.

There are 50,000 to over 500,000 files per saveset. The 500,000 is the 50 GB saveset.

Any other ideas?

Re: Need to speed up expansion of very large savesets

Jon Pinkley — Sat, 28 Apr 2007 15:23:50 GMT

Maybe I wasn't specific enough. By copy the saveset I mean copy the single file, not use backup to restore the individual files.

for example:

$ copy dkb5:[savesets]disk1.bck dka3:[somedir]

or

$ backup dkb5:[savesets]disk1.bck dka3:[somedir]/interchange/own=par ! this copies the file disk1.bck, it does not expand to individual files

Re: Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 15:29:36 GMT

I am not neccesarily needing to copy the saveset. I need to greatly improve the performance of expanding large savesets with 100s of thousands of files each.

Re: Need to speed up expansion of very large savesets

Robert Gezelter — Sat, 28 Apr 2007 16:40:40 GMT

Kenneth,

As I noted earlier, it is very far from clear, without performance data, where the bottleneck actually is.

Looking at the followup postings, it would not be surprising if the issue is not the access to the savesets themselves, but the actual output files themselves. Information about the actual performance of the system is needed to understand what the bottleneck is, and then how the problem can be addressed.

- Bob Gezelter, http://www.rlgsc.com

Re: Need to speed up expansion of very large savesets

Jon Pinkley — Sat, 28 Apr 2007 16:44:06 GMT

My point is that by using LD container files, you will not need to recreate millions of files. That is a very slow operation on VMS. By using LDDRIVER you can move the whole disk filesystem as a single file.

If a single spindle is sufficient, then they could just access the files directly on the disk you provide. If they have HBVS licensed, you could even use that to make the files available while they were being copied (via the shadowing software), although while it is copying, the performance is going to be poor for other users, and the amount of I/O to get the files moved will be higher (I think at least double). That would be good if you needed availitly instead of raw performance.

The purpose of copying the saveset is just to measure the "best case" speed. Copying a single large file is going to be faster than copying a whole bunch of small ones. It doesn't have to be a saveset, just some large file. If the time it takes to copy the data as a single file is too long, then you need to look at other options, like upgrading the system.

You haven't provided any metrics for what type of performance they are expecting once the files are loaded on the customers system, or even what you consider to be acceptable time to restore. If you are expecting to be able to restore 130 GB in 1 hour, that's nearly 40 MB/sec, and I doubt your are going to be able to achive that coming from a single spindle (even if you were restoring to a RAM disk). Hein can probably provide soem good guidelines for what hardware is needed to support sustained raw transfer ratess.

Without you giving us more info, we really can't provide an optimaal solution. I don't think you are going to find a magic bullet that will be best for any arbitrary case.

If you are interested in considering the LD driver approach, I can give a few more details, otherwise, I'll let others give you their favorite tuning tweaks.

If you think you may be interested, please visit http://www.digiater.nl/lddriver.html for a good description of what it is.

Re: Need to speed up expansion of very large savesets

Hoff — Sat, 28 Apr 2007 17:29:33 GMT

>>> My point is that by using LD container files, you will not need to recreate millions of files. That is a very slow operation on VMS. By using LDDRIVER you can move the whole disk filesystem as a single file.<<<

I don't understand this. You still have to create the files inside the LD device, right? And LD still has to write through to the underlying disk, right? Yes, LD has a single file system, but if you're restoring a saveset into individual files you're still working with a file system inside the LD device. (And LD isn't a RAM-based disk; it does write back to disk for the blocks written into the LD device.)

Re: Need to speed up expansion of very large savesets

Kenneth Toler — Sat, 28 Apr 2007 17:36:09 GMT

What is LDDRIVER?

If it is not part of the as-delivered OpenVMS 7.3, I can't use it. If it is part of OpenVMS 7.3, please tell me how to access and use it.

Re: Need to speed up expansion of very large savesets

Jon Pinkley — Sat, 28 Apr 2007 17:47:29 GMT

Re: Hoff's question.

Reading between the lines, this appears to be an ongoing requirement.

If he used an LD device on his system, the files would be created in the container file. This it would be easy to transport the "disk" to the other system as a single file.

So Hoff is corrrect, it doesn't solve the problem of getting from a saveset to the disk. However, it avoids the need in the future.

Tim: How can I make my car go faster?
Tom: Why does it need to go faster?
Tim: I need to drive 2000 miles, and it takes too long.
Tom: Did you consider taking a plane?

Re: Need to speed up expansion of very large savesets

Hoff — Sat, 28 Apr 2007 17:57:56 GMT

LDDRIVER is part of OpenVMS Alpha V7.3-1 and later (with some wrinkles), and fully part of V8.2 and later. LDDRIVER is also available via the Freeware, and (among other uses) central to recording optical media on OpenVMS.

Information on LD is available in the V8.2 and later OpenVMS manuals, and details on configuring and using LDDRIVER are also embedded in the recording details at: http://64.223.189.234/node/28

LD would work as a way to pass over a block of data -- for this volume of data, do ensure you have the current LD ECO, as older drivers had a bugcheck with volumes in this range. As for approaches similar to LD, I'd be tempted to perform the BACKUP or gzip locally, then transfer the saveset or the archive as a unit. Or work on compressing or reducing the quantity of data involved. Or both.

Though pending the information on the system and I/O hardware and on the particular limitation or bottleneck here between the source and the target, I'm only really left to speculate.