BACKUP over DECnet, file extensions, performance

[EdgarZamora_1]

I inherited a script that basically did a BACKUP over DECnet (i.e. BACKUP dev:[dir]*.* rnode::rdev:[rdir]saveset.bck/save) (notice use of DECnet proxy) The files being backed up are mostly huge RMS indexed files. There are 6 directories that this script backs up (to different savesets) and it did so one at a time. Total elapsed time about 3 hours.

I thought I could decrease that elapsed time so I had this brilliant idea to just fire off six separate procedures simultaneously since the network guy said we had great bandwidth. Total elapsed time: over 24 hours! After much digging I determined it was file fragmentation (all savesets were being written to the same remote disk, which had lots of free space and a free space fragmentation of 0.) The savesets were constantly being extended and competing with each other. DFU gave me a file fragmentation index of 643395904.000 (poor) whereas it was close to 0 previously.

I'm not ready to give up on running multiple backups (and other than writing to separate remote disks) does anyone have any better suggestions? I saw a previous note... http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1248535 where Bob Gezelter recommends setting rms/extend on the remote node. I'm going to test that later on, but I'm not really thrilled about having to add that set rms/extend command in the login.com (for network jobs) because there could be other network jobs for the account that could be doing other stuff.

Any helpful comments/ideas are most welcome.

[Steven Schweda · Accepted solution]

I don't see a way to pre-allocate a big save
set file, but perhaps it would make some
sense to create a big LD device on the
destination system for each job, and write
the output save set to that. At least the
jobs wouldn't be fighting over allocations on
the same file system. Also, MOUNT /EXTENSION
(or SET VOLUME /EXTENSION) might let you set
a bigger default extension value on an LD
volume without bothering LOGIN.COM anywhere.

You would need to guess a good LD device
size ahead of time. Too small, and the job
fails. Too large, and you've tied up a bunch
of extra space, and you might need to copy
the save set off the LD to somewhere else to
be able to recover it.

> Any helpful comments/ideas are most
> welcome.

If you insist on "helpful", it gets harder.

[Hein van den Heuvel]

>> but I'm not really thrilled about having to add that set rms/extend command in the login.com

Don't be afraid to set SET FILE/EXTEN to the max (65K), or at least set it to several thousands which is likely to be 100 times better then it is.

Most/many usages of the extent for new files creates will truncate when done.

But admittedly, if you later have hundreds of little files come in by FTP then you want to select a middle of the road value (like 1000?).

Maybe you can use the time of day for a clue as to how to set the extent, or the originating node for the connection?

You probably also want to yack up SET RMS/NETWORK)BLOCK_COUNT

>> The files being backed up are mostly huge RMS indexed files.

You may want to question the value-add of using BACKUP for a few huge files.
You may be better of just using COPY or even CONVERTing to a remote sequential file or PULLING the file instead of pushing, to control the output better (pre-allocate).

Finally, going from 1 job to 2 concurrent ones might just give you ample improvement over what you had whilest avoiding teh worst of the contention.

Hope this helps some
Hein.

[Steven Schweda]

> > but I'm not really thrilled about having
> > to add that set rms/extend command in the
> > login.com

> Don't be afraid [...]

Or set up a new account with its own
LOGIN.COM, and the appropriate proxies to let
you use it. Then go wild.

[Hoff]

Tweak the receiver DCL, or the FDL.

http://64.223.189.234/node/598

[Robert Gezelter]

Edgar,

I have recommend three things using a SET RMS command conditioned on a NETWORK login (using F$MODE() to check):

- EXTEND=66535
- BUFFER=255 (Hein will disagree with me)
- BLOCK=127 (Hein will disagree with me)

These are the maximum numbers, you can experiment with lowering them based on observed performance. I presume that memory is not a problem. Paging and working sets may also need to be increased.

As to the issue of other jobs, there are a variety of ways I could see conditionalizing things more precisely (one of which is cloning the account and using a special account for the BACKUP operations). One could also possibly do something other tricks, I would have to check into the details.

And yes, I have seen impressive speedups by varying these settings, even when using DECnet remote file access within a node.

- Bob Gezelter, http://www.rlgsc.com

[Jon Pinkley]

Edgar,

What bottleneck are you trying to avoid by having multiple streams?

If all the input directories are on the same device (not clear if they are or not), and there is a single output device that will hold all the save sets, you may be increasing the time due to increased head movement on the source and destination disks.

Large file extensions and network buffer tuning will help whether you have multiple streams or not.

I am not convinced that fragmentation is necessarily the cause, I would guess it is the frequent extensions, the multiple streams reducing the effectiveness of the extent caches, and the contention for the single output disk. By creating the LD devices, you can eliminate the fragmentation, but the disk with the container files will still trash as the heads seek from one container file to another.

Since you are backing up "mostly huge" files, if the backup job is the only thing accessing the drive, I would expect single stream to be near optimal as far getting the data off the disk. By multi-streaming, you may be able to get more than your fair share of network bandwidth, but unless you are competing with other network activity, the only advantage multi-streaming can provide is more buffering. And if you can increase the buffers available to the single stream, you may be able to get higher utilization of the network with a single stream.

Summary: multi-streaming is not always better than single streaming, especially when the streams are contending for a common resource.

Jon

[Hein van den Heuvel]

>> - BUFFER=255 (Hein will disagree with me)

Yes I do, but I should re-run my tests some day soon.

Still, as a mental excercise, how could anything more than a hand full of buffers possibly help with sequential access?

IMHO all that does is, firstly set false expectations, secondly eat some memory and, and finally increase risks a tiny bit.

Just imagine an infinite fast network as well as source. Those buffers at full size will allow the receiver to fill up 16MB of memory. Then what? They still need to be written to the disk and the connection should stay until that is done. So now you potentially launch 255 IO's. Is that going to help anyting? Do you want to blow out of DIRIO? did you want to change from a simple sequential write pattern to effectively random? Do you like spiking you controller cache and hurting other users instead of throttling?

Now nothing is infinitely fast.
In reality you will end up using 2, maybe 3 buffers. If that's the case, then why confuse the world by suggesting that hundreds of buffers will help!
Due to the transient nature of tests, anything will be hard to prove.
Maybe $SET PROC/SSLOG ?

If 255 buffers were to be used for real then RMS would have to walk its buffer Descriptors (BDB's) to find the right one to use. Those are linked in VBN order, so that would get harder and harder. In my prior tests walking 255 BDBs actually took measurable time, when done for each record added. It became slower than doing the IO!
Fortunately, you typical Network IO is done in sequential only (SQO) mode, and RMS will forgot about the buffer right after the IO as the application promissed not to look back.

>> - BLOCK=127 (Hein will disagree with me)

I'm fine with that, allthough I tend to pick 124 'just in case'.

Some tests suggest that writing in multiples of 16 block will reduce XFC overhead a little and thus increase trhoughput, and it does. Blocks=112 is the best you can do for that. But it is hard to argue with doing 10% fewer IOs.

Other tests suggest trying to keep IO started at 4 (16?) block LBN numbers will help the IO controllers (notably EVA).
But to accomplish that the cluster size and buffer size must both be 4 (16?) block multiples. My 124 choice helps with imporoving those odds, while not increasing the number of IOs too much.

fwiw...
I recently started experimenting with changing VCC_MAX_IOSIZE to 126.
This allows one to choose the simple SET FILE/RMS/BLO=127 as a tool to bypass the XFC for those places where you do not come back the data soon.

Cheers,
Hein.

[Wim Van den Wyngaert]

You can use T2T.

backup files node::"task=x"/sav/b;ock=32256/group=0

and x.com on the other side

convert/fdl=sys$input disk_file.sav
xxx

where xxx is an fdl contents of your backup file with big allocatioin and extend size.

Wim (not tested it, using something like this it for remote backup to tape)

[Robert Gezelter]

Hein,

With all due respect, I will not disagree with you that 255 buffers almost never get used. The key word in the preceding is ALMOST.

While I have not run them in a while, I did run some interesting timing tests in an environment where head movement was a given (e.g., a single user disk environment where the application was doing one-for-one processing of a sizable sequential file). The number of buffers used, even for large block sizes was impressive.

The preceding caused that relatively show CPU, with comparable disks, to range from 10% utilization to saturation, with the only variable being the number of buffers and their size.

Admittedly, that environment did stabilize short of the maximum, but it was very sensitive to the performance of the different elements of the system. In that case, the speed match was between the disk and itself, allowing for fragmentation, window turns, and file extends.

Edgar has shared little with us on the precise configuration of the systems and network involved in this. Even with large extend sizes, BACKUP is impressive at generating a data stream. In my (admittedly limited) experiments using DECnet within a node (effectively an infinite speed network), I have certainly maxed out at more than a handful of buffers, although I do max out the CPU running BACKUP.

As is said with vehicle mileage stickers, "Your mileage may vary". If there is one thing I have learned in all of my years of performance tuning, for systems and applications, one must be prepared to be surprised by the unexpected.

- Bob Gezelter, http://www.rlgsc.com