VMS Poor SDLT performance

[Guenther Froehlin]

MOUNT/CACHE=TAPE_DATA is only supported for TMSCP speaking controllers: HSC,HSD,HSJ (for those who remember this old stuff).

And before speculating/tweaking the tape side do a backup to the null device to give you an idea where the bottleneck might be. In most BACKUP performance cases I have been involved it was the disk input.

/Guenther

[Cass Witkowski]

Tim,

I'm not understanding what you mean by thrashing of the HBA. From my understanding of Backup there is one buffer that in one phase of backup there is a lot of disk I/Os queued up to get data off the disk and into the buffer. Once the buffer is full then that activity stops and the data is sent from the buffer to the tape drives. Which would give you a sort of see-saw effect of the disk is busy and tape is idle then the tape is busy and disk is idle.

I don't know if backup is capable of double buffering where one buffer is being filled from disk and one is being written to tape. It may be so but the AS1200 may not have enough oomph to do it fast enough to keep up with todays tape drives. I didn't know the AS1200 supported the FC HBAs.

If the CPU is your limiting factor you may have to just play with the buffer size WSMAX WSEXTENT to see what setting gives you the best throughput.

Cass

[Uwe Zessin]

'thrashing' means doing something with a lot of force or behaving violent.

In this situation, BACKUP does a lot of I/O through one fibre channel adapter to the storage array to fill its in-memory buffers (while the tape drive is idle). Next, it throws all data at once over the other fibre channel adapter to the tape drive (while the disks are now idle). After it has finished, BACKUP starts again to excruciate the disks.

[Jan van den Ende]

Uwe,

I think I have to agree with Cass on this.
It is INTENDED BEHAVIOR for Backup to first do ONLY a lot of disk I/O, and thereafter ONLY a bulk of tape I/O.
... and the disk I/O is even divided in two separate phases: first, from Index info, the layout of the buffer is determined, with every file segment's pointing to its logically-contigous location IN the buffer.
Then, ALL those file segments are specified in ONE I/O request (maybe split up again by hardware limits, but that is XQP, not BACKUP) to force the disk into heavy-load-mode, so as to optimise head-movements, ie, fill the predetermined buffer in the order of disk-layout, random with respect to buffer layout, and so reduce the biggest time loss: waiting for head-movements (at the cost of memory and some CPU).

So, if the memory available to backup is not too small, you will see alternating DISK- and tape-I/O's.


Jan

[Tim Nelson]

Sorry for the loose use of the word "thrash".

Thanks again to all as everything stated is right on the money.

I continue to investigate only because I feel this process can be better/faster.
Others in similar environments are seeing 60-80GB/hr not my dismal 16-20GB/hr.

Increasing system resources, i.e. buffers, quotas etc, actually slowed down the process. It did open up the throttle on reads but simply increased the io queue, (actually reading too fast).

I am planning on testing some read speeds by using null as the output device. Just to see what top read speed is.

I also plan on doing some timings to the same tapes from an ES40 to see if my little AS1200 is just going as fast as it can.


Thanks to all for the assistance !!!

[Uwe Zessin]

Sigh, I must be blind or you (Jan) must be seeing ghosts...

I can't see that he said he found the behaviour good (and I don't think I explicitly said I found it bad, either).

And why do you think this behaviour is good when it puts the tape drive into a start/stop mode which makes everything go slower? An SDLT 1 tape drive can do about 11 MegaBytes/second, an SDLT 2 drive can do 16 MegaBytes/second to media. In the storage trainings I got told that for compression you need about 3-times the data rate to keep the tape moving.

I don't see what model of tape drive Tim has. Remember that the little AS1200 is already running at 100% CPU.

[Guenther Froehlin]

In BACKUP's case larger is not always better!

High DIOLM can cause some disk I/O
subsystems to collapse (resets, hangs, slow
downs). DIOLM=100 is more than enough to
allow RAID sets and optimizing disk
controllers to have something to "work with".

WSQUOTA: The larger WSQUOTA the longer it
takes BACKUP to map files into its buffers
sized by WSQUOTA. During this time the tape
drive remains idle. WSQUOTA=(10000..40000)
is an optimum range.

Here's why: Once BACKUP has completed
mapping files into its buffers and has
issued ALL I/Os in a tight loop disk I/Os
start filling the buffers. Once the first
buffer in order is ready to go (files are
stored in the save set in alphabetic
order which is not necessarily the
order files are stored on disk)
it is queued to the tape drive. The larger
WSQUOTA the more "UN"-likely it is that the
first buffer fills first thus delaying I/O
to the tape drive. Tape drives typically
have an onboard cache of multiple MBs. Once
the data is in this cache the host
application (BACKUP) gets a complete I/O
signalled. And BACKUP will not start another
disk scan until all tape I/Os have
completed. So with a nicely small WSQUOTA
the tape drive has still work to-do while
BACKUP thinks the tape drive is
done and starts another disk scan. With a
large WSQUOTA the tape's write-back cache is
nearly unnoticeable.

And really...DO A TEST RUN TO THE NULL
DEVICE!!! It tells whether-or-not the disk
input is a limiting factor.

And do yourself another favor...DO A TEST
RUN /PHYSICAL DISK-TO-TAPE!!! It tells
whether-or-not your overall infrastructure
(drives, busses, network) have a bottleneck.
With an SDLT1 and compaction you should
reach 20+MB/sec at least.

/Guenther

[Uwe Zessin]

> In BACKUP's case larger is not always better!

Well, that's what I already wrote on Aug 8, 2004 11:01:39. It is good to see that a real expert[*] confirms this.

I remember that I wrote something similar to comp.os.vms some years ago and I almost got stoned to death...


It about time that VMS engineering works

- on the documentation for BACKUP tuning.

- on BACKUP itself again to teach it double-buffering.


[*] I do know Guenther - this is not meant cynical!

[Antoniov.]

Uwe,
you posted DIOLM = 2048 not 4096;
Guenther posted DIMLM = 100.
I Guess Tim need know what's the best value for backup.

Antonio Vigliotti

[Guenther Froehlin]

(...hallo Uwe!)

The chapter "Using Backup" in the "System
Manager's Manual" has been updated for V8.2.
I am not sure how much clearer it is now
on this complex issue.

The recommended value for DIOLM is 100 and the recommended start-tuning value for WSQUOTA is 10,000 pagelets.

WSQUOTA is the most sensitive tuning
parameter for BACKUP. DIOLM and FILLM come
next but are far less sensitive. The other
ones just need to be larger to avoid BACKUP
to run into some quota/limit starvation and
take a shortcut or wait to overcome the
temporary shortage.

To make sure the current BACKUP scheme is
not interrupted by running out of some other
quota:

FILLM = 100 (start value, incr. by 10,
should be < CHANNLECNT - 20)
PGFLQUOTA >= WSQUOTA + 25000
ASTLM >= DIOLM + 100
BIOLM >= FILLM + 100
BYTLM >= 256 * FILLM + 6 * DIOLM + 10000
ENQLM >= FILLM + 100

But before you start changing your quotas
and limits...TIME A BACKUP TO THE NULL
DEVICE and TIME A /PHYSICAL to tape!!!

Without these two simple tests you have no
idea about your current limits.

/Guenther

[Jan van den Ende]

Uwe,

Guenther's explanation really makes sense!
Obviously you knew all, but I never before saw the tapedevice-buffering argument. Maybe should have thoufgt of it myself, but the gray stuff obviously is getting rusty...
And I just tried to live by the documentation (always do, until show better, and WHY) So, this make your note on the need to upgrade the doc's VERY relevant.
Until I've seen them, I keep my hopes high after Guenther's 8.2 remark.

All in all, this thread IS a real eye-opener!!


Jan

[Jan van den Ende]

Sorry, hit too quick...

The concept of double buffering REALLY should have occurred to me earlier, I should have had a head-lead on that....

Way back in the very early 80's, before my move to VMS, I even devised and wrote a Qume-printer device spooler based upon the principle (only in those day that ADVANCED printer's buffers were a mighty 128 BYTES!!)

I must be getting old...


Jan

[Uwe Zessin]

Sorry, Antonio, you must be seeing ghosts, too.

Please re-read my respones from 'Aug 8, 2004 11:01:39' and 'Aug 9, 2004 16:01:17'. I cannot see that I suggest DIOLM = 2048, I only state that this is the maximum outstanding I/Os on an EVA's host port. It applies to *all* hosts together on single port! In the second response I also question if the SCSI driver really creates such a high queue on the array. In that case you will only fill you non-paged pool with I/O requests.

Perhaps somebody who is more familiar with the SCSI driver can say how large the I/O queue can be. I think the 'standard' supports up to 255 or 256. HP-UX uses 8, but that can be tuned. Others use between 16 and 32.

[Uwe Zessin]

I forgot:

> I Guess Tim need know what's the best value for backup.

If there were a 'best value' - be assured that VMS engineering would have encoded / documented it - they are sometimes slow (e.g. when keeping documentation current), but not dumb.

-----

Jan,

I don't know if you have noticed ;-), but I work on multiple platforms and have seen other backup software, too.

There is life outside OpenVMS and documentation / online-help is not always correct/current!!

[Dave Gudewicz]

Guenther said:

The chapter "Using Backup" in the "System
Manager's Manual" has been updated for V8.2.
I am not sure how much clearer it is now
on this complex issue.

A link to this chapter when it becomes available please?

Many thanks,

Dave...

[Antoniov.]

Uwe,
I read into your link of August, 8th.

Recommended Process Quotas for Efficient Backups Process Quota Recommended Setting
WSQUOTA
Equal to system parameter WSMAX

WSEXTENT
Equal to WSQUOTA quota

PGFLQUOTA
Equal to or greater than WSEXTENT quota

FILLM
Less than system parameter CHANNELCNT

DIOLM
Either 4096 or three times the value of FILLM quota, whichever is greater

I read DIOLM 4096 or greater.

Antonio Vigliotti

[Antoniov.]

Hi,
I'm confused;

DIOLM - Guenter says start from 100 and this have sense for me while HP documentation reports 4096 or 3 times FILLM.

WSQUOTA - HP reports equals WSMAX

FILLLM - Guenter says CHANNELCNT - 20, most accurate than HP.

I think Guenter method is good but why HP documenation report theese values?

Antonio Vigliotti

P.S.
Here HP link for V7.3-2
http://h71000.www7.hp.com/doc/732FINAL/aa-pv5mh-tk/00/01/119-con.html

[Jan van den Ende]

Antonio,

that looks rather much like MY text!

Jan

[Uwe Zessin]

Let's try to clear it up, Antonio.


What you quoted was the contents from a link to a HP web page - again, *I* did not write or suggest this!!

Apparently you have missed the sentence immediately before the link. Here it is again, so you don't have to scroll up:

*>It looks like another case of were the documentation has not kept up.
->http://h71000.www7.hp.com/doc/732FINAL/aa-pv5mh-tk/00/01/119-con.html

Now, go back and re-read the full text from 'Aug 8, 2004 11:01:39'. I hope you recognize, that I did _not_ suggest DIOLM=4096 nor that the documentation is current. OK?

[Guenther Froehlin]

V8.2 is still too far out. Who knows...I
might have to change the recommendations,
again. But I presented the basic changes in my previous replies.

/Guenther

[Antoniov.]

Uwe,
I didn't post you say ...
Because I read all links in thread I'm amazed HP documentation reports no right informations :-(
In my mind Guenter post has more sense of HP guide but obviously I can mistake.
I'm confused and I guess also Tim can be confused; as Guenter posted, backup is sensible to DIOLM and there is a big difference between 100 and 4096!

Antonio Vigliotti

[Antoniov.]

Uh,
I reread the thread (very long) so I saw Tim set DIOLM=4096 (as suggeste by HP).
Now I think Tim can reduce this quota and increase as Guenther's hint.
Jiri hints DIOLM=200.

Antonio Vigliotti
P.S.
Guenther, sorry I mistaked your name.

[Uwe Zessin]

I think you're trying to split hairs. It is true that you did not write I 'said', but you wrote I 'posted'.

> Sep 16, 2004 16:02:11 GMT
...
>Uwe,
>you posted DIOLM = 2048 not 4096;

That is semantically no difference to me - you could also have claimed that I 'wrote something'.

-----

I just checked my responses, but I don't see I made _any_ value suggestion for a system parameter or process quota. And I don't intend to do so - as Guenther does understand this way better than me.

[Gary Cantwell]

tinyurl'd Antonio's link to clean thread formatting,

Thanks,

Gary

[Antoniov.]

Uwe,
I don't like this kind of questions, however because you persist we can still post about my posts and/or you posts :-(

It's true you never posted about system parameters, sorry for my funny english.
But I wish notice you posted

Sep 16, 2004 15:52:55 GMT
> In BACKUP's case larger is not always better!

Well, that's what I already wrote on Aug 8, 2004 11:01:39. It is good to see that a real expert[*] confirms this.

and in you you mentioned there is a link to HP documentation where HP printed DIOLM 4096 blah blah blah
So any reader if your post can confused when you post you idea and you link report opposite values.
May be I cannot write right english but I don't see any ghosts and my hairs are not splitted :-O

Cheers
Antonio Vigliotti