topic Re: speed up backups in Operating System - OpenVMS

speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 18:37:14 GMT

I have an openvms 7.3-2 system connected to XP1024 storage via 2 HBA. Disks are balanced between the HBA (half and half - not load balanced - harder to find that out). I am trying to speed up backups that are disk to disk backups.

I have a test file that I am using that is 15gig. It is taking approximately 26 minutes.
I have been changing the backup account, quotas, working set values, etc to see if I can speed this up any.

I have even had a HP VMS person looking at VMS and another checking the storage & SAN with nothing showing as a bottle neck.

Was hoping someone here might have ideas of what else to try.
I have checked out page on hoffmanlabs site. Tried those quotas.

Currently the account is:
Fillm: 2032 Bytlm: 556768
Shrfillm: 0 Pbytlm: 0
BIOlm: 2032 JTquota: 4096
DIOlm: 6096 WSdef: 1024
ASTlm: 6116 WSquo: 75000
TQElm: 200 WSextent: 100000
Enqlm: 2034 Pgflquo: 1074048

Re: speed up backups

Robert Gezelter — Tue, 27 Oct 2009 18:54:58 GMT

Peter,

What is the command line that is being used? Is the output a save set?

If it is, check the increment that it is extending.

- Bob Gezelter, http://www.rlgsc.com

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 19:22:45 GMT

Basic command
backup disk:[dir]file.img scratch1:[dir]file.img

Not save set.

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 19:29:35 GMT

FYI - Initial file is only 2 extents. So not a fragmented file.

Re: speed up backups

Hein van den Heuvel — Tue, 27 Oct 2009 19:46:43 GMT

>> 15gig. It is taking approximately 26 minutes

So that's just 10mb/sec. Not much!

Speaking of not tot much, I don't have much time today, so just a few quick hints.

> DIOlm: 6096 WSdef: 1024
> ASTlm: 6116 WSquo: 75000

Those are pretty ridiculous 'limits' and Backup will try to use them, and scare the living daylights out of the storage system cache controller and fibre channel arbitration control for the queue depth.

The backup folks have finally also realized that and throttled down some.
On 8.3 there is the /IO_LOAD to throttle it down to say 4 or 8.

If that switch did not make it to 7.3-2 then I would recommend a re-test with DIOLM = 10, just for grins. (check PQL minimums).

Google for FC queue depth topics like:
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1143614

There were some OpenVMS driver issues where the back-off / ramp-up mechanisme was broken. Once you were slapped on the wrist, it would go very very slow.

note... I believe that the RMS settings like EXTEND/BLOCK/BUF are only important when creating savesets.

Finally, tries COPY as a baseline?
Again the 8.3 Copy has a nice large block size defaul, and a switch to controll it. Not sure that made it back to 7.3-2. Worth a try!

Good Luck,
Hein/

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 19:52:03 GMT

PQL diolm is 100

I did read up on the io_load capability in the new OS. Unfortunately it will be awhile still before I can upgrade.

From the log file:

Process Quotas:
Account name: SYSTEM
CPU limit: Infinite Direct I/O limit: 100
Buffered I/O byte count quota: 555680 Buffered I/O limit: 2032
Timer queue entry quota: 199 Open file quota: 2030
Paging file quota: 1068208 Subprocess quota: 20
Default page fault cluster: 96 AST quota: 6115
Enqueue quota: 2034 Shared file limit: 0
Max detached processes: 0 Max active jobs: 0

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 19:57:09 GMT

Wish formatting was kept when actually submitting.
Hope this looks better.

Re: speed up backups

Hein van den Heuvel — Tue, 27 Oct 2009 20:19:27 GMT

Yeah, attachments with the real data is the way the go in this forum.

it helps a little to check: [x] Retain format(spacing).
(before hitting the submit... I always get that wrong :-)

So create yourself a play account for a while, not system.

That PROCESS QUOTA picture is static right?
Tyr $SHOW PROC/CONT and with the Q character for the dynamic picture... or is that also 8.3 ?
There is always SDA to check behind the scenes.

How much resources did backup use? (IO count, CPU) ?
Can you figure out the IO size and such from there...

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 20:49:59 GMT

the show process/quota is static right before the backup starts.
I then use the quota.com I got from DEC and monitor it.

I had to use a different system in the cluster to modify for pql_mdiolm = 10.

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 20:57:19 GMT

Can play with account quotas. Read up on documents. Linking for references:
http://h71000.www7.hp.com/doc/73final/6017/6017pro_045.html

http://labs.hoffmanlabs.com/node/49

Re: speed up backups

Peter Zeiszler — Tue, 27 Oct 2009 23:05:51 GMT

Performed a copy. Copy was about 21 min.
Modified pql_mdiolm to 4 and account to 4.

Re: speed up backups

Volker Halle — Wed, 28 Oct 2009 06:46:09 GMT

Peter,

if you use MONITOR DISK/INT=1, do you get a steady value for I/O operation rate on the source and destination disk ? How about MONI DISK/ITEM=Q/INT=1 ?

Did you check the 'QF seen' counters for the FC HBAs ?

$ ANAL/SYS
SDA> FC SHOW DEV FGA ! also for FGB
SDA> FC SHOW STDT

Volker.

Re: speed up backups

Peter Zeiszler — Wed, 28 Oct 2009 14:26:09 GMT

Monitor disk - shows queue of 1.
Only getting in the 250-400 i/o operations. Steady numbers on those 2 disks as well as other disks (normal operations are running on the systems).
DSA4001: BACKUP2 394.00 309.21 197.00 394.00
DSA5005: SCRATCH1 334.00 309.84 255.00 342.00

anal/sys fc show stdt no change in the qf seen. Only number changing is the Dev I/O. I remember looking at this last week.

Re: speed up backups

Volker Halle — Wed, 28 Oct 2009 15:14:37 GMT

Peter,

so where's the bottleneck then ? Read or Write ?

Try BACKUP source_dsk:file NLA0:dummy.sav/SAVE

and see how fast you can read from the source disk.

Volker.

Re: speed up backups

Volker Halle — Wed, 28 Oct 2009 15:37:31 GMT

Peter,

a simple example:

BACKUP a file from a 2-member shadowed local SCSI disk on a rx2600: 800+800 = 1600 IO/sec.

BACKUP of the same file from the same source shadowset disk to another local single-member SCSI disk shadowset: about 200-300 IO/sec.

Where do you think the bottleneck is ?

Volker.

Re: speed up backups

Volker Halle — Wed, 28 Oct 2009 16:04:29 GMT

Peter,

and the throughput was:

Backup of 1 GB file disk-to-disk: 6.6 MB/sec

SYSGEN CREATE file of same size on dest disk(high-water-marking on): 8.6 MB/sec

BACKUP file to a saveset: 3.7 MB/sec
(with SET RMS/EXT=65535/BUFF=127/BLOCK=32)

Volker.

Re: speed up backups

Peter Zeiszler — Wed, 28 Oct 2009 16:25:44 GMT

Volker - great suggestion. When dumping to NL: I am getting only 300ish I/O out of the disk.

Tried mounting a device privately to the system (so its not going across to other systems in the cluster). I did the backup to NULL and it peaked at 450 to start but then settled around 250 i/o. This was using normal account - not the one tweaked with lower DIOLM.

Re: speed up backups

Peter Zeiszler — Wed, 28 Oct 2009 16:30:57 GMT

FYI -
Backup2 is a 2 member shadow set. Disks exist on different XP frames.
Scratch1 is a single member shadow set.

NULL test was on the single member shadow set. Tried 2 different systems.
Disks were originally created with nohighwater.

Re: speed up backups

Volker Halle — Wed, 28 Oct 2009 16:58:20 GMT

Peter,

so the bottleneck in your case seems to be in the Read path - unusual !

Volker.

Re: speed up backups

Peter Zeiszler — Wed, 28 Oct 2009 17:08:29 GMT

quick test on local scsi to null - 2400 i/o sec.
Setting up for scsi to scsi.