speed up backups

[Peter Zeiszler]

I have an openvms 7.3-2 system connected to XP1024 storage via 2 HBA. Disks are balanced between the HBA (half and half - not load balanced - harder to find that out). I am trying to speed up backups that are disk to disk backups.

I have a test file that I am using that is 15gig. It is taking approximately 26 minutes.
I have been changing the backup account, quotas, working set values, etc to see if I can speed this up any.

I have even had a HP VMS person looking at VMS and another checking the storage & SAN with nothing showing as a bottle neck.

Was hoping someone here might have ideas of what else to try.
I have checked out page on hoffmanlabs site. Tried those quotas.

Currently the account is:
Fillm: 2032 Bytlm: 556768
Shrfillm: 0 Pbytlm: 0
BIOlm: 2032 JTquota: 4096
DIOlm: 6096 WSdef: 1024
ASTlm: 6116 WSquo: 75000
TQElm: 200 WSextent: 100000
Enqlm: 2034 Pgflquo: 1074048

[Robert Gezelter]

Peter,

What is the command line that is being used? Is the output a save set?

If it is, check the increment that it is extending.

- Bob Gezelter, http://www.rlgsc.com

[Peter Zeiszler]

Basic command
backup disk:[dir]file.img scratch1:[dir]file.img

Not save set.

[Peter Zeiszler]

FYI - Initial file is only 2 extents. So not a fragmented file.

[Hein van den Heuvel]

>> 15gig. It is taking approximately 26 minutes

So that's just 10mb/sec. Not much!

Speaking of not tot much, I don't have much time today, so just a few quick hints.

> DIOlm: 6096 WSdef: 1024
> ASTlm: 6116 WSquo: 75000

Those are pretty ridiculous 'limits' and Backup will try to use them, and scare the living daylights out of the storage system cache controller and fibre channel arbitration control for the queue depth.

The backup folks have finally also realized that and throttled down some.
On 8.3 there is the /IO_LOAD to throttle it down to say 4 or 8.

If that switch did not make it to 7.3-2 then I would recommend a re-test with DIOLM = 10, just for grins. (check PQL minimums).

Google for FC queue depth topics like:
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1143614

There were some OpenVMS driver issues where the back-off / ramp-up mechanisme was broken. Once you were slapped on the wrist, it would go very very slow.

note... I believe that the RMS settings like EXTEND/BLOCK/BUF are only important when creating savesets.

Finally, tries COPY as a baseline?
Again the 8.3 Copy has a nice large block size defaul, and a switch to controll it. Not sure that made it back to 7.3-2. Worth a try!


Good Luck,
Hein/

[Peter Zeiszler]

PQL diolm is 100

I did read up on the io_load capability in the new OS. Unfortunately it will be awhile still before I can upgrade.

From the log file:

Process Quotas:
Account name: SYSTEM
CPU limit: Infinite Direct I/O limit: 100
Buffered I/O byte count quota: 555680 Buffered I/O limit: 2032
Timer queue entry quota: 199 Open file quota: 2030
Paging file quota: 1068208 Subprocess quota: 20
Default page fault cluster: 96 AST quota: 6115
Enqueue quota: 2034 Shared file limit: 0
Max detached processes: 0 Max active jobs: 0

[Peter Zeiszler]

Wish formatting was kept when actually submitting.
Hope this looks better.

[Hein van den Heuvel]

Yeah, attachments with the real data is the way the go in this forum.

it helps a little to check: [x] Retain format(spacing).
(before hitting the submit... I always get that wrong :-)

So create yourself a play account for a while, not system.

That PROCESS QUOTA picture is static right?
Tyr $SHOW PROC/CONT and with the Q character for the dynamic picture... or is that also 8.3 ?
There is always SDA to check behind the scenes.

How much resources did backup use? (IO count, CPU) ?
Can you figure out the IO size and such from there...

[Peter Zeiszler]

the show process/quota is static right before the backup starts.
I then use the quota.com I got from DEC and monitor it.

I had to use a different system in the cluster to modify for pql_mdiolm = 10.

[Peter Zeiszler]

Can play with account quotas. Read up on documents. Linking for references:
http://h71000.www7.hp.com/doc/73final/6017/6017pro_045.html

http://labs.hoffmanlabs.com/node/49

[Peter Zeiszler]

Performed a copy. Copy was about 21 min.
Modified pql_mdiolm to 4 and account to 4.

[Volker Halle]

Peter,

if you use MONITOR DISK/INT=1, do you get a steady value for I/O operation rate on the source and destination disk ? How about MONI DISK/ITEM=Q/INT=1 ?

Did you check the 'QF seen' counters for the FC HBAs ?

$ ANAL/SYS
SDA> FC SHOW DEV FGA ! also for FGB
SDA> FC SHOW STDT

Volker.

[Peter Zeiszler]

Monitor disk - shows queue of 1.
Only getting in the 250-400 i/o operations. Steady numbers on those 2 disks as well as other disks (normal operations are running on the systems).
DSA4001: BACKUP2 394.00 309.21 197.00 394.00
DSA5005: SCRATCH1 334.00 309.84 255.00 342.00

anal/sys fc show stdt no change in the qf seen. Only number changing is the Dev I/O. I remember looking at this last week.

[Volker Halle]

Peter,

so where's the bottleneck then ? Read or Write ?

Try BACKUP source_dsk:file NLA0:dummy.sav/SAVE

and see how fast you can read from the source disk.

Volker.

[Volker Halle]

Peter,

a simple example:

BACKUP a file from a 2-member shadowed local SCSI disk on a rx2600: 800+800 = 1600 IO/sec.

BACKUP of the same file from the same source shadowset disk to another local single-member SCSI disk shadowset: about 200-300 IO/sec.

Where do you think the bottleneck is ?

Volker.

[Volker Halle]

Peter,

and the throughput was:

Backup of 1 GB file disk-to-disk: 6.6 MB/sec

SYSGEN CREATE file of same size on dest disk(high-water-marking on): 8.6 MB/sec

BACKUP file to a saveset: 3.7 MB/sec
(with SET RMS/EXT=65535/BUFF=127/BLOCK=32)

Volker.

[Peter Zeiszler]

Volker - great suggestion. When dumping to NL: I am getting only 300ish I/O out of the disk.

Tried mounting a device privately to the system (so its not going across to other systems in the cluster). I did the backup to NULL and it peaked at 450 to start but then settled around 250 i/o. This was using normal account - not the one tweaked with lower DIOLM.

[Peter Zeiszler]

FYI -
Backup2 is a 2 member shadow set. Disks exist on different XP frames.
Scratch1 is a single member shadow set.

NULL test was on the single member shadow set. Tried 2 different systems.
Disks were originally created with nohighwater.

[Volker Halle]

Peter,

so the bottleneck in your case seems to be in the Read path - unusual !

Volker.

[Peter Zeiszler]

quick test on local scsi to null - 2400 i/o sec.
Setting up for scsi to scsi.

[Peter Zeiszler]

Local scsi back to itself - 400 i/o sec.
disk is nohigh, ods-5 (just initialized it for test).

[Peter Zeiszler]

I did similiar test on itanium (running 8.3 with EMC San) depending on the /io_load value in backup command (had to play with this) I get 2k-3k i/o. This system has 4 paths to each disk - one active, others in standby.

EMC san is also hooked to our alpha - almost same patch level as system connected to xp1024. However this one has the "VMS732_FIBRE_SCSI V16.0" patch. We had to install that patch to get EMC working properly. Otherwise it showed tons of trespass errors on the EMC frames.

The test alpha to emc frame peaked near 3k i/o going to null. disk to disk average 860 read & 860 write.

[EdgarZamora_1]

> Volker - great suggestion. When dumping to NL:
>I am getting only 300ish I/O out of the disk.

I would say that is WAY TOO SLOW. A few years ago I was doing some performance testing to compare XP1024 with EVA5000 (using HBVS shadowsets on Alpha OVMS 7.3-2) and IIRC I was getting at least 1500-2500+ IO/sec. using VMS backup.

I would investigate the backend here.

[Peter Zeiszler]

5 years ago we migrated from HSJ on different servers to these. Huge boost from that.
However our database keeps growing - even with purges, cleanup, etc. 50% in one year and compressions now have to be done every 3 months. So I start looking into where to speed things up and noticed disk to disk times. Been looking at this awhile now. Even have had HP involved - guess I have to try to get this escalated. (of course I always get the feedback of upgrade to latest OS & patches).

[Colin Butcher]

Why not cluster in an IA64 box (or blade) with zero votes that has 2x 4Gbps (or faster) HBAs and use that to do the backups? Or, why not use array based snapshots /snapclones / mirrorclones if the XP array will do that for you?

It'll beat any Alpha equipped with 2Gbps FC HBAs provided that you can drive the HBAs at their full speed, one reading, the other writing.

It'll also offload all that workload from the Alpha too, freeing it up for real work.

Cheers, Colin (http://www.xdelta.co.uk).