1748041 Members
4930 Online
108757 Solutions
New Discussion юеВ

Re: speed up backups

 
Peter Zeiszler
Trusted Contributor

Re: speed up backups

Local scsi back to itself - 400 i/o sec.
disk is nohigh, ods-5 (just initialized it for test).
Peter Zeiszler
Trusted Contributor

Re: speed up backups

I did similiar test on itanium (running 8.3 with EMC San) depending on the /io_load value in backup command (had to play with this) I get 2k-3k i/o. This system has 4 paths to each disk - one active, others in standby.

EMC san is also hooked to our alpha - almost same patch level as system connected to xp1024. However this one has the "VMS732_FIBRE_SCSI V16.0" patch. We had to install that patch to get EMC working properly. Otherwise it showed tons of trespass errors on the EMC frames.

The test alpha to emc frame peaked near 3k i/o going to null. disk to disk average 860 read & 860 write.
EdgarZamora_1
Respected Contributor

Re: speed up backups

> Volker - great suggestion. When dumping to NL:
>I am getting only 300ish I/O out of the disk.

I would say that is WAY TOO SLOW. A few years ago I was doing some performance testing to compare XP1024 with EVA5000 (using HBVS shadowsets on Alpha OVMS 7.3-2) and IIRC I was getting at least 1500-2500+ IO/sec. using VMS backup.

I would investigate the backend here.
Peter Zeiszler
Trusted Contributor

Re: speed up backups

5 years ago we migrated from HSJ on different servers to these. Huge boost from that.
However our database keeps growing - even with purges, cleanup, etc. 50% in one year and compressions now have to be done every 3 months. So I start looking into where to speed things up and noticed disk to disk times. Been looking at this awhile now. Even have had HP involved - guess I have to try to get this escalated. (of course I always get the feedback of upgrade to latest OS & patches).
Colin Butcher
Esteemed Contributor

Re: speed up backups

Why not cluster in an IA64 box (or blade) with zero votes that has 2x 4Gbps (or faster) HBAs and use that to do the backups? Or, why not use array based snapshots /snapclones / mirrorclones if the XP array will do that for you?

It'll beat any Alpha equipped with 2Gbps FC HBAs provided that you can drive the HBAs at their full speed, one reading, the other writing.

It'll also offload all that workload from the Alpha too, freeing it up for real work.

Cheers, Colin (http://www.xdelta.co.uk).
Entia non sunt multiplicanda praeter necessitatem (Occam's razor).
Steve Reece_3
Trusted Contributor

Re: speed up backups

What else is going on with what you're reading at the time that you're reading it and what kinds of software and caching are in use?
I'd expect the read path to be the bottleneck in this. VMS will get a return status for the writes having completed as soon as the write hits the disk controller. The controller can then write the data to physical disk as and when it wants to or feels like it. The read, on the other hand, has to go through to the disks unless the data have already been cached.
How fragmented are the input files that you're backing up? If they're continually growing then are the disks full of hundreds of small fragments?
Has anyone looked at locking on the files being backed up?
Has anyone looked at the processor modes during the backup?
Is FastPath enabled? If not, is the primary CPU becoming a bottleneck for operations on the node doing the backup?
Rob Leadbeater
Honored Contributor

Re: speed up backups

Hi Peter,

I've not seen any info on the actual hardware being used on the Alpha side of things, or the SAN switch infrastructure.

Are you still using 1Gb/s HBAs ?

Are there lots of interconnected SAN switches ?

Cheers,

Rob
Peter Zeiszler
Trusted Contributor

Re: speed up backups

Alpha Servers are GS1280 - primary systems are 6cpu with 18gig ram, other 2 systems in cluster are secondary partions of the GS1280 with 2cpu and 6gig. 2x HBA 2gig each with each HBA going to a switch/fabric. These connect to 2 Xp 1024. Most data disks are at the local site with the destination disk being a shadowed disk with one disk at each site.

Disks are fastpath setup. Primary function during backups is other processes updating other applications and reporting. Disks are locked by the application and the application kicks off a backup (file by file). Files are RMS and are pre-allocated space. Checked fragmentation and there are some files that have 50+ extents with extent sizes of 65535.

My Test however is a single file with 2 extents with a size of 31million blocks (approx 15 gig). I also created a new file of 5gig on other disks with sysgen just to try the read on different formatted files.

Late yesterday I made all systems have same path (use fabric A or B) for the backup2 disk which has a disk at each site via shadow (I will have to setup a script so that they get set at boot instead of auto choosing). Read I/O rate fluctuated from 2k I/O to 400 I/O on the shadowed disk. I grabbed a disk local to each site and tested a backup to null on each disk multiple times. On the disks that were local to the site I was getting 2k i/o and on the remote site disk 300-400 i/o. Didn't matter which site.

So it looks like something slowed down between sites. I now have network involved looking at lines between sites. We also contacted our carrier to see if they have something.

Went back into older historical performance data too - found the drop from approximately 1200 disk i/o to averaging 800 disk i/o from a day to day for the main backup job. Captured that data and a few days around that date and sent that to networks.
Volker Halle
Honored Contributor

Re: speed up backups

Peter,

you may want to use the HP Tool T4 to measure and document performance data. It provides very detailled performance information on fibre channel disks as well.

Volker.
Peter Zeiszler
Trusted Contributor

Re: speed up backups

Have T4 running. Also have CA Performance tool (the old DEC polycenter stuff)