Disk backup between boxes

[Betty Hardin]

Two HPUX boxes; L3000 & D390. We are currently capturing database backups (progress database) several times per day and writing them to tape.

We want to eliminate the tape - and I have been testing an automated ftp script to do the job. It works - but causes a load on the system when running (there are 7 DB's here ranging in size from 4GB - 15GB). All of the scripts kickoff at the same time - I could reduce the load by making them sequential but that would increase the time for transfer (currently, it is taking about 15 minutes to transfer all 7 db's).

Someone recommended rsync - and I am looking into that. Wondering if anyone has experience with rsync and how it compares to ftp performance-wise.

Perhaps you use something different that would be more efficient?

TIA -
Betty

[Tom Pojatina]

Hi Betty,

Have you looked into virtual tape? It will allow you to start writing to disk and still keep track of the database backups. After these are written to virtual tape then you are able to back them to physical tape and put them on a regular tape rotation for D/R and regulatory compliance.

Tom

[Betty Hardin]

I have briefly looked at them ... but those I've looked at start at 'just' $35,000. Not something that the employer will part with.

[Bill Hassell]

ftp is the fastest method for large data transfers. All other methods (rsync, rdist, rcp, scp, etc) use small packet sizes and simple handshakes. You might try multiple LAN cards dedicated between the L and D machines. Start with 2 ports and distribute each transfer among the two ports. That should drop the overhead and improve the transfer rate. Another possibility is to look at faster LAN cards but the D390 is just too slow to take advantage of a GB LAN card.

[Alzhy]

How large are your Progress databases and what files do you actually back up? Does Progress also have that hotbackup feature where you can actually copy/backup the database files?

If your files are small.. then rsynch may work for you so you wont have to deal with findng out what files are static...

Use the following example syntax:

rsync -avz --delete --exclude="lost+found" prdsrv1:/u/dbfiles /prdsrv1

Assuming on prdsrv1 your progress db files are in /u/dbfiles and on your other server, you want your backup reside on a filesystem/dir named /prdsrv1 - underneath it your dbfiles tree will be backed up.

The syntax states to bring in only changed files from prdsrv1 and to skip lost+found. It also directs to remove files on your destination that are not on the source.

Hope this helps..

[Betty Hardin]

I've thought about putting in a GB card in the D390 (the L3K has one) but have concerns about the ability for the system to use it. We are getting quotes on replacing the card in that server.

Thanks for the note are rsync. I've never used it but it sounds like it would be slower than the ftp.

There are 7 databases involved here. Ranging in size from 4GB - 15GB each. These are multi-volume databases - meaning that there are several files that make up each one. The db size is preallocated with each extent being 500mb.

I am moving - in total - about 55GB of data every three hours. Because of the structure of the db, one extent without the others has no value. So I really can't go with only those that have been 'modified'. When the db is started up, all of the extents are touched.

The other thought I had -

I am moving these concurrently - and I thought it might lighten the load to space out the start of the ftp process.

I also thought maybe it would be faster going from the L3K (1GB nic) to the D390 (100mb nic). Currently, the scripts are running on the d390 and pulling from the L3k.

[Bill Hassell]

55Gb every 3 hours is a *lot* of data, especially for the D390. According to the HP parts index, there isn't a GB LAN card for the D-series. There probably won't be a meaningful difference in using the GB LAN card on the L-class. The only choice is to put additional LAN cards into the D390. Or consider retiring the D390 and replacing it with a much faster rp 2400 series box which can use GB LAN cards and takes up a lot less space. The rp will also keep ultra-high speed tape drvies running at full speed.

[Alzhy]

"(there are 7 DB's here ranging in size from 4GB - 15GB). All of the scripts kickoff at the same time - I could reduce the load by making them sequential but that would increase the time for transfer (currently, it is taking about 15 minutes to transfer all 7 db's)."

Betty, please clarify what data are actually backed up "several times per day". Are you actually backing up the database files or transaction logs? Most DB's need to be brought down (or be placed in hotbackup mode) to be usably backed up.

At 100Mbps w/c the maximum throughput the D390 is capable - you will only be able to transfer at best 9GB in 15 minutes! So I doubt that you really are transferring all 55GB every 15 minutes - unles I am reading and understanding your post wrong?

Based on years of using Rsync - I find no difference in speed w/ FTP -- in fact on trasferring text/ascii files, it is even significantly faster as it also compresses data.

HTH.

[Betty Hardin]

Re-reading my post - the 15 minutes is wrong - it's actually about 30.

I am actually backing up the db files. Progress 'freezes' the db for a couple of seconds, takes a snapshot of the db, and then does the online backup using that snapshot. The online backup takes about 10 minutes per database (yes - 10 minutes).

The script files then pickup those files and move them to the d390. The files look like this:

.vol1
.vol2
.vol3
etc

Each of these is 500mb.

I really am transferring 55gb.

When you say that it is significantly faster with text/ascii files, it sounds like rsync is the tool that I want to use. But when you say that it compresses the data, it makes me wonder about the integrity on the other side. Do the files have to be decompressed to be usable?

Think I will play with rsync and see what I get.

[Alzhy]

Betty.. even at 30 minutes moving 55GB from that D390 is still impossible with a 100Mbps network. A 100 Mbps network will only yield you at best - 36GB in 1 hour. Unless of course you have an FTP infrastructure (software and LAN compresors) that does compression on the fly?