rsync seems very slow between local volume groups

[Timothy P. Jackson]

Hello,

I am working on a DR script on a system that is in production 24x7.

system: rx3600
OS: HP-UX 11i version 3

I have two logical volumes . I need to sync the data on lvol1 to lvol2. My first thought was to use rsync, but now it seems so slow I don't know if it is the best option.

I am syncing about 45 Gigs of data from /dir1 to /dir2 and the sync is taking about 40 minutes with out coping any data. Because I the company has been on holiday for the past week none of the data on the production side /dir1, there have been no updates.

The command I am using is....

rsync -avcu --delete --stat /dir1/ /dir2/

How can I speed this process up or should I be looking for another solution?

Any help would be greatly appreciated!

Thanks,
Tim

[Mel Burslan]

rsync is not a speed daemon by any means regardless you are doing it locally or to a remote system. Especially when you are talking about updating a 40+GB storage left out of sync for a long period of time.

But I have to ask one question:did this same setup work faster in the past ? If it did, what happened since then ? If this is a new setup and you are not happy with the speed, the only thing you can do is a) if the disk are on the SAN, use a SAN frame based data replication utility of your choice (business copy for HP XP, or Business Continuity Volumes aka BCVs for EMC are some examples); b) if the disks are local, you can use UX-Mirror to create a mirror copy of the disk drive but I am not sure how this can be sued as a DR solution.

[Alzhy]

If your Filesystems contain gazillion files and directories or are uber active -- RSYNC is not the correct prescription.

You will be better off using vxdump/vxrestore (if your filesystems are vxfs) or fbackup/frestore.

You may also want to look at using LVM Snapshots and pvmove.

[Timothy P. Jackson]

Hi Mel,

Both logical volumes are local and mirrored, so let me explain my logic which could have flaws in it.

Production is running 24x7 so my best bet to get a clean copy of the data is to...

1) pause the database engine
2) rsync the data to another file system
3) enable the database engine
4) transfer the synced data off site

Because it is taking so long to sync the database I cannot pause the database engine for that amount of time. So I modified my logic to the following.

1) pause the database engine
2) break the mirror on lvol1
3) enable the database engine
4) rsync lvol1 to lvol2
5) merge the mirror on lvol1
6) transfer the synced data off site.

I would like to keep the mirrors broke as short of time as possible.

[Alzhy]

SOunds like your DBAs need to learn how to use RMAN...

Then yo can ship out those RMAN files offsite

[Timothy P. Jackson]

I have never used vxdump or vxrestore before. Is it faster than fbackup/frecover? does it use a lot of resourses?

[Mel Burslan · Accepted solution]

In which case, suggestion of vxdump/vxrestore made by Alzhy, may be a better approach for you speed-wise. rsync is a file-by-file operation whereas vxdump/vxrestore do their magic by copying volume data regardless of the number of files or their contents.

Years ago, during a SAN migration exercise, we have gained abiout 2.5 times speed advantage using vxdump/vxrestore over doing a plain cp (which is what rsync basically is) but this was a special case to our setup, using an EMC source array and XP1024 as a target. So, your mileage may vary.

[Timothy P. Jackson]

Oh Yeah...

The file systems are vfxs and are both local.

[Mel Burslan]

I second the RMAN option. Any modern backup solution have a rman module and back those puppies up to a tape or a remote disk. Takes the load off your shoulders as well. I like the idea :)

[Timothy P. Jackson]

unfortunately this is not an Oracle database it is a IBM/U2 unidata database. I am going to try the vxdump/vxrestore and see if it helps.

[Alzhy]

vxdump/vxrestore should BE faster as it is multi-threaded.

Does your DB offer any hotbackup facility where its transactions are copy on write/read only so you can vxdump your actual filesystem?

If so, then you should do that if it is feasible.

[Timothy P. Jackson]

It does but it is extremely hard to setup and maintain. It also has a very high overhead and doesn't work 100% of the time. So basically it is worthless. It was basically developed to use on just a hand full of files not thousands.

Thanks for all your help!!

[Bill Hassell]

> unidata ...

That's a semi-database engine that use tens of thousands of small files, so rsync (like tar, cpio, etc) will be strangled by all the directory and file ope/close activities.

Try a dd copy between two lvols. Make sure you use the raw volumes and a reasonable block size:

dd if=/dev/vgunidata1/rlvol1 of=/dev/vgunidatadr1/rlvol1 bs=2048k

Ideally, this must be done with both lvols unmounted so that changes in related files will not corrupt the database. Once the initial dd copy is done, try rsync again and you should find the update speed much more reasonable. I have used rsync for 10TB filsystems and millions of files -- the first pass took several days. From then on, the time dropped to 5-10 minutes to update everything for a day's worth of changes. When the two volumes wwere completely synced, a final rsync pass took less than 3 minutes.

[Timothy P. Jackson]

Hi Bill,

This is my issue....

I have already done the initial rsync and I expected it to take a long time. But, the rsync I do after the initial sync are taking about 40 minutes. I was hoping for 5 min or less but it that is not happening.

[Bill Hassell]

rsync's completion time is completely dependent on the amount of changed data, which usually corresponds to the amount of active database time since the last rsync run. If you can run rsync twice with the database stopped and the lvol unmounted, that should give the minimum run time. For my 10TB system, files varied from a few hundred KB to 100's of MB. Unidata may create much smaller files, thus creating the bottleneck.

[Don Mallory]

What about using LVM snapshotting?

You could pause the database, create a snapshot using lvcreate -s (on 11.11) or vxsnap (on 11.23 or later - requires using VxVM volumes) of the logical volumes that you have concerns about, unpause the database, then rsync them to your heart's content.

The difficulty will be the change rate of the data, but if your intent is only to rsync it off, then release the snapshot, you wouldn't need more than 5-10% more disk space available in the source volume group.

Of course, you could just make that the backup you are working from. LV snap the data over to the other LV/mount you are using then use whatever you like to get it off the host.

Just keep in mind that the data change rate will determine how much disk space you need to have reserved. 20% is usually a good starting point, but if your intent is to snapshot, then back it up to something else, it's far easier.

Best regards,
Don