Slow Database copy

[Joseph Morrison_2]

Hi

A customer needs to copy their 1TB Oracle database once a month.The 2 copies are defined as vg05 & vg06 which are identical (+- 10 lvols).Both VGs are located on the same Storageworks MA8000, 3 LUNs each,400GB per LUN,RAID 5,256MB cache per controller.

The server(L1000) has 2 FC adapters installed(A5158A) which connect to the S/Works through 2 separate FC hubs.Their are no other hosts on these hubs.VG05 accesses the disks via
FCA1, VG06 via FCA2.
For optimum utilization of bandwith, 3 or 4 filesystems are copied simultaneously.

The problem is that the copy process takes +- 33 hours to complete!

Any suggestions on what can be done to improve copy performance?

Thanx

[Richard Hood]

Joesph,

The 33 hours is pretty good for a 1TB copy, taking into the account of the backplane and FC speed.

If you are concerned that primary system is offline while you are doing this copy, then here is a work around.

You could make the secondary drives (vg06) the mirror of primary (VG05). Let the drives sync and when it is complete, break the mirror - reassign the drives to the DR server.

This way the system will not have to be down during the backup, plus the data will be identical until you break the mirrors.

You will still have an minor outage - for you will need to take down the database - prior to breaking the mirror - or you could possibly make the database inconsistant with incomplete transactions in the arch/redo logs.

just my $.02

[Jose Mosquera]

Hi,

Are you using fbackup command?

In the past I have a similar high rates and I solved this applying lastest fbackup/frecover cummulative and SCI/stape patches, pls check your current fbackup/frecover patch number by:
#swlist -l product|grep fbackup

This show you a patch number as "PHCO_nnnnn"

Then go to: http://www4.itrc.hp.com/service/patch/mainPage.do and search your patch number. check if this patch have been superceeded by a newer, use the newer with all dependencies (stape will be include as dependencie) and download them like a gziped tar boundle file. FTP it in binary mode to your box into a temporaly and safe place, unzip and untar it. Execute the file named ./create_etc_etc. When the depot file generation have done, execute:
#swinstall -s //depot
Ensure that in Options menu the items to reinstall filesets are no selected, always avoid reinstall installed patches filesets. and finally install the bundle!

Rgds.

[joseph morrison_1]

Thanks for your replies Guys.

Is 30GB/hour really OK?
What is the "real" throughput of fibre-channel?
Initially the server had 1 FCA accessing all disks.The copy ran for 27 hrs.2 FCA =slower??

The DB refresh process happens as follows:
vg05 (standby DB)-online copy from prod DB over
network =34 hours
vg06 (QA DB)-copy from standby DB over FC=33 hrs

The second copy has to happen faster as the entire process must fit into a weekend.Basically the copy window has to be reduced to +-10 hours.

BTW, "cp" is used, not FBackup.

Any ideas?

Thanx

[Vincent Fleming]

There are several factors that are probably affecting your copy speed...

First off, cp is not a speedy copy mechanism. It tends to use smaller blocksizes. If you can use dd, you can increase the blocksize to something more optimal, such as 64k. (use the bs=64k argument)

You can also run several dd's at the same time to get concurrency.

Secondly, the number of drives you're writing to is a significant factor. 2 drives are faster than one. 10 are faster than 5... 20 are faster than 10. If there's any way you can increase the number of drives, it would be helpful.

The third thing I can think of is to use some sort of snapshot technology; some of the newer arrays (notably the EVA) contain snapshot and snapclone technology. Snapclone sounds like it would fit your needs well - it will take a point-in-time space-efficient copy of the data, then (in the background) copy the data to the backup drives. Your problems with your copy window completely go away! It would be "done" in seconds, as far as your server is concerned.

Good luck, I hope this helps.

Vince

[Steven E. Protter]

If your database were on our SAN, yes, I'd say the copy is going slow, because I know the specs on ours and it should run a lot faster.

But there are a lot of variable factors. If the database is sitting on a true RAID 10 partition the copy is going to go faster than if its sitting on a RAID 5 partition.

You need to first see you SAN administrator and see what speed your setup is rated to work at based on the type of storage you have been allocated.

Now, as to making the copy faster.

Method 1

You could use Mirror/UX to maintain the logical volume. You can break the mirror in a few minutes. This should be done while the database is down. Then you'll have a new logical volume which you can mount and copy your big database off to without problems.

Method 2

You can use features of your SAN to break mirror copies and get a copy for instant use in a few seconds. But you'll need to talk to that SAN administrator.

SEP

[Joseph Morrison_2]

Hi again

Thanks for all the suggestions.

I now need to explain to the customer why the DB copied faster when it was connected to a K460 (3 processors, 3 Gb memory).At the time the 800Gb database copied in +- 8 hours.The 1 Tb DB connected to an L1000 (2x440 procs,3 Gb memory) now takes 33 hrs to complete.The S/Works config was not changed.


Any ideas?

[Chris Vail]

You mention that you use cp to copy the files. This is NOT extraordinarly fast. However, tar IS fast. You may need to get gnu tar from the great people at www.gnu.org. Their version of tar will handle files larger than 2 GB. You may also need to install their C compiler as well, but its worth it. Last Saturday night, I copied +700GB from disk to disk in about 3 hours using gnu tar. I used the syntax "tar cvf - .|(cd \$DESTDIR;tar xvfp -). I started 4 of these simoultaneously on an L class with (4) 440Mhz processors. The processors, of course, maxed out during this copy process, but thats what I wanted them to do. Once the first process finished, I started another (I was working from home).
This evening, I have a similar issue: I need to copy 688GB from one FC/10 array to another. I'll run timex on in and post back how much real time it took.

Good luck
Chris

[Vincent Fleming]

Did you change OS version along with the machine? (I'm assuming you did)...

You say the StorageWorks config did not change - does this mean that you simply imported the volume groups when you plugged in the new machine? Or did you backup the K and restore onto the L?

This can make a difference - particularly if you changed filesystem type and/or settings.

For example, if you changed to JFS or online JFS from HFS (or even changed versions of JFS), things can change. If you're using JFS, be sure to turn off logging on the volume(s) you are writing to. This will increase your write performance.

Tell us more details about your upgrade; you answer might be in there.

Vince

[Byron Myers]

Joseph, a quick kernel param to check on your L1000 is your dcb_max_pct. If you have a large file system cache, you will see a HUGE performance hit when copying a large amount of data via file system - this is because every FS block copied will encounter a cache miss, so the larger the cache, the larger the miss time.

[joseph morrison_3]

VG's were exported from the K and imported on the L. K uses a3404a HBA, L uses A5158a HBAs.
Both systems run HPUX 11.00 with JFS filesystems.

I will have a look at the logging and kernel parameter.

Thanx

[Volker Borowski]

Hi,
another approach, as far as downtime is concerned !

Do it ONLINE !

1) get a good CREATE CONTROLFILE template by using
ALTER DATABASE BACKUP CONTROLFILE TO TRACE on the source DB and do in addition
ALTER SYSTEM SWITCH LOGFILE;
ARCHIVE LOGLIST;

2) set all Tablespaces in backup mode
3) copy the files
4) End backup mode for all tablespace !
5) do a
ALTER SYSTEM SWITCH LOGFILE;
ARCHIVE LOGLIST;

6) Now copy all the archivelogs from the first LOGLIST# to the last LOGLIST# to the archive-destination of the target DB (may be more of them if needed, i.e. to reach a special timestamp)
7) Adjust the CREATE CONTROLFILE as needed (RESETLOGS, SID, REUSE etc.)

8) on Target DB
STARTUP NOMOUNT
CREATE CONTROLFILE ....
RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL
...recover as needed
ALTER DATABASE OPEN RESETLOGS

There you are. I did this many times and even across the network (using tar and remsh). Should be no real problem.

Onlinecopies, for less downtime :-)
Volker

[Joseph Morrison_2]

Thanks Volker for the suggestion.

I discussed this with the customer, but they prefer doing a complete refresh of the database.

The dcb_max_pct=50, dcb_min_pct=5.Is this ok?

Joe

[Yogeeraj_1]

hi,

To add to what Mr. Volker said above, you may consider using RMAN to duplicate the database! - which is quite simple if you have use a recovery catalog.

E.g. we use the following commands:
rman rcvcat rman_pfsdup/rman_pfsdup@catalog_db target internal/oracle@target_db auxiliary internal/kernel@aux_db;

run {
Set until time "to_date('26-JAN-2003 23:59','dd-mon-yyyy hh24:mi')";
allocate auxiliary channel fs2b type disk format='/backup01/rman/full/fs1/full_%u.%p';
allocate auxiliary channel fs1b type disk format='/backup/rman/full/fs1/full_%u.%p';
set newname for datafile 1 to '/RESTORE/u03/oracle/oradata/pfs/system01.dbf';
set newname for datafile 2 to '/RESTORE/u03/oracle/oradata/pfs/tools01.dbf';
set newname for datafile 3 to '/RESTORE/u03/oracle/oradata/pfs/slx1_cmtstore_medium01.dbf';
set newname for datafile 4 to '/RESTORE/u03/oracle/oradata/pfs/slx1_pfsdata_lmt_small01.dbf';
set newname for datafile 5 to '/RESTORE/u03/oracle/oradata/pfs/slx1_tbsadmin_medium01.dbf';
set newname for datafile 7 to '/RESTORE/u03/oracle/oradata/pfs/drsys01.dbf';
set newname for datafile 8 to '/RESTORE/u03/oracle/oradata/pfs/oem_repository.dbf';
...
set newname for datafile 40 to '/RESTORE/u03/oracle/oradata/pfs/slx1_pfsindx_lmt_xlarge01_2.dbf';
set newname for datafile 41 to '/RESTORE/u02/oracle/oradata/pfs/slx1_cmtpers_lmt_large01.dbf';
set newname for datafile 42 to '/RESTORE/u02/oracle/oradata/pfs/slx1_cmtpers_lmt_medium01.dbf';
set newname for datafile 43 to '/RESTORE/u02/oracle/oradata/pfs/slx1_cmtpers_lmt_small01.dbf';

duplicate target database to pfs_dup
logfile '/RESTORE/u02/oracle/oradata/pfs/redo_01.dbf' size 25M,
'/RESTORE/u02/oracle/oradata/pfs/redo_02.dbf' size 25M,
'/RESTORE/u02/oracle/oradata/pfs/redo_03.dbf' size 25M;
}
exit;
EOF

============================================================

I would suggest that you do try this alternative.

Hope this helps!
Best Regards
Yogeeraj

[Joseph Morrison_2]

More info :

Filesystem: logsize=1024,block size=8192
Will using more,smaller LUNs increase performance?
What would be the impact of increasing the cache on the S/Works?

Thanx