Backup / Restores slower using FC connected LT04 verses SCSI connected LT04

[Maddog1]

We have a RX2660 server running Openvms8.3-1h1 update patch level V10 which has the disks connected via fibre using dual port HBA

As this will be the disaster recovery server, the LT04 drive we want to use for restores in a Dr situation is also connected via the same fibre

The Live server directories are backed up to a scsi connected LT04 cartridge and take about 45 minutes.

The restore from this backup via the FC connected LT04 drive to the Dr server took about 6 hours!

I presume the problem is the tape drive and disk drives are using the same fabric to connect.

Does anyone know how I can monitor / Show this?

Is there anything I can do to improve the restore time?

[abrsvc]

The most common cause of this is when the disk is already mounted (no performing an image restore) and the disk has high water marking turned on.  This forces the clearing of all data which can take some time with large files and high numbers of files.  Try turning highwater marking off and report the differences.

Dan

[Maddog1]

Thanks for the response.

The disks are mounted with high water marking on, but the backup time from disk to the FC LT04 drive is also incredibly slow?

Changing the volumes to nohighwater wouldn't effect this, or would it?

[abrsvc]

Your initial problem description indicated that the restore was 8 times slower than the backup.  Highwater marking can be a cause of this.  The backup time to the tape was reported as 45 minutes.  Without knowing the amount of data and the number of files, I can not make any particular comments on that.  Changing the parameters of the backup operation can significantly affect hte backup times.  Do you have record turned on for example?   Post the backup command here and we can comment on what can be done to change the time.  Also, what does the disk look like?  Many small files or a few large ones?  There are many factors here.  More info about the environment is required to provide reasonable guidance.

Dan

[Maddog1]

The backup command is as follows and is done on exactly the same hardware / software except the LT04 drive is connected directly via Scsi. The backup of the first drive is shown

$ INIT/MEDIA_TYPE=COMPACTION 'tapedrive' SECUR

$ MOUNT/MEDIA_TYPE=COMPACTION/FORE 'tapedrive' SECUR

$back/ign=(label,int)/noass DGA11:[dir]*.*;* 'tapedrive'dir.bck

The directory contains 4783 files, 21751276/22661952 blocks

This takes about 5 minutes on the live system

Same backup commands on DR server to LT04 drive connected by fibre.

The directory contained 4893 files,  11714295/12963398 blocks

This took 47 minutes to complete.

[Bob Blunt · Accepted solution]

Well, you might as well face it...and brace for the commentary.  If you're using /ignore=(label,interlock) then (IMHO) you're wasting your time.  I understand that it takes a while to save but you're not getting a good copy using /ignore=interlock and you won't really know it until you restore and it'll be doubtful you can easily recover from that condition.

Having said that, it is out of the way.  Consider when you're using the same fabric that you're, in some cases, doubling the I/O to the giblets on the SAN.  Now most proponents of SAN will tell you that it probably won't saturate a well-designed SAN but it still does increase the load.  You haven't described what type controller you've got for the disks or any of the intermediate hardware between the host and the tape.  All that adds up.  Unless there is some commanding reason that you absolutely must have the tape on the SAN I'd (first choice) put the tape on the SCSI and leave it there or (second choice) use a separate fabric for the tape and disks.  I know all the selling points about shared tapes but if you have a speed problem getting the data onto mylar you need to solve that.

There are some nit-picky things I'd recommend on your initialize and mount commands.  First, I'd find out what density the LTO4 uses (it may be /DENSITY=LTO4, but I haven't one to use to check).  Make triple sure that you specify all commands relating to the initiaize, mounting AND BACKUP to the tape using /DENSITY, /MEDIA_FORMAT and, on the $MOUNT I'd add /CACHE=TAPE_DATA (not available for init or backup).  Then on the $BACKUP command I'd add, at least, /BLOCK=61440 (or some other multiple of 4096 larger than the default 8192).  The other crucial factor that should be remembered is that with some devices and conditions you have to be careful to not overwhelm the I/O subsystem.  You can, in some cases, far exceed the ability of the SAN devices if you use account quota-ish settings that are too aggressive.  BACKUP was modified to make this less likely (around V8.2, I think).

And, unfortunately, all you can do is test, test and retest until you find a COMBINATION that works best for your environment.  Personally, I'd still put the tape on the SCSI even if it does make it more inconvenient.

bob

[Hoff]

In no particular order...

Process quotas can differ among systems.   Here's a list of the HP recommendations.

FC SANs aren't the same, even on the same fabric.  Different-speed HBAs, for instance.  Different SAN controllers.  Details of the hardware here, please?  Is it end-to-end the same configuration that you're testing?

SANs are good at sharing. Is there contention from other SAN users and other SAN client hosts that are sharing access into the same controllers?  These other hosts might be completely-other operating systems, and using completely-other disks, but still bogging down the controllers or the controller HBAs.  (Even with the same fabric, contention can arise when the I/O paths differ.)

Differences with highwater marking can be involved, as has been mentioned.

Monitor the I/O queue lengths to see where the bottleneck lurks.

And monitor the configuration in general; find the bottlenecks.

What happens with a SCSI-connected LTO-class device, directly connected to the Integrity host via UltraSCSI?  That'll eliminate the SAN and the HBAs and contention as a potential trigger.

Having the same FC SAN fabric used for disaster tolerance?  That implies there might be FC bridging somewhere here, and that bridging can imply narrowed data links exist somewhere in this connection.  Having the same fabric can also imply host-based volume shadowing, and that can also effect the performance of the I/O configuration, and particularly when (as is the case here) writes to the (shadowed?) disks are involved.  And there can be different-speed or flaky FC SAN switches, too.  (All of which gets back to knowing the configuration.)  

And as stated up-thread, the only reason to use /IGNORE=INTERLOCK is to ensure the data inconsistencies in the archives are not reliably reported; those interlocks that are being ignored are there for a reason, after all; to ensure data consistency.

[Maddog1]

I noticed the device description of the LT04 scsi attached tape drive has the following

default buffer size is 8192 

The device description of the LT04 FC attached tape drive has the following

default buffer size 512

Could this be a factor and if so, how would you amend it to match?

[GuentherF]

47 minutes for 11714295 blocks that calculates to 2 MB/sec. NO WAY!

The 5 minutes for the amount of data on the LIVE system seems about right.

My first check would be for the LTO. Can you attach the FC connected drive to a direct SCSI bus and run a test like "BACKUP SYS$SYSDEVICE:[*...]   lto:..." and get the timing?

Would be good to know how the LTO for the DR system is connected. Not how you THINK it is connected but how it really is. Verify it. Tell us all the details.

/Guenther

[John Gillings]

Maddog,

   I'm wondering why you DR plan involves tape at all? Why not avoid this question altogether? just replicate the disks on both sides? If they're not connected by fibre, just transport disks instead of tapes. Your recovery plan becomes faster, simpler and quite possibly cheaper.