Data restore

[Lionel Liu]

Hello there,

We have a 20G data disk. We use following command to backup to tape, it took 70 minutes

Backup/image dkd105: mke500:dkd105050305.sav/sav/block=65024

When restoring the data from tape to disk
using following command, it took more than three hours.

BACKUP/IMAGE MKE500:DKD105050305.SAV/SAV DKD105:

Is that normal?, Anyway to speed up restore?

Thanks

Lionel

[Hein van den Heuvel]

I have not actually measured this, but my gut feel says it is normal.

On the backup, the system can use a lot of caching, and does not need to care about some data (like bitmap.sys), and can issue concurrent read IOs as one read will not influence the other (other then speed)


On restore files have to be created, space allocated, one at a time. Yes the user data writes can (and are?) done in parallel, but the file and directory stuff around it goes one at a time pretty much, with lots of write IOs to bitmap and indexf.sys and so on.
I'm speculating a little here. Maybe Guenther can correct me. Still, it feels right for it to be slow.
Also, when a slow a little the tape may stop streaming, in which case you slow a lot more.

Finally... are you sure that was the right restore command? When I just verified that (on a ram disk), it created a flat restore structure. Don't you need DKD105:[*...]

Cheers,
Hein.

[Uwe Zessin · Accepted solution]

Hein,
you might have forgotten the /IMAGE qualifier on the restore...

[David B Sneddon]

Lionel,

What you are seeing is fairly typical (at least it
is in our situation). What can really impact the
restore is a large number of small files. You would
want to check how many files are in the backup and
some idea of number of large files compared to number
of small files.
If you have lots of small files you will not likely
be able to improve the restore time.

Regards
Dave

[Hein van den Heuvel]

Uwe> Hein,
you might have forgotten the /IMAGE qualifier on the restore...

Right right. I spaced.
I tried that first, but it did not do anything. Looking back that was because I had the output mounted normally.
I got the files back dropping the /ima, but should have remounted the output /foreign. With that in place the XQP is no longer involved and Backup can use its own Files-11 structure create commands.

Still the main problem remains. When creating a file, many data elements need to be updated: indexf, bitmap, directory and finally data. So for restore many more (write) IOs will be needed which become more signifincat as you have more and smaller files.

Hein.

[Uwe Zessin]

Right, lots of small files are really a problem. If you can dismount the source disk during the backup you can try a BACKUP/PHYSICAL.

This will save ALL disk blocks, which might not make a big difference if the disk is quite full.

The downside is that you cannot restore a single file from the physical backup. At least in older versions of OpenVMS you had to restore to a disk with the same size (and in the early days even with the same geometry).

[Ian Miller.]

Restores always take longer - the more files then the longer it takes.

As BACKUP creates each file individually using XQP QIOs (I think) then changing the RMS parameters won't help either.

[Jan van den Ende]

Then again,

how often do you have to do a BACKUP save?
Quite often, I hope!

How often do you do a restore?
Very rarely, I hope!!

So, improving SAVE speed has a much bigger impact than improving RESTORE speed.

... but if you need that restore, it may be a nuisance.

Then again, IF you need that restore, my guess is that the (guaranteed!!) quality & integrity of that restore is MUCH more important than the speed.


just my EUR 0.02

Proost.

Have one on me.

Jan

[Uwe Zessin]

A backup design usually includes the RTO ;-)

[Antoniov.]

Hi,
AFAIK backup/image without /noini qualifier always initialize output device so it can't run any concurrent process writing on disk. It's true restore command has to write to index, bitmap and directory, but comparing backup time (70 min.) with restore time (210 min.) the rate is 3:1. I'm not sure that's the mere reasion.
I believe the tuning is not at the best.

Antonio Vigliotti

[Uwe Zessin]

/INITIALIZE allows BACKUP to change some volume characteristics. The rest of the restore process is the same.

The problem is that a lot of meta data needs to be written on a restore and that is a synchronous operation - I doubt that BACKUP does any optimizations. Also, you cannot optimize any head movements.

Show us how to tune a restore!!

[Wim Van den Wyngaert]

Initialize your tape with the highest density possible for your drive.
Use /group=0 to reduce the overhead by 10%.

Wim

[Antoniov.]

Hi Lionel,
assuming you make restore with default /INIT option; because of this you have to work in single user mode.
Assuming no other users/processes are running while you make the restore.
Assuming in your saveset there are many sequential files.

You can temporary trick your system to avoid multiple i/o on disk.
You can turn on deferred write with
$ SET VOL DKD105: /NOWRITE
Also you can increase multiblock count (it works fine only on sequential files)
$ SHOW RMS !--read current value
$ SET RMS /BUFF=128 !-- set new value
Also you can increase two system parameters ACP_DINDXCACHE and ACP_DIRCACHE. They are both dynamic.
It is very importart you restore all modified parameters after you terminate saveset restore.

Antonio Vigliotti

[Ian Miller.]

as BACKUP creates files using XQP calls rather than RMS I don't think changing RMS parameters will help.

[Hein van den Heuvel]

Ian> as BACKUP creates files using XQP calls rather than RMS I don't think changing RMS parameters will help.

And that would be for non-image restores.

For /IMAGE restores, as discussed here, the target device is mounted /FOREIGN and it is all Backup's doing. No RMS, no XQP. Backup knows and executed the ODS layout with logical IOs, not virtual IOs.

I discussed this some with Mark H. We believe image restore does do the bitmap in a single IO, but the indexf.sys in a single block IO per file. Furthermore it is all synchroneous, in line = slow. This in contrast to the backup phase which will do concurrent IO. This could obviously be improved upon, but over the past decades there has been no business justification for that.

Hein.

[Jan van den Ende]

Uwe,


A backup design usually includes the RTO ;-)


Yes, of course. And Management should have a written, advance, realistic estimate.

But, in my experience (scarse on VMS, but witnessed from the side-line on *UX and Billyware all too frequently) the DECISION to restore always took longer than the actual resore itself.

And yes, the discussion always starts with ANY time being too long. But in the balancing between restore duration, restore reliability and backup duration usually restore duration looses priority.
Much more can be gained by a real rigid (and rehearsed) procedure for the restore-decision process.

YMMV !!

Proost.

Have one on me.

Jan

[Antoniov.]

I understand RTO means Run-Time Optimization and in my mind is stupefyingly backup takes 70 min., writing to slow device (tape), while restore takes 210 min., writing to quick device (hard disk).
I know restore takes more time than backup in vms, but the rate 3:1 seems to me too high.

Antonio Vigliotti

[Uwe Zessin]

We're talking about a backup solution,
so RTO = Recovery Time Objective.
It means that the customer specifies the time in which the data must be available again and the backup solution is planned accordingly (or the RTO must be relaxed for reasons like not enough money or other constraints ;-)


Antonio,
haven't you followed recent discussions here?

*** Modern tape drives are not slow ! ***

A tape media is just moving forward (a LTO-3 at up to 80 MegaBytes/second, uncompressed). A disk drive must move its arm to the correct cylinder and then wait until the right sector passes the head.

Re-read Hein's excellent descriptions about why a restore is limited to lots of small, synchronous I/Os, which will make the restore slow.

[Antoniov.]

Uwe,
thank you for RTO meaning.
I'm with you about new quick tape devices but, in this case, tape throughtput is less than 0,285 Mbs per minute (20Gb/70min) so I believe it is a DAT tape, no some device like DLT.
I'm still convinced the rate 3:1 is unreasonable, in this case.

Antonio Vigliotti

[Uwe Zessin]

What is "0,285 Mbs" supposed to mean? As far as I can tell, a small "b" usually indicates "Bit".

I guess we need to compare out maths:
20,000,000,000 / (70*60) = 4,761,904

Thats 4.7 MegaBytes/second to me.

[Antoniov.]

Uwe,
you are right!
The throughtput is around 5Mbs.
However, 5Mbs is the rate of DAT tape.

Antonio Vigliotti

[Uwe Zessin]

And a TZ-86 (DLT) can do about 6 MegaBytes/second max. What is your point?

In the early 1997 I was working in a project where we could read between 850 KiloBytes/sec from the system disk and multiple MegaBytes/sec from the data disk (large database files) to the same destination (a DECnet node with an attached HSZ RAID subsystem).

We don't know the hardware and we don't know what the disk's file structure looks like.

[Cass Witkowski]

You don't say whether the disk is a direct connect SCSI or on an HSZ controller. If on the HSZ40 controller and it has write back cache then make sure it is turned on for this drive. This can help speed things up.

Cass