Selective restore "several" directories tape to disk...

[Art Wiens]

Is it valid to say (in DCL):

$ backup $2$mga1:valid_ssetname./save -
/select=([DIR1...]*.*;*, -
[DIR2...]*.*;*, -
[DIR3...]*.*;*, -
[DIR3...]*.*;*, -
etc. etc. 40 odd directories listed
[lastDIRn...]*.*;*) -
$40$DKA400:[*...]*.*;* /by_owner=orig

I sorted the list of directory names before I created the command so they are in alphabetical order. There are other directories interspersed that I want to skip.

The process restored the first two DIR's (taking quite a bit of time between the two) and then went pretty well solid compute, doing minimal DIO's, writing nothing to disk.

Is this too much for BACKUP to handle? I don't have the disk space to restore the whole saveset.

It's an SDLT/MDR on fiber attached to an Alpha 800 VMS v7.2-2 .

Cheers,
Art

[Allan Bowman · Accepted solution]

Art,

I have had similar problems in the past with extracting multiple directories. When it does work, it seems to take forever. I usually resorted to multiple restores with just one directory specified in each. If you run the command interactively (or monitor the output directory if in batch), you can kill the restore as soon as the desired directory has completed and start the next one. This method is a bit painful, but sometimes faster than the multiple dirs.

Allan in Atlanta

[Art Wiens]

I tried it again, skipping the first two it had already done, and it "reacted" the same way. I looked at the source disk for what was in the intervening directory before the next desired one, there's about 80,000 small files a few levels down!

I think it's actually working (100% cpu!), just taking a pile of time to get through those.

I'll try it again this weekend when I can have the tape drive a little longer.

Other than that, I'm not looking forward to doing this manually ~45 times :-(

Cheers,
Art

[Hein van den Heuvel]

>> don't have the disk space to restore the whole saveset.

An other approach, which goes entirely against my 'Performance Consulting' nature would be to have a silly batch job sit in a loop deleting undesirable files.
It seems awfully wasteful, but it is just the systems that is busy doing and ondoing file restores which has to be better than you selecttively restoring a tape 45 times.

How much physical memory do you have? I don't suppose the directories to be restored are less than say 1/4 of the memory? Maybe you can do a RAM disk to restore into. Now as soon as you see a new directory appear with a 10-second DCL loop decide to either delete the prior one directly, or copy it to the target and then delete it. If that is feasible at all, then pick a seperate (spawned/batch) task to do the copy such that the deletes's can keep up.

Hein.

[Volker Halle]

Art,


I think it's actually working (100% cpu!), just taking a pile of time to get through those.


BACKUP should do some DIRIOs to the tape from time to time to read the saveset blocks, otherwise it would be caught in a loop. If you run it interactively, CTRL-T should tell, otherwise look at the $2$MGA1: device operation count.

Volker.

[Hein van den Heuvel]

Even if it still did _some_ IO, it sure sounds like an 'n-square' problem lurking inside.

I have send of a mail to an (ex)Backup developer soliciting his comments or a further referal

Control-T may indeed shed some light.
Is the memory space growing while burning CPU?
All usertime?

fwiw,
Hein.

[Art Wiens]

Thanks guys. Just talked to Ops and they said I can have the drive until 11 tonight so we'll see if it can get the job done in 14 hours!

The saveset is very large (54GB - 65K BLOCK saveset size - I'll be reducing it after my previous thread's discoveries about saveset sizes). It is doing a few io's to the tape drive, perhaps the larger block size means fewer io's to get a bigger chunk to grind through?

I have done selective restores from this series of tapes before, but usually one or two directories tops. It just seems to be _way_ more cpu than "normal".

A good Saturday afternoon project!

Cheers,
Art

[Volker Halle]

Art,

looks like RESTORE will compare each file found in the backup saveset against all the filespecs in the /SELECT list.

An AlphaServer 800 is not the most powerful machine to run something like this on...

Volker.

[Art Wiens]

It's the most "powerful" thing I have that can be used for this purpose :-(

One day I'll finally have them convinced that we need to invest in this environment. The app migrations to Unix imploded a while back ... VMS continues to perform in our "sliding" 3-5 year migration window 3+ years later.

The restore is running now, SHOW PROC/CONT shows Working set at 182 and Virtual pages at 10678 both steady, not growing.

Art

[Volker Halle]

Art,

if what I think is true, if will take 40 times more CPU specifying 40 directories than specifying just 1 directory in the /SELECT list. If CPU time is the limiting resource - the FC tape should be fast enough and it may even start to thrash, if the AS800 can't issue read IOs fast enough - you would do yourself a favour limiting the no. of parameters to the /SELECT qualifier.

Volker.

[Art Wiens]

Lessons learned - patience!!

Accounting information:
Buffered I/O count: 191322
Peak working set size: 2912
Direct I/O count: 1179112
Peak virtual size: 170848
Page faults: 512
Mounted volumes: 1
Charged CPU time: 0 01:12:59.62
Elapsed time: 0 02:18:17.87

It restored ~33,000 files. It must have been that huge directory, it took more than an hour to get past it before restoring the next directory.

Thanks again,
Art

[Jan van den Ende]

Art,

so this (once again) shows an instance where big directories ARE worth the effort to avoid/circumvent them!

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

You could do some antenna tuning :
1. /nocrc after /save (simple restore of 30K files used 15 % less cpu).
2. if you didn't create the backup with /group=0 then use that to avoid raid like calculations during restore.

Wim

[Art Wiens]

Thanks Wim ... I'll try that.

Art