How Vms backup works ?

[Jiri Koutnik]

Hi everyone,

perhaps got stupid Q, but Im not able to get comprehend answer about how VMS backup works.
Does it read block-by-block from disk and write the data to the tape in buffers specified by /BLOCK_SIZE ?
if the file has free space inside (like datafiles) what does VMS backup with it ? also read all block and write them to the tape ?

It would be very helplful for me to better understand how VMS backup works.

We are comparing it with RMAN, BTW :-)

thanx,
Jiri

[Steven Schweda]

> Does it read block-by-block from disk and
> write the data to the tape in buffers
> specified by /BLOCK_SIZE ?

More or less. I'd expect it to do reads more
cleverly than one block at a time, however.

What's a file with "free space inside (like
datafiles)"? How would BACKUP recognize one?

http://h71000.www7.hp.com/doc/83final/6048/6048pro_018.html#bku_u

[Hein van den Heuvel]

Backup has no knowledge of the contents of a file. You need to use application specific tools like CONVERT for RMS and RMAN for ORACLE to get the potential for added value, such as skipping unused space in a file.

Good luck,
Hein.

[Jiri Koutnik]

What do u mean reading block-by-block is less or more ? What does it depend on ?

free space inside the file I mentioned in conseqence with compression and time what it takes to backup the file, I mean when the file has got allocated size but is without data inside it can take shorter time to backup it, or not ?

thanx,
Jiri

[Hoff]

You would likely want to use the Oracle tool first in a sequence of tools, whether the Oracle tool is RMAN or Rdb RMU /BACKUP, and you may well then choose to use BACKUP to read the created database BACKUP files and send those out to secondary storage.

BACKUP is not compatible with and not synchronized with databases, but it's far and away the best tool for (not open for write) generic OpenVMS files and generic OpenVMS file archiving.

Vendor-provided database tools such as RMU /BACKUP know how to interoperate with the particular database, and how to appropriately flush caches and to get a consistent view of the database contents. BACKUP does not.

As for the direct question, BACKUP reads from the first block to the last block of the file. BACKUP reads in hunks and overlaps its I/O reads and I/O writes to get the most out of the hardware. (The last round of testing I'm aware of showed BACKUP was within a very small percentage of the theoretical limits of the underlying hardware; that BACKUP was running about as fast as it could manage.) (Guidelines for setting appropriate values for BACKUP operator username quotas are in the system manager's manual.)

BACKUP does not synchronize with any open activity underway within the file; it honors the interlocks unless told to override them -- and overriding involves full knowledge of the risks. There's a discussion of BACKUP /IGNORE=INTERLOCK over in the FAQ. (This is also where RMAN and RMU/BACKUP come into play.)

There have been noises from HP about a compressing BACKUP, though that's not documented and available as yet. That approach and an approach using (supported) incremental operations are approaches toward better efficiency. BACKUP has incremental capabilities. Databases also tend to have these, and tools such as RMU/BACKUP can provide it. (FWIW, RMU/BACKUP actually uses the BACKUP API.)

Other archival approaches include the use of volume shadowing, and splitting volumes out and back into the shadowset.

The usual approach -- for the case I am guessing you are looking at -- is to use RMAN or RMU to create a database archive file, and to then have a subsequent pass of BACKUP transfer the file out to secondary or tape storage per some DCL-based selection and migration and deletion logic.

If you are seeking and using a tool for archiving generic OpenVMS files, the first choice for that tool is generally BACKUP.

If you're operating with industrial-scale configurations involving herds of servers and herds of clusters and equivalently monster-scale storage DLT libraries, then you might well look toward network-capable tools first.

When the dust settles, BACKUP is very, very, very, very, very good at dealing with tape errors and device and media glitches, and at restoring your files -- even with missing blocks on the media. And that's what it's all about...

[Ziggy Filek]

It is perhaps also worth mentioning, that VMS BACKUP does not back up files concecutively block by block, but rather opens multiple files (input streams) in paralell, interlocking the inputs in the save sets. That means that if a long series of data bits is lost from the save set (like in case of tape splicing in good 'ol days, or loss of a disk sector nowadays), only a single bit is lost from each input stream allowing for 100% recovery of all data, because BACKUP uses error correcting codes. This makes BACKUP save sets extremely robust.

Ziggy

[Jan van den Ende]

Hoff wrote

>>>
BACKUP is very, very, very, very, very good at dealing with tape errors and device and media glitches, and at restoring your files -- even with missing blocks on the media.
<<<

and I have to add: ...as long as the hardware allows...
And THAT is where the SCSI tape devices _UTTERLY FAIL__ !!!

I once had a (TH70 - that is how long ago) tape which showed a Fatal Drive Error on a SCSI device.
Reading it on a DSSI drive, it signalled ONE recoverable error.

To contrast: even longer ago, I had to restore a 800 bpi tape real.
Instead of the expected 15-20 minutes, it took several hours, and then reported over 33.000 (!!) recoverd errors. THAT is how good BACKUP is (if hardware let's it)

fwiw

Proost.

Have one on me.

jpe

[GuentherF]

For what it is worth...

BACKUP saves all USED blocks of a sequential file and all ALLOCATED blocks for all other file organizations (see DIRECTORY/FULL).

BACKUP reads the file blocks (across multiple files) by ascending disk block number where each extent (contiguous blocks of a file on disk) is read seperately. BACKUP issues many read I/Os in parallel only limited by process quotas/limits like DIOLM, WSQUOTA, FILLM. All blocks read are collected in memory until they can be written to the save set in a file-by-file increasing-file-block-number order.

RMU, even though designed a bit after VMS BACKUP does not call the BACKUP API. It reads DB pages from each area file in parallel and immedialetly writes it to the .RBF file. It can do compression and does not save empty pages. Pages are read in sizes of CLUMBS, an attribute of each database. Because RMU does not order I/Os by disk block numbers and the fact that it may use multiple threads reading from the same disk at different places the read part is less efficient than the way VMS BACKUP does it.

RMAN for Rdb uses the same read strategy except now the data in memory is written by some 3rd party application to the output save set.

I have no idea how RMAN works with Oracle Server.

/Guenther

[Sheldon Smith]

And note that while Backup normally reads each file and just backs up the data within the files, you can specify /PHYSICAL in which case it DOES read block-by-block. EVERY block.

[Jan van den Ende]

Jiri,

from your Forum Profile:


I have assigned points to 0 of 26 responses to my questions.


Maybe you can find some time to do some assigning?

http://forums1.itrc.hp.com/service/forums/helptips.do?#33

Mind, I do NOT say you necessarily need to give lots of points. It is fully up to _YOU_ to decide how many. If you consider an answer is not deserving any points, you can also assign 0 ( = zero ) points, and then that answer will no longer be counted as unassigned.
Consider, that every poster took at least the trouble of posting for you!

To easily find your streams with unassigned points, click your own name somewhere.
This will bring up your profile.
Near the bottom of that page, under the caption "My Question(s)" you will find "questions or topics with unassigned points " Clicking that will give all, and only, your questions that still have unassigned postings.

Thanks on behalf of your Forum colleagues.

PS. - nothing personal in this. I try to post it to everyone with this kind of assignment ratio in this forum. If you have received a posting like this before - please do not take offence - none is intended!

PPS. - Zero points for this.

Proost.

Have one on me.

jpe

[Hein van den Heuvel]

Hello there Jiri,

Judging by the points assignted to the replies the anwers received apparently did not answer the question asked. "help you better understand how VMS backup works?"

Or did you perhpas not like the answers, even though technically correct.?

It seemed to me the explanations offered where reasonably complete and to the point.
Now that you have several good starting points, maybe you want to restate the real question?

fwiw,
Hein.

[Jiri Koutnik]

Hi everyone,

let me summaries all your comments.

1)Backup has no knowledge of the contents of a file. => ok, nothing to add, expectations.
2) There have been noises from HP about a compressing BACKUP, though that's
not documented and available as yet. => here still Im confused, what does depends compression on ? we tested to backup empty datafile against data-full datafile and we can see the differencies at number of datafiles we are able to back up to the tape. So there must be some compression dependencies which calculate how much is datafile full of data or empty but in both cases is allocated on OS.
any idea ?
3) how VMS backup works => it reads in ascending disk block number order block-by-block all allocated block of the file (or files) in several parallel streams to memory, put them to defined buffer that goes to the tape.
am I correct ?

thanx alot, expect more points :-)
Jiri

[Volker Halle]

Jiri,

re 2) compression could happen at different levels: software (BACKUP) or hardware (tape). Adding compression into BACKUP may be an option possibly showing up in a future release of OpenVMS.

Compression algorithms may be able to better compress 'empty' files than 'full' files - if empty means: all ZEROs in the file.

Volker.

[Hein van den Heuvel]

>> we tested to backup empty datafile against data-full datafile and we can see the differencies at number of datafiles we are able to back up to the tape.

Interesting. It may well have been the tape defining the throughput. Backing up to the NL: device or a disk, or a virtual tape may remove that variable component from the test.
Virutal tapes are a new feature for the LD driver. See http://www.digiater.nl/lddriver.html.

>> So there must be some compression dependencies which calculate how much is datafile full of data or empty but in both cases is allocated on OS.

>> any idea ?

1) Could really be the tape hardware.

2) Secondary hardware and usage issues.
For example, whn you allocate space on an EVA controller you only get the PROMISS of space, not the actually disk blocks. It is only when you touch (chunks) of data for the first time that the EVA finds a free chunk.
So I recall one instance where sequential reads were surprisingly slow. It turned out that the data had been written randomly through the allocated range resulting in random access pattern for sequential reads.
Ever since that experience, when we do a serious benchmark, I first write zeroes to the allocated storage (with DD on unix or INIT/ERASE on vms) to make sure the result will be predictable. Maybe not optimal, but predicatble. This obviously slows us down and voids some EVA features (re-balancing) but that's ok by me.

>> thanx alot, expect more points :-)

It wasn't about the points themself. I took them as an indication that there were lingering, un specified problems / questions.

Hope this helps some more,

Hein van den Heuvel
HvdH Performance Consulting

[GuentherF]

"3) how VMS backup works => it reads in ascending disk block number order block-by-block all allocated block of the file (or files) in several parallel streams to memory, put them to defined buffer that goes to the tape.
am I correct ?"

BACKUP reads by extent size. See DCL-DUMP/BLOCK=COUNT:0/HEADER and look at the "Count:" number for each extent. Extents larger than the /BLOCKSIZE (which is also the size of BACKUP's internal buffers) are broken down into smaller I/Os to fit the remaining space in a buffer.

/Guenther

[Jiri Koutnik]

Thanx alot, I thing the provided information got me pretty good knowledge how VMS Backup works, or should works :-)

Jiri