Pros and cons of HIGHWATERMARKING and INIT/ERASE

[Jan van den Boogaard]

Hi everyone,
OpenVMS has features for file security: SET VOLUME/HIGHWATER MARKING and INIT /ERASE and SET VOLUME/ERASE_ON_DELETE. What are the pros and cons of these options? I heard say that HIGHWATER marking means an overhead, but doesnt SET VOL/ERASE have an overhead as well? ( INIT /ERASE clearly has a one-time-only overhead.)

In cases of applications where new files are constantly being created , but deletions seldom happen, would SET VOL/ERASE be better than HIGHWATER_MARKING ?

Thnx, jan

[comarow]

Jan:

Those settings are to prevent scavangers
from allocating disk space, then dumping the
information. The benifit is very clear,
security.

The penalty is performance.

With High Water marking, as one allocates
blocks of disk space, the space is written with zeros.

Set Volume/ERASE is great security, as every time a file is deleted, the blocks
are written with zeros.

They both have a performance penalty. The choice is dependent on your environment.

Neither will satisfy DIS regulations.

If you have DFU and want to undelete a file,
(in that short space of time on an active disk), /erase will make that impossible.


The performance penalty is significant.

Hav fun

[Hein van den Heuvel · Accepted solution]

comarow wrote...

>> Those settings are to prevent scavangers
from allocating disk space

Right.

>> With High Water marking, as one allocates
blocks of disk space, the space is written with zeros.

WRONG WRONG WRONG.

With HWM zeros are written as one READs beyond the current HWM for the file, wich normally is at the EOF of the file.

For normal, sequential, file writes HWM has NO OVERHEAD. Only badly behaving applications, trying to write beyond where they have written, get a penalty... which seems fair to me.

HWM will NOT protect against privilleged user access unallocted/uninitialized blocks through non-file-structured, logical block, IO.

>> Set Volume/ERASE is great security, as every time a file is deleted, the blocks
are written with zeros.

Right, and this will thus also prevent privved users from using brute force to read the data after delete (of course they probably had 'all the time in the world' to read that data before it was deleted).

They both have a performance penalty. The choice is dependent on your environment.

Jan,

INIT/Erase is a nice simple thing to do when not in a rush to release a new drive to usage.

SET VOL/erase sounds like a good suggestion in your case of low delete, but normally is a 'waste of time' as instead of writting an erase pattern after last usage, you can just write fresh data on re-use and write zeroes on attempted re-use before writing (HWM).

Hight Water Marking has been designed as a no-overhead security, but its erase pattern might not serve your needs.

Met vriendelijke groetjes,
Hein.

[Steven Schweda]

> For normal, sequential, file writes HWM
> has NO OVERHEAD. [...]

This assumes that the FAB$M_SQO bit is set,
right? That is, a naive application may
easily see a disk lock up for a long time if
a file is extended by, say, several GB.

For a good time, Zip-compress a file of a GB
or two, then UnZip it using an UnZip before
5.52. Then compare the behavior of UnZip
5.52 (or later). 5.52 sets the
sequential-access-only flag. Earlier
versions do not. Disable highwater marking
on the volume, and the difference goes away.

[John Gillings]

Steven,

>That is, a naive application may
>easily see a disk lock up for a long time
>if a file is extended by, say, several GB.

No. That shouldn't happen. HWM will only write zeros if an attempt is made to access data beyond the HWM (which should normally be at or above EOF). Extending the file isn't a problem, the blocks are allocated to the file and still contain the old data. Writing at EOF will overwrite with new data, extending both EOF and HWM. If you try to read the data above the HWM (and implicitly, above EOF), the file system will first fill the gap between HWM and your target block with zeros, update the HWM, then read the block (now containing zeros).

Normally HWM and INIT/ERASE do quite different things, BUT in the specific workload where data is rarely deleted, INIT/ERASE and SET VOL/ERASE_ON_DELETE achieve the same result, but at higher "cost". All the "ERASE" options are a definite cost - you will incur the I/O to erase the blocks. With HWM, you will only incur the cost if you absolutely have to - that is, someone attempts to read data that they haven't written.

The only overhead well behaved applications will suffer from enabling HWM is that of updating the HWM in the file header, but for sequential files, HWM will follow EOF, so there shouldn't be any extra overhead.

[comarow]

Thanks.

I'm researching the behavior of HWM.

[Wim Van den Wyngaert]

Also read this one.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=603651

Wim

[Wim Van den Wyngaert]

If I remember correctly, "mon fcp" item "erase rate" is an indication of how many hwm-erases are realy done.

Wim

[comarow]

Thanks.

I'm very interested in where you obtained the description of the behavior.

I examined
http://h71000.www7.hp.com/doc/732FINAL/aa-q2hlg-te/aa-q2hlg-te.HTMl

Which describes high water marking, but does not describe how it is accomplished.
Similarly, the Guide to System Performance
mentions how to turn off high water marking to improve performance. But it to does not describe the actual mechanics.

All the Performance Cookbooks recommend turning it off if not needed.

This does not mean you are wrong.

However, we did some testing.

Here are our results:


Bob, my testing shows that a file gets zero'd on allocation but maybe there is another way to allocate blocks to a file that will not zero them

What I did was fill DVA0 with some files with lots of text

then I deleted the file and turned on HWM

Then using an FDL file I created a file w/ a large allocation

DIR showed the file as 0/2849

Then I turned off HWM and did a SET FILE/END and DUMP/BLOCK=(S:2000,C:1) and I got all zeros back

So unless CREATE was doing something behind my back I never used VBN 2000 and it should have had random text in it


Perhaps you would like to repeat the test?

I'd be very interested in how you came to your conclussions.

[Steven Schweda]

> No. That shouldn't happen. [...]

When I started to add large-file support to
[Un]Zip, I quickly tired of having the system
go to sleep for minutes during the initial
allocation of a large output file on a disk
with highwater marking enabled. Adding the
code to set the SQO bit was the cure.

You can run the experiment using UnZip 5.52.
The SQO code is all in [.VMS]VMS.C, just
above the sys$create() call which creates the
output file(s). Simply Zip a 2GB (or so)
file, and then UnZIp it. Disable the code
which sets the SQO bit, rebuild UnZip, and
run it again.

For even more impressive delays, you'd need
to switch to the not-yet-released BETA source
kits for Zip 3.0 and UnZip 6.0, as Zip 2.x
and UnZip 5.x are limited to files no larger
than 2GB.

For the original discussion on comp.os.vms,
see:

http://groups.google.com/group/comp.os.vms/browse_thread/thread/e40c7dbd70bab7c9/f8a01603c0ff49b3