topic Re: High water marking in Operating System - OpenVMS

High water marking

Wim Van den Wyngaert — Thu, 27 May 2004 10:23:37 GMT

I activate high water marking on a disk.
Then I create a file of 400.000 blocks on this disk. The command takes about 30 seconds. During this time, f$search hangs and f$file too.

It seems that the disk is locked (but not for dir).

Why ?

Is it possible that a database server (sybase) crashes because of it (e.g. too much outstanding IO for the given quotas) ?

Re: High water marking

Ian Miller. — Thu, 27 May 2004 10:27:46 GMT

I guess a writelock has been taken out on the disk so DIR is ok as it just reads.

re sybase - depends on how it handles quota failures - it should cope but...

Re: High water marking

Wim Van den Wyngaert — Thu, 27 May 2004 10:42:38 GMT

No writelock on the disk. I checked and could write a new file to the disk. But I couldn't delete it (hang again).

Re: High water marking

Uwe Zessin — Thu, 27 May 2004 10:55:30 GMT

No, not a general write lock as in MOUNT/NOWRITE - just a synchronization lock, e.g. on one of the bitmaps.

Re: High water marking

Hein van den Heuvel — Thu, 27 May 2004 10:58:53 GMT

You'll need to get a hold of the Kirby McCoy's "VMS File System Internals" book ISBN DP 1-55558-056-4 to read up all the gory details. Notably check out chapter 5.4.5 "Dynamic Highwater Marking".

The basic premiss is easy. Never allow a process to read a block from a fiel that was not writen too that file earlier. The protetcs against 'scavenging'. That is, it protects against soem job just askgn for a bunch of block and then reads those to see if there is something 'interesting' left behind from a prior usage of that block (while it was part of a different, now deleted or truncated file).
Sounds easy, but ia hard to do right in a full sharing environment. The system has optimizations to avoid writes as much as possibly, notably when it detects 'sequential only access'. However, if you just allocate a bunch of blocks for a non-sequential file, or write a block for out into the file, the XQP will erase' all intermediate blocks... while holding a volume lock! (This can probably be fixed using a lower level lock, but that is not how it is done today). So writes to a file that is already open can go on, but any volume operation will stall.
You have to understand the application/file-usage to decide whether high-watermarkign is really needed, and what the expected costs are. You are not the first one to be bitten by this (myself, I get a painfull reminder once every 3 years or so: Hmmm.. is the system down? no? then why is my 'DIR' hanging? Oh duh... HWM!).

Groetjes,

Hein

Re: High water marking

Uwe Zessin — Thu, 27 May 2004 11:04:14 GMT

Hein,
he said he could create a new file, but not delete it - isn't the former an operation on the volume, too?

Wim,
did you just create an empty file or did you write data, too?

Re: High water marking

Jan van den Ende — Thu, 27 May 2004 12:49:00 GMT

Hi all,

as I have come to understand things, it is working fine.
Maybe a monitor IO/item=que for the disk will illustrate.
Creating a file is OK. (should that be initially so as well, while the SET VOL/HIGH is still active? Or is this assuming the data present before issuing the command was and still is not that important? Or IS this what CREATE is doing: writing VERY much meaningless data? ) But then, the file gets deleted, and the full area of the disk that WAS the file, now first has to be (multiply?) written, so as to release only 'truely erased' disk blocks.
So, for a big file to delete, your 'hang' is simply waiting for an IO to complete, that is somewhere at the back of a BIG que of disk IO entries...

correct me if I'm wrong, but this is what I remember of the info way back when, when highwatermarking was introduced (somewhere V5-ish ? )

Jan

Re: High water marking

Martin P.J. Zinser — Thu, 27 May 2004 13:15:26 GMT

Hello Wim,

well the first question obviously is, do you have an applicaiton requirement to pre-zero
the diskblocks with Highwatermarking? If not,
having it enabled will generally incurr a penalty on random access files.

Greetings, Martin

Re: High water marking

John Eerenberg — Thu, 27 May 2004 14:10:04 GMT

> "Then I create a file of 400.000 blocks on this disk. The command takes about 30 seconds. During this time, f$search hangs and f$file too.
It seems that the disk is locked (but not for dir).
Why ?"

The short answer is the highwater erase, f$search, f$file, etc. are, by design and many good reasons, single threaded through the FCP (File Control Primitive). For the FCP to erase 400,000 blocks takes roughly the time you mentioned. The detailed answer has already been given.

Re: High water marking

Wim Van den Wyngaert — Fri, 28 May 2004 01:01:17 GMT

It is still not clear. There is some kind of volume lock but I can still write (Uwe : with data blocks).

What are exactly the read/write locks that are taken ? A database system is much simplier to understand than RMS.

Second, I find it a bad implementation if applications are stalled because of it. Suppose you have a real time application that need to close a valve but can't do the job because some DBA is creating a big database file (would block the disk for several minutes or even hours).

Re: High water marking

Jan van den Ende — Fri, 28 May 2004 01:59:34 GMT

Well Wim,

Highwatermarking IS a very-high-security feature, to be used IF YOU NEED IT, explicitly AT THE COST of performance.
I guess yhis implies that highwatermarking and real-time functionality are mutually exclusive.
The way out of this dilemma is quite simple:
have the Sytem Manager make sure that your Highwatermark-requiring data are on another physical drive then your real-time files.

Jan

Re: High water marking

Wim Van den Wyngaert — Fri, 28 May 2004 02:12:47 GMT

Jan,

I disagree. Performance impact yes, blocking activities no. Simply a bad implementation. Even if there are workarounds.

I don't use it (I put it off where I see it). But used it for testing f$esrach.

Re: High water marking

Martin Johnson — Tue, 01 Jun 2004 14:19:58 GMT

It is not a bug, it's a feature!

These type of delays are typical of High Water Marking (HWM). If your applications are too sensitive to these delays then you have two options. One, turn off HWM, or, two, control your allocations, either by delaying large allocations to the off hours or breaking up the allocations into small chunks.

HTH
Marty

Re: High water marking

Wim Van den Wyngaert — Wed, 02 Jun 2004 01:31:48 GMT

Features should be documented. This is an implementation of which the side effects are not documented.

Re: High water marking

John Gillings — Fri, 04 Jun 2004 00:43:50 GMT

Wim,

This may not be OpenVMS at all. For some combinations of controller and disk drive, the HWM ERASE function is done in the drive itself. Just an I/O ERASE operation, rather than a series of writes to individual blocks, then wait for it to return (or, to be more precise, one I/O per allocated extent). So, all the stuff you CAN do may be the result of caching, or operations that the drive and/or controller can handle in parallel. Anything else just has to wait.

Bottom line, this IS the "performance impact" that the documentation talks about. You can argue about other ways of doing things, but they're all tradeoffs. That's what engineering is all about - picking the least worst choice.

With security, there are always nasty things to think about. For example, suppose we didn't lock the disk and zeroed the allocated blocks in small chunks, obviously that would take MUCH longer to complete. Apart from the cumulative cost of multiple I/O's, we'd have to write actual blocks, so data would need to be buffered and sent over the wires to the disk.

Worse, maybe it would be possible to request the CREATE asynchronously and access the blocks from a parallel thread, ahead of the ERASE fence, thus defeating HWM?

The engineers who designed this weren't complete morons, so if they've taken the fairly drastic step of holding a lock for a long time, it definitely wasn't without good reason. If you really want to know the answer, log a case with your local customer support centre and have it escalated into engineering. They will either say "of course! how stupid" and fix it, or they may explain why it has to be that way to guarantee security, or they may say "it has to be that way, but we're not going to tell you why becuase it involves security".

This is the burden of being OpenVMS. Other operating systems can take short cuts, possibly leaving security or reliability holes around the place, but we have to keep everything secure and failsafe, sometimes at the cost of performance.

Re: High water marking

Wim Van den Wyngaert — Fri, 04 Jun 2004 01:29:27 GMT

John,

Whatever the reason, no disk should be locked during a longer interval.

High water marking may not be used in financial market systems, real time applications and high availability systems, unless EVERYBODY is aware of the problem.

The performance impact that I found in the doc only talks about the extra IO for erasing those extents, not about the disk lock.

Wim

Re: High water marking

Hein van den Heuvel — Fri, 04 Jun 2004 03:05:04 GMT

> High water marking may not be used in financial market systems, real time applications and high availability systems, unless EVERYBODY is aware of the problem

hmmm... beg to differ.
Highwater marking may be MOST important for those financial market systems.
The impact is generally NOT found in day to day usage. For a 'normal' sequential file write no extra IO is needed. Hey, that's why the HWM is there! It's when you create the possibility to access beyond HWM that the erase is needed.
Any other real-time/oltp task on the disk with the files already open can just keep on reading/writing from those files there is no data IO lock (that concept does not even exist!).

Wim, is this question for the system you asked about for the caching? There you indicated sequential files were used a lot and indexed files where used little? Do you have a self-designed / supported data retrieval system to deal with? If so then indeed you will need to learn to deal with the 'fine print' like using the 'SQO' bit.

fwiw,

Hein.

Re: High water marking

Wim Van den Wyngaert — Fri, 04 Jun 2004 03:55:59 GMT

>The impact is generally NOT found in day to >day usage.

Any create/fdl (or extent) with a substantial file, any database file creation (sybase). This is very common on our site.

>Any other real-time/oltp task on the disk >with the files already open can just keep >on reading/writing from those files

All jobs creating files, re-opening files are blocked. Also very common. I tested with dcl read/key (file was open before create/fdl was started) and found that the procedure was still blocked during 11 seconds !!!

>Wim, is this question for the system you >asked about for the caching? There you >indicated sequential files were used a lot >and indexed files where used little? Do you >have a self-designed / supported data >retrieval system to deal with? If so then ?>indeed you will need to learn to deal with >the 'fine print' like using the 'SQO' bit.

Nope. Just trying to understand things without a cause.

If you try to buy some shares on the stock exchange, close valves of reffineries or other such stuff, you can miss those suspends.

Wim

Re: High water marking

Hein van den Heuvel — Fri, 04 Jun 2004 04:11:59 GMT

OK. Understood.

I must speculate that the slow READ/KEY was caused by the 2 to 5 IOS required to do so each had to wait in a large queue for the disk itself. ( 2 to 5 = index root, index lower-level-if-needed, data bucket, RRV-bucket-if needed)

Hein.

Re: High water marking

John Eerenberg — Fri, 04 Jun 2004 13:25:38 GMT

Wim,
If I may . . .

> I disagree. Performance impact yes,
> blocking activities no. Simply a bad
> implementation. Even if there are
> workarounds.

It seems like you are not accepting the reasons for blocking. We can debate the documentation's lack of clarity another time (acutally I agree with you).

Bottom line is if there is another way to do the erase without blocking other IO on the volume, how would you do it?

The goal of the erase is to prevent disk scavenging (there are other goals, but that's the one I am concerned with). What does a given application need? Does the application have to prevent disk scavenging? If one is worried about disk scavenging, then you have to erase before using the disk blocks. No other choice.

If you have another choice, how would you do it? This is not, as you claim, a "bad implementation" nor is it a design flaw or poor practice. Again, to guarantee security with the goal of disk scavenging, what would you do otherwise?

If you allow another IO to occur during the erase, you can bypass the attempted erase function, read the blocks that have not been erased (theoretically *all* of the blocks) and scavenge the data. The only kind of IO I could consider is a write IO. But the read IO's have to be blocked so to be safe, block all IO.

If this is a bad implementation, how would you guarantee security if you allow even one IO during the erase?