OpenVMS i64 8.2-1, ACLs, and performance issue

[EWL]

We have a user that regularly creates 25 - 30 individual files and then appends then into 1 file. The files are created on a spool disk which has many many files of all different sizes and is very active. Disk is about 6gb and has plenty of space. We've defragged just in case. We have ACLs on the directory where the files are being created and the user has one of the 10 rights identifiers able to use the directory. The rights ID he's using is the 3rd in the search order. The ACLS are sequenced highest priv to lowest. If I add the rights id that is 1st in the list his time for this process is cut in half. I didn't think reading thru ACLs should take that long. Anyone have any thoughts?

[John Gillings]

EWL,

Please note that the ORDER of ACEs in an ACL can be critical to its correctness! Before changing anything, make sure you're not breaking the security intention.

I'd be amazed if you could measure any difference in performance by moving ACEs within an ACL. Although your logic is self evidently true (matching on the first entry in a sequential search, rather than the 3rd will obviously be quicker), the matching operation (in memory) will be a miniscule fraction ot the total processing cost of checking an ACL. The disk I/O required to read the ACL in the first place (which you will incur regardless of the position of the matching ACE) will be, by far, the dominant time expense.

There may be some performance issue if the ACL is long enough to require multiple I/Os to read. Use DUMP/HEADER/BLOCK=COUNT:0 to verify that your ACL fits in a single file header. If it overflows into a second header, that will affect your performance orders of magnitude more than ACE order (but even that increase may not be measurable in the overall performance of your process).

If it's a concern, look to reducing the size of the ACL by coalescing identifiers, or replacing individual user idetifiers with a general identifier (granted to each of the users).

If your process is so performance sensitive that ACE order really matters, I'd be looking for a way to get rid of the ACL altogether!

[Hein van den Heuvel]

>> If I add the rights id that is 1st in the list his time for this process is cut in half.

Wow, I would never expect that and suspect a 'side effect'. Frankly I am surprised you can even measure the time it takes to append a handfull of files.


That's not to say appending a file is invisible: Open output, open input, copy records, update headers and such, once per file.

But supposedly those files are recently created and still in cached, and how could the reading of an extra acl in the header make that much of a difference.

What are we talking about here 1/4 second --> 1/2 ? KB files or GB files?

Things to check
- Multiheader files? (DUMP/HEAD/BLOCK=COUNT=0)
- How is the device mounted? 'normal'?
- ACP Cache misconfigured? (Headers, Directory)
- SET WATCH FILE/CLA=ALL

If there non-target device activity as a result of the appends (rightslist? auditting?)

And uh... why fart around with 8.2? Go 8.3 already! Ya never know...

More details welcome!
Hein.

[DECxchange]

Hello,
In VMS system management classes I've taken in the past, they do say that ACLs will reduce performance. But I didn't think it would make that big of a difference. What kind of access time differences are you noticing? Seconds? Minutes? Fractions of a second? Have you done any average time measurements to see if this a REAL or a PERCEIVED difference?

I agree that you should go with VMS 8.3. It seems to correct work pertty darn good.

[Wim Van den Wyngaert]

Did a test on AS500/400 7.3.

Create 50 files with 1 line of text. Then append all files into a new file.

Did the test without acl, with 1 line acl, with 19 lines of acl in which the first gave access, with 19 lines in which the last line gave access. No difference in cpu time nor elapsed time.

Wim

[Willem Grooters]

Though it might seem to be a matter of ACL processing given the difference you mention, there might be other issues involved - given the size of it.

* Where does the delay occur: ion file creation, or on the appending stage?
* What is the size of the individual files?
* How are these files created : pre-allocated, or on-the-fly?
* How are the files appended to one file: DCL, program control? One-by-one of one command?
* what is the default extent size?
* How many files are contained in one directory - what size is the directory file?
* What is the disk's clustersize?

WG

[EWL]

Thanks for all the replies/questions/comments. Iâ ve tried to respond to most of them:
-8.3 is not an option right now. We are awaiting the app vendor to verify their code.
-Dump/Header/Block-Count:0 shows a single file header
-Canâ t reduce number of ACLs because of separation of duties and other security needs
-Disk is mounted shared, cluster, multiple I/O paths, with write-back caching enabled Cluster size is 10
-ACP_DIRCACHE=Current= 14988,default= 22; ACP_HDRCACHE=current=54464, default= 36
-One job stream creates the individual files and then appends. The DB is on a disk by itself, but all of these files are created on the same disk, same directory.
-Directory size is 179/2370
-Donâ t know how these files are created as this process is dependent on vendor code
-Using 3 different accounts for testing: 1 account using the 3rd rightsid in the list ran 14 files in 3 minutes; 2nd account that was the 4th in list ran 11 files in 2 ½ minutes; the account using the 1st rightsid in the list ran 12 files in 27 seconds. As you can see, the timing is definitely different

Just for fun, hereâ s the ACLs
Access Control List
(CREATOR,ACCESS=NONE)
(DEFAULT_PROTECTION,SYSTEM:RWED,OWNER:RWED,GROUP:,WORLD:)
(IDENTIFIER=A_RIGHTS,ACCESS=READ+WRITE+EXECUTE+DELETE+CONTROL)
(IDENTIFIER=B_RIGHTS,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=C_RIGHTS,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=D_RIGHTS,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=E_RIGHTS,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=F_RIGHTS1,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=*,ACCESS=NONE)
(IDENTIFIER=A_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+EXECUTE+DELETE+CONTROL)
(IDENTIFIER=B_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+EXECUTE+DELETE)
(IDENTIFIER=C_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+EXECUTE+DELETE)
(IDENTIFIER=D_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+EXECUTE+DELETE)
(IDENTIFIER=E_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+EXECUTE+DELETE)
(IDENTIFIER=F_RIGHTS,OPTIONS=DEFAULT,ACCESS=READ+WRITE+DELETE)
(IDENTIFIER=*,OPTIONS=DEFAULT,ACCESS=NONE)

[Wim Van den Wyngaert]

Do all accounts have the same quotas (in authorize)?

Wim

[EWL]

All accounts have the exact same quoatas. That's how we narrowed down the ACLs.

[Jan van den Ende]

EWL,

where the tests also tried in reverse order, ie. has it been ruled out that caching causes the difference?

Just trying to seek SOME cause here...

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

redid my test with the same kind of acl. Same result.

[Hein van den Heuvel]

[dang, I lost a first, extensive, reply.
The one time I did not save it into notepad first :-( ]

My money is on Auditing being the issue.
Does any of the tested appends trigger alarms/messages?

Next best bet is cluster / cache interactions. Were all users tested from the same node? If the slow users are on a different node from the one were the data was generated, then the additional real IO could explain the difference easily enough

> Thanks for all the replies/questions/comments. I've tried to respond to most of them:

And thank you for a good, timely, follow up.

-8.3 is not an option right now. We are awaiting the app vendor to verify their code.

Sure it is an option.
Just say NO so stalling and waffling.
You and I know there is nothing that actually will need to be done. The customer pays for support ? Well, insist on the silly rubber stamp for that money.

-Dump/Header/Block-Count:0 shows a single file header.

For the directory right? To be expected, as those are contiguous. But add enough ACE's and one might need an extention. Still...

-Can't reduce number of ACLs because of separation of duties and other security needs

Hmmm, I'm sure you are right, but it seems to me you could replace all ACEs with a single 'dir_right' and give 'dir_right' to the holders of a_right, b_right,...
Then again, I never need to worry much about security, so don't trust my advice here.

-Disk is mounted shared, cluster

Ah, a cluster. Were all accesses from a single node?. The (XFC) file system cache can easily play games explaining vast differences.

>> -ACP_DIRCACHE=Current= 14988,default= 22; ACP_HDRCACHE=current=54464, default= 36

Those a relatively high numbers. Autogen gone wild or really helpful? I would doublecheck with $MONI FILE and/or T4.

-One job stream creates the individual files and then appends.

Thus on 1 node.

- Directory size is 179/2370

Not too too big, and plenty of room to grow.
Anyway, appends do not create directory entries.

-Don't know how these files are created as this process is dependent on vendor code

So ask the vendor for support.

-Using 3 different accounts for testing: 1 account using the 3rd rightsid in the list ran 14 files in 3 minutes; 2nd account that was the 4th in list ran 11 files in 2 à ½ minutes; the account using the 1st rightsid in the list ran 12 files in 27 seconds.

That's a convincing, and significant, difference. And I guess it is also reproducable, otherwise you would not be here asking.
If this was my system I would want to see thet next level of detail: DIRIO, BUFIO, CPU times, possibly simply form $SHOW PROC/ACCOUNT before and after the append.

I suspect only the elapsed time to show a difference, just to make it a harder problem! Now DIRIOs are counted the same, whether from cache or not.
Consider:
$show mem/cach=(file=...

- As you can see, the timing is definitely different

Agreed.

Hope this helps some,
Hein van den heuvel ( at gmail dot com )
HvdH Performance Consulting

[Volker Halle]

EWL,

this would be a very nice case for running T4 with a small sampling interval (e.g. 5 seconds) for a short period of time while your APPEND test is runnung. T4 will collect all relevant data from the system and you could make the .CSV (or a zipped .CSV) file from your experiment available. This will provide real data from the system and greatly reduce speculation on what may be the reason for this perceived performance problem.

Volker.