Operating System - OpenVMS

Re: Tracking Software applications

Regular Advisor

Tracking Software applications

I got lot of help with accounting utility at http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1202327682833+28353475&threadId=1198578

Now still continuing but with a slightly different question(S)
1) Are these correct steps for tracking software applications?

2) Good and inexpensive Tracking Softwares for tracking queues, applications

1) Question1:::
(We have OpenVMS 7.3-2 4 node alpha servers (DS25) cluster system with common system disk)

I would like to know more about the best way to track usage of certain software applications.

I do know that one can do so with accounting utility using $accounting /enable=Image.

For this I need to (since accounting file can become really huge due to Image Logs (correct ??) )
1) In the startup directory setup node specific
logical file for accounting
if node = A
define /system /executive_mode accountng disk3:[acct2]acct2_info.log

if node = B
define /system /executive_mode accountng disk3:[acct3]acct3_info.log

Then I do in each node
$set accounting /new_file=disk3:[acct2]acct2_info.log

$set accounting /new_file=disk3:[acct3]acct3_info.log

Question2: Commercial applications
Are there any low cost applications for tracking specific software applications and queues. My preference: should be easy to use and configurable and cost effective (cannot exceed $3K)

Thanks in advance.
Honored Contributor

Re: Tracking Software applications

What's your goal? Seriously.

What exactly are you trying to monitor, and why?

Tracking applications and tracking accesses can be for a couple of different purposes, such as "is xyz even used", and depending on the goal, there can be different answers.

Tracking access can range from passing interest to charge-back systems to mandatory access controls and military-grade auditing.

And as for your command examples and your core questions, do run some tests.

Image accounting does consume I/O bandwidth and does consume storage, but it's not otherwise harmful. Or you could test on a non-production system (as is often the case when production applications are involved) and gain a feel for the load there.

Inferring much from the question, the fashion the forum and the approach you're initially considering, I'll make some assumptions in the following...

I'd not tend to use image accounting until and unless I wanted to know about every image invoked by a particular user (via the user-level auditing), or by all users. If I had specific interests in specific areas (which I why I asked above), I'd target the data acquisition, and would not use image accounting.

In the typical case, I'd use security audits on the objects (files, queues, etc) I cared to track. Do expect the audits to pile up, and you'll want to test this locally to see how fast they'll accumulate.

Chargeback systems and gonzo system auditing packages were once available for OpenVMS, but I've not seen these advertised in years. Poke around at http://mvb.saic.com/freeware/ (using the google site:mvb.saic.com/freeware syntax) to look for existing Freeware.

And do remember the classic pick-two law of computing (good, fast, cheap, pick two) when you make your decisions here. Going cheap is often not cheap, in my experience. It's the "pay later" option. (And the more that you or another staffer in your area has to deal with the package or with the solution, the more that the solution really costs.)

Of course if you do go for image accounting or of you hang audits off a whole bunch of busy objects, acquiring storage is often entirely feasible, as 36 GB and larger disks can be widely available at low cost on the used market. In the commodity storage space, a bare one terabyte drive is around US$250. Racks of HP and Compaq-branded (used) 36 GB drives are available for little money, and racks of larger drives aren't that much more when compared with how much you and your coworkers might cost.
Willem Grooters
Honored Contributor

Re: Tracking Software applications

Image accounting is possible but requires a relatively massive amount of storage, because it does not filter by image. EVERY image - including the ones of VMS itself - will be logged. Interactive use of the accounting facility will be, well, slowed down, due to the sheer amount of data to read through (ACC is not very efficient in this respect)
Auditing allows a much more fine-grained logging: When you set an audit-ACL on the basic procedures and images of an application and on the queues you want to monitor, and all access will be logged.
The only thing is to retrieve this data from the tools.

As for commercial applications: you may find something at pointsecure (http://www.pointsecure.com)
Willem Grooters
OpenVMS Developer & System Manager
Fekko Stubbe
Valued Contributor

Re: Tracking Software applications

You can also install images with /accounting.
In this way you do not need image accounting for all images (may be very expensive), but only for the images you need.
Anton van Ruitenbeek
Trusted Contributor

Re: Tracking Software applications

The question of Hoff stands: what do you want to count ?

On our cluster we have a sort of menu system for starting applications. By using this we added a line in a separate file whith the start time, username etc. when a user selects a application. After this the application will be started (ofcourse).
When the application is stopped, we add also a line when stopped.

By doing so, we only account the applications we want and not all the separate executables.
And we can also get the usage and mod. usage of applications.

NL: Meten is weten, maar je moet weten hoe te meten! - UK: Measuremets is knowledge, but you need to know how to measure !
Anton van Ruitenbeek
Trusted Contributor

Re: Tracking Software applications

The question of Hoff stands: what do you want to count ?

On our cluster we have a sort of menu system for starting applications. By using this we added a line in a separate file whith the start time, username etc. when a user selects a application. After this the application will be started (ofcourse).
When the application is stopped, we add also a line when stopped.

By doing so, we only account the applications we want and not all the separate executables.
And we can also get the usage and mod. usage of applications.

We do have a cluster with all applications running on all nodes, so we aren't interested (in the first place, but can reproduce because we also register) in on which application on which node is started.

NL: Meten is weten, maar je moet weten hoe te meten! - UK: Measuremets is knowledge, but you need to know how to measure !
Jon Pinkley
Honored Contributor

Re: Tracking Software applications


I was hoping you would give us some more info. I started this note on the 6th, but was hoping for some clarification as requested by some of the others. In the mean time, others have already provided some of this info.

What do you mean by "software applications"?

Are you interested in for example how much the EDT editor is used, or how much the C compiler is used, or by application do you mean a suite of programs. For example, all compilers.

To get a list of every image name the following will work:

$ accounting/type=image/sum=image/report=(elap,proc,dir,buf,fau,rec)

Note well: This is name only, the accounting utility makes no distinction between $1$DGA1:[JON]COPY.EXE and $1$DGA1830:[SYS1.SYSCOMMON.][SYSEXE]COPY.EXE; both will be included in the summary for COPY.

To get a list of resources used using the any of the images, EDT, TPU, TECO32_TV, summarized by user.

$ accounting/type=image/sum=user/report=(elap,proc,dir,buf,fau,rec)/image=(edt,tpu,teco32_tv)

A little remembered fact about image accounting is that it does not have to be turned on for all images (but it was remember by Fekko Stubbe). You can use the install program to install an image with the /accounting switch.

$ help install add /accounting





Enables image-level accounting for selected images even if image
accounting is disabled on the local node (by using the DCL
command SET ACCOUNTING/DISABLE=IMAGE). When image accounting
is enabled on the local node, it logs all images, and the
/NOACCOUNTING qualifier has no effect.

However, this will only account for execution of the installed image, for example if I do the following:

$ install add SYS$SYSDEVICE:[USERLIB.UTILITY]UNZIP.EXE;1 /accounting/open/head/shar

followed by

$ unzip:== $sys$sysdevice:[utility]unzip
$ unzip -v abc.zip

This will create an image accounting record like

UNZIP Image Termination
Username: JON UIC: [JON]
Account: ILSMA Finish time: 6-FEB-2008 21:56:10.48
Process ID: 20A19AE9 Start time: 6-FEB-2008 21:56:10.44
Owner ID: Elapsed time: 0 00:00:00.03
Terminal name: VTA3755 Processor time: 0 00:00:00.03
Remote node addr: Priority: 4
Remote node name: Privilege <31-00>: 00108000
Remote ID: TELNET_AC1ECD51 Privilege <63-32>: 00000000
Remote full name:
Posix UID: -2 Posix GID: -2 (%XFFFFFFFE)
Queue entry: Final status code: 00000001
Queue name:
Job name:
Final status text: %SYSTEM-S-NORMAL, normal successful completion
Page faults: 80 Direct IO: 13
Page fault reads: 8 Buffered IO: 232
Peak working set: 5408 Volumes mounted: 0
Peak page file: 6672 Images executed: 27


However, if your purpose it to be able to track the usage and resources used, unless the program will not work without being installed, there are easy ways to circumvent the image accounting, because there will only be accounting if the installed image is used. For example, doing the following will not create an image accounting record, because the image activator will not consider it to be a known image (installed).

$ unzip:== $sys$sysdevice:[utility]unzip; ! the specification of a version bypasses the known file list
$ unzip -v abc.zip

If image accounting for all images was active, then the previous command will generate a record like the above, but with the image name listed with the version number (as below)


So, if you intend that installed images are run, don't include a ";" in the foreign command.

The point is, if you want to be able to track the images that run, then I suggest you turn it on system wide, and plan for the extra disk space and i/o. And you may want to dedicate a drive for that purpose. Otherwise, you can't be assured that there will be any accounting records.

And be aware that if your users have read access to the images, and you are charging more to run specific images, be aware that if a use copies a program to a different name, it won't be obvious in a summary report.

I.e. copy [expensive]application.exe [mycopy]copy.exe will roll the resources for their copy into COPY instead of APPLICATION in a report like:

$ acc /user=jon /sum=image /type=image /report=(elap,proc,dir,buf,fau,rec)

P.S. What are your goals in accounting for application use? Unless you are doing chargeback, the built in accounting utility does quite a bit.

We don't use accounting for chargeback. At our company, we are like the library, everybody pays the taxes whether they use us or not. But we still have image accounting turned on all the time. The reason is that image accounting is extremely useful for more things than chargeback. It is great for debugging, or knowing how long something is likely to take, if you know that a job like it was recently run (and the other conditions are the same). Or to see if a change to a program caused its resources to double, or to be reduced because the programmer did something in a more efficient way, (or because you have enabled global buffers on a shared file).

Between ACCOUNTING (with all types including IMAGE) and Auditing (with a lot of items audited), we have about 900 MB per week. Our biggest accountng.dat file was 900000 blocks last week; removing all IMAGE accounting records left 45000 blocks, or about 20 to 1 or 5% of the size when image accounting was included. But what the percentage is really depends on the ratio of images executed per process, and how many non-process (print) records there are.

I would suggest turning on IMAGE accounting on one node for 1 day, and see how big the file is. Then you will have something to experiment with, and to decide whether image accounting is worth the disk and I/O resources it consumes.

To post process the accounting file to remove the image accounting records, you use this command:

$ acco input_file /type=("-",image) /bin /out=output_file

Here's a summary of steps to enable image accounting for a day, and then to turn it off, and recreate the "missing" accounting file

$ sho accounting
$ sho log /system /ful accountng ! this is probably what you want to restore the value to later
$ directory/nohead/notrail 'f$parse("ACCOUNTNG","SYS$MANAGER:.DAT;")' ! display current account file with version

Note, the above isn't guaranteed to show you what the current accounting file is, as the logical name accountng is only used at the time a new accounting log file is created.

To verify, you can do the following using the "device" from the directory output above

$ pipe show device/file sys$sysroot | search sys$pipe /nowin job_control

If you see the file name, then that was the correct file. If not, you will need to use
$ analyze/system
SDA> set proc job_control
SDA> show proc/chan

and look for what appears to be an accounting file. I don't know why show accounting can't give us the name of the current accounting file, but as far as I know, there is no way to get that (reliably) in an easy fashion.

Note, if you will want to be able to recreate an accounting file like the one that would have been created without
the image accounting records, you will want to leave a "place" for it when you start a new accounting file with image accounting disabled. So remember what this version is, when you start using this location for accounting again, you will want to force accounting to create a version 2 higher than what was just displayed. Details later.

$ create/dir/own=system new_disk:[accountng.node1] ! I would use the name of the node for the directory name
$ define/system/exec accountng new_disk:[accountng.node1]accountng.image
$ set accounting /enable=image /new_file

let it run for a day

If you want to recreate the accounting file without the image records in it, and you want it to have the correct file version, then take the previous accounting file name with you determined above, and add 2 to the version number. In the example I showed, you would get SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT;62 Now you have to tell the job controller that you want it to use that file name when it opens a new accounting file.

$ define/system/exec accountng "SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT;62"

Now create a new version on the accounting file without image accounting turned on.

$ set accounting /disable=image /new_file

Now we have to delete the logical name with the specific version specified. The above is the default location, if yours is there, then just deassign the logical name.

$ deassign/system/exec accountng

If it was somewhere else, you need to move redefine it without the version number.

$ define/system/exec accountng "SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT"

$! recreate accounting file without image accounting records (if you want to have the "standard" set around)
$! This is why we had to leave the hole, so we could stuff the imageless accounting file in the correct sequence
$ acco new_disk:[accountng.node1]accountng.image/type=("-",image)/bin/out=SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT;61

Now you can play with the image accounting file and if you have anything that was expecting the "standard" accounting file to exist where it should, it is there.

Other than the fact that the forward link in the SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT;60 file will not be pointing to SYS$SYSROOT:[SYSMGR]ACCOUNTNG.DAT;61, and the creation and modification dates of the ;61 file will be the time the pruned accounting file was crated, the accounting file will be the same as if it were created without image accounting records.

Summary: Unless you give us the reasons why you want to track applications, we can't give you very good advice.

The ACCOUNTING utility can do quite a bit, and I wouldn't want to deal with the hassle of chargeback accounting. I've done that in a past job, using what started out as UIS PACS PLUS and then got sold to RAXCO. I don't think it is available for sale any more. Perhaps if someone asked nicely, they would release the source to the freeware disk. I am reasonably sure it would still work, or would be reasonably easy to make it work. I don't think the format of the VMS accounting file has changed much, if any, in the last 15 years. But setting up chargeback is a pain, especially if you maintain different rates for different hours of the day, and weekends, and holidays, etc. I haven't missed PACS PLUS, not that it was a bad product, it was just busy work to generate reports, etc. But I would miss having the ACCOUNTING utility.

One thing I wish the ACCOUNTING utility did (and PACS did not do this either) is to allow you to see what was running at a specific point in time in the past. It would have to scan the files and report only the records where the start time was before the specified time and the end time was after the specified time. Something like that would be useful for answering questions like "What was running at the time we got a file locked by another user error".

it depends