Operating System - OpenVMS
1748129 Members
3700 Online
108758 Solutions
New Discussion

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

 
Volker Halle
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

Ana,

 

how big is the Directory file (xxx.DIR) ? Deleting files in a big directory file is a very costly operation, as the .DIR file contents needs to be shuffled around. And all other users of that directory will see 'bad performance'.

 

Volker.

H.Becker
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

Deleting directory entries in big directories can be expensive.
Directorie entries - that is files in the directory - are sorted by name (filename and type) in ascending order. Each name entry has one or more version entries, which are sorted in descending order. Each disk block of a directory file contains at least one name entry or as many name entries as fit into the block. Deleteing a file removes a version entry or if it was the only one a name entry. If the block becomes empty (and it was not the last block in the directory file), the remaining blocks are "shuffled around". That is, deleting files from the front of a big directory is expensive, when the corresponding directory block becomes empty. (The usual applies, the longer the name, the less name entries will fit into a directory block, the sooner the block can become empty).
If you have a file naming scheme that always creates files with names "greater" than the previous names, for example for log files with date and time stamps, then the new name entries will always go to the end of the directory file. If you delete the oldest log files, this will always be at the front of the directory file. Then (sooner or later) shuffling of directory disk blocks will occur. With big directory files this will take a noticable time, an increase in disk I/Os and a delay for parallel access to the very same directory.
This behavior is known and was described/discussed several times in this forum (or its predecessor).
Ana M. García Olivencia
Regular Advisor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

The directory was 163000 files. I know that these are a lot of files and that the deletion operations are costly but, in our system, this was a special case (a cache directory of a web with many files). The problem was that, after realizing that the deletion operation was the cause of the performance degradation, we stopped the process that was running the deletion and the situation continued after some time, affecting the overall system .

 

Anyway, as we have, in general terms, good performance in our systems (it's a cluster behaving, basically, as a web server) and the problem described is difficult to evaluate and sporadic (except the disc with all the webs logs that has high I/O rate compared to the others) , I want to focus in increasing these parameters to better manage these special cases where the I/O rate of a disk increases sporadically.

 

 

Hoff
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

There are no particular ACP parameters around altering the performance of what OpenVMS now considers large directories, there are environment and application changes, and OpenVMS upgrades.    Your OpenVMS version has the large-directory fixes that were implemented (in V7.2) and various other RMS and XQP performance tweaks), so that's not an option.

 

The usual approach for rapidly deleting what OpenVMS considers to be large numbers of files in these directories is the reverse-delete hack.  Code your own delete, and delete the files in reverse alphanumeric order, or scrounge up one of the freeware tools that does this.  The default forward-delete behavior of the DELETE command is pessimal, given the OpenVMS directory structures.

 

Put another way, you're not getting hammered so much on the ACP parameter settings here, you're getting hammered on fiundamental design decisions within the application, and how these decisions interact with fundamental design decisions within OpenVMS and its XQP processing, and particularly around the directory I/O processing, and around how VMS doesn't cache this directory data in memory; VMS really, really wants to write the data to disk.  (When one of these directories gets rebuild from a mass-delete with cluster locking, other hosts can have delays writing to the directories, and I can see SCS locking Getting Busy with the directory locks; where you can, split your write processing and avoid using the shared storage, and start work to split up this directory.

 

Which in aggregate means a move to faster storage hardware (SSD will completely obliterate HDD performance) or a move to a RAM-based pseudo disk, or related I/O changes, or application design changes, or a move to a platform that better fits the needs of these sorts of application designs.

 

I'd also run a check for hardware and network errors, given the transient nature of this report.   That probably isn't the case, but transient errors involving (for instance) heavy I/O activity can become unstable in the presence of lower-level and hardware errors.

 

Andy Bustamante
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

As noted, delete operations on a large directory can be expensive.  I supported an application with similiar issues.  Some of the options to mitigate this behavior, depending on your budget. 

  • RAMdisk.  configure your report files on a virtual disk.  A DEC-Ram license and memory can resolve all sorts of I/O issues.
  • Use a search logical and spread the I/O to multiple disks and directories.  On the fly updating of the disk sequence allows you to schedule delete operations on aged directories with minimal user impact
  • Same option but spread directories on a single disk.  Less cost, and less benefit, but keeps directory size small.  I used a batch job to shuffle directories every 10 minutes.  One site schedules daily deletes after midnight and kept writing report files to new directories.
  • Review file allocations and bump the cluster size to something that allows most of these files to fit within one allocation.  Faster create and delete time. 
  • If you have anything else generating I/O on the cache disk, move it.

The 2.x version of our web app skipped the temporary report file and and and just sent data to users.  More database overhead if they wanted to go back to an old report but better performance since the disk bottleneck was relieved. 

 

Don't expect tuning to have a significant impact.  Complete this step and get caught up but be ready to move on to the next phase.

 

 

If you don't have time to do it right, when will you have time to do it over? Reach me at first_name + "." + last_name at sysmanager net
Ana M. García Olivencia
Regular Advisor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

Thanks all.

 

I'll try to follow your recommendations.

 

I close this post.

 

Ana

Hein van den Heuvel
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

Hoff>>  There are no particular ACP parameters around altering the performance of what OpenVMS now considers large directories

 

I beg to differ. yes there is: ACP_MAXREAD 

That paramter defines the IO size in blocks which will be used during the directory shuffle.

The current default is 32

You could set to 64 for temporary releave, while working towards a real solution (cleaning out that directory).

(Set to 1 for the pre 7.2 behaviour :-)

 

fwiw,

Hein

 

 

Hoff
Honored Contributor

Re: INFORMATION ABOUT ACP SYSGEN PARAMETERS

I'll bet the reverse-delete hack still wins.