CPU Utilization plays Catch Up

[DWood85]

I have a program that is run in multiple instances (22 detached process) each that communicate via TCP/IP (sockets C calls) several times every 100 millisconds to a network device (each program communicates to a unique network device).
We are running Openvms 7.1 on an Alphaserver 4100 5/300.

The problem is that CPU utilization of each program doesn't accrue regularly. No CPU will be charged for minutes at a time and then suddenly one or more of the 22 instances (about every 5 minutes) will start getting charged large amounts of CPU (up to 30% for a minute at a time) time that seems to try and catch up the CPU utilization.
What's going on here?
I use scheduled wakeup at 100ms and Hibernate at end, but I tried other timing mechanisms (event flag waitfr, and lib$wait) but I still see the process in HIB a large amount of the time.
Any way to smooth the CPU use out?

[Hein van den Heuvel · Accepted solution]

This could be just an observation issue giving a "moiré pattern" with pronounced peaks and valleys instead of a uniform 'grey'.

The CPU charging takes place by the clock interupt routine, charging whichever process was active on a CPU, before checking what to run next.

Therefor, a task which is (in_directly timer scheduled and runs less than a clock tick might never be visible as far as CPU time is concerned.

Put a short task which takes 10% cpu right after a repeated longer tasks taking 95% of a clock tick and the short tasks may appear to consume 50% CPU with the long task none. "Bad timing".

What TCP/IP stack/version is being using? The older UCX versions had several high contention interlocks, serializing access and potentially magnifying the above suggestions.

Hardware assisted CPU cycle sampling, such as offered by the PRF extention to SDA can show what is really happening... but that is probably not available on OpenVMS 7.1

Is this question born out of curiosity, or is there a real problem to be addressed?

If there is a performance concern then it may be more fruitful to go the upgrade path first. Can you upgrage the Alpha and or OpenVMS and its TCP stack at all? The 5/300 only has a fraction of the power of a recent alpha chip. And OpenVMS has much improved in the performance area as well.

Hope this helps some,
Hein van den Heuvel
HvdH Performance Consulting.

[DWood85]

Thanks for responding,

System is using Digital TCP/IP services for OpenVMS Alpha Version V4.1, ECO 10.

This is causing some problems from time to time there seems to be an alignment of multiple programs catching up and there being nearly no cpu time availalbe. Timing constraints are retrieving data 10 times a second).
I wonder if there's a way to trick it so they don't ever get charged?

[Robert Gezelter]

DWood85,

As Jan comented, the accounting time aggregation mechanism was never designed to do accounting for time at this level of detail. The fact that all of the programs are running the precisely same processing will only reinforce the "moire" possibilities that Jan was referring to.

For instrumentation, I would consider putting in some mechanism that allows the processes to be monitored. If nobody is actually having problems, I would not recommend any action.

Of course, more detailed review may disclose some issue, but that is more detailed than can be handled in this forum.

- Bob

[Hoff]

At the risk of stating the obvious, why not have your application monitor itself?

I'm quite fond of instrumenting application code for debugging, performance and monitoring purposes, as external tools can and do have limitations. Out-board monitors never seem to have all of the detail I want, so I incorporate hooks or routines into the application code that out-board monitors and tools can then use.

Shared memory can be used to instrument the code, or otherwise.

These hooks might be selectively-enabled in certain cases (and particularly in the most performance-critical code paths), but I've found having debugging and monitoring tools "baked in" greatly simplifies support and upkeep.

And if you want to experiment with some potential "free" performance, try an upgrade to V7.3-2 or later, or an Alpha processor upgrade, or both. A number of folks around have found V7.3-2 faster than earlier releases, for various reasons.

It can be worthwhile optimizing old releases and old gear, but the costs do need to be compared with the costs of upgrading. And there can be diminishing returns here, too. And a cheap AlphaStation XP1000 or analogous will likely run your applications far faster than this AlphaServer 4100 box.

Stephen Hoffman
HoffmanLabs LLC

[Wim Van den Wyngaert]

If you have performance advisor, you can do sampling with :
adv coll sys /out=xxx/end=[time]/int=zzz
adv coll rep sys /out=yyy xxx

It will report the cpu per process based upon sampling every zzz 1/100 sec(default 1/100 sec and lower not possible).

Wim

[DWood85]

Thanks for the suggestions/info.
FYI I found that creating the process real time priority (20) and and using the prc$m_noacnt attribute when creating the process diminished the cpu utilization spikes significantly.

Also per HP, using a SYS$Hiber for this relatively quick (100ms scan cycle) timing was NOT a good idea at all. Changed the code to use a simple sys$setimr and sys$waitfr event flag, and all this made a significant difference in smoothing out the cpu oddities etc.

Done!

[DWood85]

closed