Can three looping processes, zero priority, affect performance significantly?

[Clark Powell]

We have a 4 cpu, ES45 running Cache and GE Flowcast web framework. This app is a hybrid web app that uses a browser user interface and telnet to gather data from Cache. Occasionally, it will terminate the telnet process incorrectly and create a looping LOGINOUT.EXE process at zero priority and with the "no delete" bit set. We have to reboot to get rid of them but, since OpenVMS recognizes the futility of these processes, it sets the priority to zero, they don't do any IO, and they don't interfere much. At least that what I think but now we are having a debate on this topic and I would like to poll you on this issue.

We have 500 interactive users and three of these looping, zero priority processes. My position is that these processes are not going to cause a problem until there are 10 or 20 of them. At that point the book keeping or balancing done at a much higher priority would begin to have an affect. The opposing view says that there are 4 processes in the compute queue and that's too high and higher than normal. I say, there are three looping processes that will always be waiting to be in the compute queue so that number is not so high. Of course, nobody has asked the users if they are experiencing a slowdown...

What do you all think, who's correct? or let's say, more correct?

thanks
Clark Powell

[Hoff · Accepted solution]

Processes looping at zero are inconsequential, unless you're right at your process limit, or unless the applications are accessing or holding some resource that causes interference with other system activity, or if you're running other processes at zero.

Something similar to the bug you're describing here was fairly common telnet bug back around TCP/IP Services V5.0, but that was fixed in an ECO and in more recent releases.

Clearing the PCB$M_NODELET bit and fully deleting these processes can be performed with some simple kernel-mode code, if that's permissible to load and run here.

[Jon Pinkley]

Clark,

Software priority zero still has higher precedence than the idle loop. One thing the idle loop does is zero pages for the zeroed page list, which used to satisfy demand zero page faults. If the idle loop never gets time, then demand zero page faults will take longer, since the pages must be zeroed before they are given to the process requesting them.

I don't know how much you would notice that.

What will notice are other processes running at priority zero.

What is setting PCB$V_NODELETE? I thought this was normally only set for special processes like the swapper, NETACP and NET$ACP (OpenVMS Alpha Internals, Scheduling and Process Control V7.0, bottom of page 419)

RE: "since OpenVMS recognizes the futility of these processes, it sets the priority to zero". Where did this information come from?

Priority zero isn't a "normal" priority that I would expect VMS to set. Are you sure you don't have something that is doing this lowering of priority?

Jon

[Robert Gezelter]

Clark,

I concur with Hoff wholeheartedly.

The wonders of statistics. Three of the four items in the compute queue are the "almost dead" priority zero processes.

"Items in the COM state" as a metric is a crude bludgeon. It does not take into account processes that are CPU "black holes" (e.g., SETI, or a background job whose purpose is to soak up otherwise wasted CPU cycles doing some other socially useful task). Such jobs often use virtually (pardon the pun) no resources.

If one wants a metric that is more useful, one can gather the data oneself and calculate the effective Computable queue (excluding the CPU soak tasks and zombies).

When using metrics to measure performance, it is always important to remember the limitations of the data.

- Bob Gezelter, http://www.rlgsc.com

[Clark Powell]

We are running tcpip V5.6 - ECO 3 and we had not experienced any problems terminating telnet in any manner for a number of years until we installed the GE web framework. The product is, as I said, a hybrid of web client and telnet so it tends to do things differently resulting in this problem.

What program will un-set the no delete bit and what is the risk of using it? We have to be 100% confident in the program's safety before we can use it at any time of day. We are a hospital and cannot tolerate any interuption in our patient care computing. Since this is a clustered system, we can reboot without interrupting service and don't mind the extra work of rebooting if it means a near zero chance of a service interuption.

thanks
Clark Powell

[Clark Powell]

Reguarding the Zero Priority

In the past, when a similar incident would happen we would set the priority to zero. Then during this latest loop problem I would happen on the processes and find that they were already zero priority. I assumed that my colleague had done this but later when I mentioned it he said no. Then discussing this problem with Colorado Springs (or whatever it is now,) they said that there was a feature in OpenVMS that would identify these processes and lower the priority automatically. I don't know if this is true but it make sense.

Reguarding the Impact
I haven't pulled out the T4 data yet but I have an impression that as time goes by, with a fixed number of looping processes, the proportion of Kernal mode processing might go up in these situations. Or it could be that each additional process creates a disproportionate increase in Kernal mode processes. Kernal mode runs 37 to 76 out of a max of 400.

[Hoff]

1: This is a kernel bug in either the GE software, or in TCP/IP Services.

Accordingly, call your support organization. Probably whomever is supporting the GE software here, though this might be transferred along to the TCP/IP Services folks if the core bug is in that and not in the GE software.

2: Program? I usually create a slightly customized version of something akin to this tool:

http://labs.hoffmanlabs.com/node/767

You'd need a little more code here than what is in that example, as you'd have to scan the processes and then jump to kernel mode to clear the bit. Scan for the processes via $process_scan or $getjpi or such, jump to kernel, set up a kernel-mode signal handler, convert the PID to the PCB, clear the NODELET bit, and off you go. Typically a few hours to design, code, test and document this tool.

Might well be feasible to script this via delta or xdelta, but there's less error-handling available when using that approach.

[Hoff]

FWIW...

Kernel looping at priority zero is still typically inconsequential.

Yes, it'll look nasty in the process mode displays.

Yes, it'll look nasty in any COM queue displays.

Yes, it'll look nasty in the CPU time displays.

Yes, it'll consume a little memory.

In reality, you've usurped what used to be known as the NULL process. OpenVMS itself loops like this while it's waiting for work, too; OpenVMS has an idle loop.

These looping processes will remain inconsequential until and unless you've other tasks at zero or you fill up available process slots or if there is resource contention or there are too many BG devices or other such; the other sorts of contention discussed earlier.

Oh, and if you're running OpenVMS under emulation (and which you are not doing here), these loops can confuse idle loop detection during hardware emulation.

Go talk to your support folks for the GE software, and get the process leak fixed.

And strictly for giggles, toss a SET PROCESS /SUSPEND at these processes, and see if that locks them down.

Or pay for somebody to write a NODELET tool for this for you, and skip the reboots.

Stephen Hoffman
HoffmanLabs LLC

[EdgarZamora]

Are you current with your patches... look into VMS83A_DCL-V0300, it might fix your problem with these processes not getting deleted.

[John McL]

Less a reply than a question to respondents here... will these priority 0 jobs run to completion of their quantum or will they be interrupted by processes at higher priority?

And Clark you might like to check for any resources that they are holding. If some resources are held there may be impact on the system but that's only a "may". If they hold 1 of 10,000 items it would be no big deal but if they hold 1 of 10 (or less) getting rid of those processes becomes more urgent.