RLIMIT_AS vs. VSS

[jmgallag]

Hi,

We use a 64 bit COTS product on HP-UX 11.31. On occasion, we have found bugs where the vendor's code gets stuck in a loop mmap()'ing memory. Since this is 64 bit, a single process can easily exhaust virtual memory on the server. Because the problem was with mmap()'ed memory, the regular process limits in the kernel (maxssiz_64bit, maxdsiz_64bit and maxtsiz_64bit) do not apply.

In order to limit the problem process, I built a shared library that calls setrlimit(). The shared lib is loaded via LD_PRELOAD and sets RLIMIT_AS. RLIMIT_AS was the only limit I found which could limit mmap()'ed memory.

I have discovered that the RLIMIT_AS set via setrlimit() can be hit, even though the "Total VSS" as reported by glance is substantially lower. For example, I picked 3GB as my RLIMIT_AS. But a VSS of close to 1GB can cause the process to run into my limit and crash.

I am looking for an explanation of how the RLIMIT_AS limit is implemented and how to reconcile the difference between that limit and what glance reports as VSS. I think they should be the same, but that is not the case.

Thanks,
Jim Gallagher

[Dennis Handly]

>I have discovered that the RLIMIT_AS set via setrlimit() can be hit, even though the "Total VSS" as reported by glance is substantially lower.

Perhaps RLIMIT_AS also includes mmaps for shared files or text?

If shared, you or glance would think there is 0 cost.

You can write a program to sum up the VM usage of your process by calling pstat_getprocvm(2).