Application Hangs in HP-UX B.11.11

[iammitan]

Hi,

We are running a multithreaded application in HP-UX B.11.11 box. This application is initialy developed on linux box and ported to HP UNIX. also this application is complied using gcc 3.4.2 version. When the size of process increases to 1 gb(using top command) the application hangs and does not process any data. The increase in the memory could be a memory leak. As a temporary solution we had increase the values for maxdsize and maxtsiz parmeters. Can anybody tell me why my happlication hangs in this senario.

Thanks in advance.

[Don Morris_1]

Well, no one can know for _sure_ without tracing the application [which is the job of your developers and all]... my assumption would be that you've compiled as 32-bit, using the default address space layout (1Gb for data+private dynamic objects+stack) and you've exhausted your private address space... and the application handles ENOMEM failures by either sleeping or just dropping work. Raising kernel tunables won't help any -- you can't grow beyond the quadrant boundaries for a 32-bit application.

Another possibility is that your application is consuming all the physical memory on the box -- and the "hang" you're perceiving is because vhand is paging it out or deactivating it to free up memory. It doesn't sound like you think the whole box has gone non-responsive... but you never know. You don't say how much RAM the box has and all.

The first thing I'd do is see if you *do* have a memory leak. Anything that simply gives the application more room to work would just be eaten by the leak if this is the case. (There are various tools to do this that your developers should be aware of already).

If your developers are adamant that the application just takes X Gb (where X is obviously > 1) to do the workload you're demanding of it, then you have a few options:

1) If the hardware and 11.11 installation support it ("getconf _SC_KERNEL_BITS" returns 64), you could recompile your application as 64-bit. This will give you terabytes to work in, making your only real limits maxdsiz_64bit [so check that value and set it appropriately... usually it's pretty reasonabe already] and your swap space. Since HP-UX (unlike Linux) reserves swap whenever virtual address space is grown, you *must* have sufficient swap to cover your application's memory needs [plus the rest of the system]. This can be mitigated by use of "pseudo-swap" -- which is a Kernel accounting trick that just considers memory to swap to itself, allowing the total virtual address space to be larger than device/FS swap. Check "swapinfo -atm" for your Total swap.

2) If you must stay 32-bit for whatever reason, consider changing your compilation model to EXEC_MAGIC, allowing your Text and Data to use the same 2Gb.. assuming your application doesn't have 1Gb of code + static data, that will give additional space for dynamic data.

3) Move to 11.23 on IPF and use MPAS (which is the closest to Linux you can get on HP-UX).

[Steven E. Protter]

Shalom,

The number one cause of application hang is bad code.

Do make sure your box is properly patched, this mimimizes the chance of the OS doing it.

I would suggest hanging a tusc trace on the process(s) that are causing trouble to track down what actually happens right before the hang.

SEP

[Dennis Handly]

>Steven: The number one cause of application hang is bad code.

Right, it could be a mutex deadlock.
Also, is it really a hang or a loop?

>I would suggest hanging a tusc trace

And getting stack traces on all threads:
(gdb) thread apply all bt