Memory Problem (Leak?) in Remedy Action Request System

[Grant Frost]

We have had an ongoing problem for many months now with Remedy Action Request System 4.5.2 running on HP-UX 11.

Remedy has been unable to help us with this.

Our Remedy System Server process (arserverd) gradually uses more and more memory until it reaches around 1.7-1.75GB and then the process dies and Remedy monitor restarts it.

As a result, our Remedy application crashes and restarts every 2-3 days whenever the process reaches this memory threshhold.

Any recommendations or suggestions as to what we could focus on would be greatly appreciated.

[Jose Mosquera]

Hi,

Could you provide us an output file with your kernel configuration?

Rgds

[Bill Hassell]

Without the Remedy source code and the authors, there's not much you can do. The program(s) are growing and a really common programming error is to ask for RAM but never return it until the program stops running. It's possible the solution is due to library patches (lack of) on HP-UX but this is just guesswork. I would be surprised that Remedy support does not have the fix. Are you the only Remedy customer that's every seen this problem? The 1.75Gb limit is a 32bit program limitation, both for shared memory and program data areas. There's nothing you can do in HP-UX to fix the broken program(s).

[Jeff Schussele]

Hi Grant,

Please, PLEASE forgive me but as a Remedy user myself, I just *have* to ask the obvious.....

Have you placed a Remedy call to your help desk on this?
What's your ticket #, Sev level?

All kidding aside, as BH has told you, the absolute only thing you can do in this situation is to determine just where it bleeds. And you'd need to use tools such as lsof, adb, tusc, kitrace...not to mention time, blood, sweat, etc.....
And the bottom line is only they can fix it - it's their code. The license your mgmnt signed not only protects them from suit if the code harms you, it bars your right of self-remedy (here we go again....somebody stop me) so no touching the code.

He's also dead-right that the vendor at least knows something about your problem & the 1.75 Gb size is a huge clue.

So my advice to you is that until they fix it (read honoring their maint contract) my advice to you would be to schedule regular stop & clean sessions. Script it, cron it - knock yourself out - but it ain't gonna fix itself.

In the meantime I'll check with the SA here who has the Remedy systems, to see if they know anything about this particular symptom & try to get back to you Fri.

Rgds,
Jeff

[Grant Frost]

Thanks for everyone's comments so far. It is greatly appreciated. Jose: I will post our kernel configuration tomorrow.

Grant

[Grant Frost]

In reply to Bill Hassell's post:

Bill, Remedy has basically told us this: That the memory growth problem is a limitation of HP-UX and they claim that Remedy is operating normally.

They claim that whenever our system requests X amount of RAM, that if there is no single contiguous block of previously requested RAM (which is now free) available, that it cannot use any of the existing RAM (despite how much RAM there is if it is fragmented), but that it needs a complete continguous block, so they are claiming that if there is not a contiguous block of existing RAM available that their application just creates another malloc (with the corresponding free) and hence the memory size for their app continues to grow.

They claim that this is a limitation of HP-UX and they leave the memory management architecture up to the OS.

Personally, I think that they should build some sort of "garbage-collection" routine into their application which manages the ram that they allocate to the process which 'defragments, collapses, and frees any unused RAM' to give it back to the OS. Is this possible in HP?

Remedy is telling us that so long as our application continues to make requests for large blocks of data (greater than, let's say, 500K or so), that RAM utilization will continue to increase.

Personally, I think that there is something wrong with their application on this platform. I find it very very hard to believe that their application cannot find a few megs of contiguous space in the blocks of RAM that is already allocated to the process.

We have a trouble-ticket escalated with Remedy and we are working on the problem with their senior engineers and programmers but we are getting nowhere. To the best of my knowledge we have all the correct HP patches applied. From what I can tell, Remedy is having us look at symptoms of the problem (operations or functions which cause memory to grow) rather that trying to identify the root cause.

Remedy has asked that we try to put them in touch with an HP systems engineer who might be able to work with them and assist them with this problem. They claim that it's impossible to release RAM back to the HP-UX OS but they said if there is a way to do that, they would like to know how. Is there anything that you could do to help me put an HP person in contact with our Remedy contacts?

Any help or assistance with this would be very much appreciated.

[Grant Frost]

Thanks for your comments Jeff. If your Remedy resource knows anything that could help us I would really appreciate his guidance. I can provide you with any level of detail of the problem that you need. Just let me know if you would like more granular or more specific information on the problem.

We have had a Remedy engineer (from Remedy) come onsite to help us, and we are currently looking at hiring a team of very senior Remedy engineers from another company to come onsite to see if they can offer any additional assistance. So far, we have not gotten very far with Remedy.

Remedy has provided us with a series of debuggable binaries to replace the regular one that we use to try and debug the problem, but it has only shown that all the mallocs, callocs and reallocs all have corresponding 'frees'. And yet the memory growth still continues...

One of my colleagues has suggested that it might actually be a problem with our remote Oracle database connections. He suggested that Oracle can hang onto the memory after it is finished if there is a problem with the Oracle configuration so right now we are focussing on that. He said that he has seen scenarios in HP-UX where if the Oracle listener is restarted, that the RAM usage by arserverd (Remedy AR Server Daemon) drops right back down to its normal startup value. I am going to test this hypothesis over the next few days to see if it works. I will post the results here when I get them.

Unfortunately, I am not a UNIX SA, so some of the tools that you have mentioned I am unfamiliar with. I have used tusc before but in a very limited way.

If there are any tools or utilities that you could suggest might be helpful, please let me know.

Thanks again for your continued help on this. I never expected to get such timely responses from anyone and I really appreciate the support :)

[Grant Frost]

Jose,

As requested, here are our kernel parameters. Please let me know if there is any other information that you would like.

Thanks!!

Grant Frost

-----------

Name Current Value

NSTREVENT 50
NSTRPUSH 16
NSTRSCHED 0
STRCTLSZ 1024
STRMSGSZ 65535
acctresume 4
acctsuspend 2
aio_listio_max 256
aio_max_ops 2048
aio_physmem_pct 10
aio_prio_delta_max 20
allocate_fs_swapmap 0
alwaysdump 0
bufcache_hash_locks 128
bufpages 0
chanq_hash_locks 256
create_fastlinks 0
dbc_max_pct 15
dbc_min_pct 5
default_disk_ir 0
disksort_seconds 0
dnlc_hash_locks 64
dontdump 0
dskless_node 0
dst 1
eqmemsize 15
fcp_large_config 0
fs_async 0
ftable_hash_locks 64
gab_flowctrl 5
gab_logbufsize 48100
gab_logflag 29
gab_max_cpu 128
gab_numnids 32
gab_numports 32
hdlpreg_hash_locks 128
hfs_max_ra_blocks 8
hfs_ra_per_disk 64
initmodmax 50
io_ports_hash_locks 64
ksi_alloc_max 19360
ksi_send_max 32
llt_fastalign 1
llt_maxnids 32
llt_maxports 32
llt_nominpad 0
max_async_ports 50
max_fcp_reqs 512
max_mem_window 0
max_thread_proc 64
maxdsiz 2063835136
maxdsiz_64bit 4390046511103
maxfiles 2048
maxfiles_lim 2048
maxssiz 401604608
maxssiz_64bit 1073741824
maxswapchunks 5120
maxtsiz 1073741824
maxtsiz_64bit 4390046511103
maxuprc 400
maxusers 300
maxvgs 10
mesg 1
modstrmax 500
msgmap 42
msgmax 8192
msgmnb 16384
msgmni 50
msgseg 2048
msgssz 8
msgtql 40
nbuf 0
ncallout 2436
ncdnode 150
nclist 4900
ncsize 7024
ndilbuffers 30
nfile 10000
nflocks 1000
ninode 6000
nkthread 4251
no_lvm_disks 0
nproc 2420
npty 300
nstrpty 60
nstrtel 60
nswapdev 10
nswapfs 10
num_tachyon_adapters 2
o_sync_is_o_dsync 0
page_text_to_local 0
pfdat_hash_locks 128
public_shlibs 1
region_hash_locks 128
remote_nfs_swap 0
rtsched_numpri 32
scroll_lines 100
scsi_max_qdepth 8
scsi_maxphys 1048576
sema 1
semaem 16384
semmap 202
semmni 200
semmns 300
semmnu 30
semmsl_override 2048
semume 10
semvmx 32767
sendfile_max 0
shmem 1
shmmax 1073741824
shmmni 200
shmseg 120
st_ats_enabled 1
st_fail_overruns 0
st_large_recs 0
streampipes 0
swapmem_on 1
swchunk 2048
sysv_hash_locks 128
tcphashsz 0
timeslice 10
timezone 420
unlockable_mem 0
vnode_cd_hash_locks 128
vnode_hash_locks 128
vps_ceiling 16
vps_chatr_ceiling 65536
vps_pagesize 4
vx_maxlink 32767
vx_ncsize 1024
vx_ninode 0
vx_noifree 0
vxfs_max_ra_kbytes 1024
vxfs_ra_per_disk 1024

[Bill Hassell]

Aha, shared memory fragmentation...very, very common problem for 32bit programs. Yes, Remedy is correct that if all the programs running in the system that are using shared memory are stopped and restarted or the segments are reserved and then returned, then the shared memory will of course be fragmented. After all, 32bit programs are constrained to a single shared memory area consisting of one quadrant (about 960 megs) or 2 quadrants (about 1750 megs) for MAGIC executables. And the shared memory area also has memory mapped files and other structures in there (see shminfo below).

The fix is simple: have Remedy get with the program and compile 64bit versions of their programs. Poof! no more fragmentation issues. In a way, Remedy is causing the problem by asking for 500 megs, and when it fails, asking for smaller chunks until they get it.

Remedy engineers need to study two white papers on your system:

/usr/share/doc/mem_mgt.txt
/usr/share/doc/proc_mgt.txt

(or print the Postscript versions)

Then, immediately get the shminfo program so a complete map of shared memory can be seen:

ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

Finally, the solution is easy: Remedy needs their own private shared memory area so no other programs can fragment the common area. This is where memory windows will be the solution (if 64bit Remedy is not an option). Get the necessary patches, then ask Remedy which processes will use the shared memory area together. Now, all of these processes will be started with a wrapper program that assigns the programs a private shared memory area. All other non-Remedy programs will continue to use the default or common shared memory area.

[Grant Frost]

Bill,

Thanks very much for this excellent information. I have passed it along to Remedy. I'll update this thread as soon as I have some news.

Jose & Jeff...I'm hoping that you have other insight that you might be able to provide.

Do you have any feedback on our kernel parameters? Anything out-of-the-ordinary there?

Thanks!

Grant

[Jeff Schussele]

Hi Grant,

I'm sorry I haven't got back to you yet. I've been unable to speak w/the SA of the Remedy systems as I've been battling a perf issue in the test labs all day.
I'll try to get info for you Monday.

Sorry,
Jeff

[Grant Frost]

Thanks very much Jeff, I look forwards to hearing back from you.

Grant

[Grant Frost]

Hi all,

Our engineering contact at Remedy has clarified the situation:

"Our application simply issues malloc calls based on requirements of the workload dictated by the user
of our application. If a user requests X amount of data, we will try our best to facilitate the request, and subsequently malloc the space in order to temporarily store the data as it is pooled
before being sent the originating client. We simply request the memory space by performing a malloc(). The way the OS implements malloc() is as follows:

1. If an application requests X amount of bytes of memory, the in-process memory manager will attempt to satisfy the memory requirement based on in-process, free/available memory.

2. If this is not possible, for whatever reason (e.g. fragmentation, no available mem, corrupted mem), the in-process memory manager requests for more 'core' memory from the kernel via the 'brk' command.

The results of the 'brk' command is what makes your process size grow from a utilities perspective. 3. If the OS can't satisfy the 'brk' call (e.g. no available core memory, process limit) a NULL is returned an errno is set to ENOMEM.

Once a huge block of memory is returned to a process, these blocks cannot be turned back to the OS for redistribution until the process exits, or 'brk' is called with a negative number (see man page for brk).

Negative brk calls are not recommended by the OS vendor. We are not claiming this behavior is necessarily a bug on the UNIX vendor's part ... it's simply how it works. I have a very small C program that can illustrate this behavior if you'd like to see it."

Any thoughts or comments on this would be greatly appreciated.

[Gert Andreasen]

Hi Grant and others,

we have seen the same memory leak problem with a Remedy vers. 4.5.2. P1167 on HP-UX 11.11, so I wonder, how You managed to come around with this problem:

1) Did You apply HP patches ?
2) Did You apply RMDY patches ?
3) reconfigured HP-UX ?
4) Upgraded Remedy to v. 5.x.x. ?

or other solutions, or do you still have
the problem?

Gert Andreasen

[Arthur Huber]

We have the same problem with oracle 9.2 64-bit database and 32-bit client. Is there a patch in HP11 fixing this memory leak?
Thanks
Arthur Huber