topic Serious Sys Mem Problem in Operating System - HP-UX

Serious Sys Mem Problem

Justin Willoughby — Thu, 29 Sep 2005 11:53:00 GMT

I have two L3000 boxes with MC/SG. One of the two has an issue. The boxes should be pretty close to being the same.

One of the boxes slowly has the sys memory go up until the point the system is unusable, e.g. there is no real or swap memory left for user processes.

The other box does not have this issue. On the box with the problem it takes several months for all the memory to get used up and then we have to reboot.

Right now glance shows that sys mem is 4.5GB, user mem is 207MB and phys mem is 5GB.

I have used the following command but it does not account for all the sys memory (or even close) that is in use.
UNIX95= ps -eo vsz,ruser,pid,args | sort -rn | head -50

I have attached a screen print from glance and also running ipcs.

I have tried using different command to see what is using all the system memory but I can't seem to figure out what is going on. Any ideas on how to track this down?

Thanks so much,

- Justin

Re: Serious Sys Mem Problem

Jeff Schussele — Thu, 29 Sep 2005 12:00:11 GMT

Hi Justin,

Can you say memory leak?
I knew you could.
The best cure I've seen for this malady is a Louisville Slugger liberally waved at the developers.
What you need to do is scan the PIDs for memory "growth" over time.
Short of that, you're not going to be able to do much about it.

Rgds,
Jeff

Re: Serious Sys Mem Problem

Devender Khatana — Thu, 29 Sep 2005 12:05:59 GMT

Hi,

The difference in load can be due to difference in packages running in both nodes. How many packages are there ?

Have you tried swaping packages in between them and see the utilization?

Is there something else running out of MC Service Guard?

Also "swapinfo -atm" will give memory utilization details.

HTH,
Devender

Re: Serious Sys Mem Problem

Justin Willoughby — Thu, 29 Sep 2005 12:07:51 GMT

No PIDs seem to be using any more memory then usual, at least from what I can see from using

UNIX95= ps -eo vsz,ruser,pid,args | sort -rn | head -50

I am assuming there is some sort of shared memory usage that is not being released but I don't know for sure or how to track it down if that is what is happening.

- Justin

Re: Serious Sys Mem Problem

Justin Willoughby — Thu, 29 Sep 2005 12:18:10 GMT

Devender,

There are only 2 packages running on this box and 1 package running on the box that's ok. The applications are very similar, one is for production and two are semi-production.

I have not swapped which servers they run on long enough to see if it's the packages or the node, but I might have to.

The swapinfo command does not really tell me much:

(ifback)root# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 1097 2999 27% 0 - 1 /dev/vg00/lvol2
reserve - 1036 -1036
memory 3881 3881 0 100%
total 7977 6014 1963 75% - 0 -

Re: Serious Sys Mem Problem

Jeff Schussele — Thu, 29 Sep 2005 12:23:42 GMT

Hi Justin,

The "virtual" (vsz) can be misleading.
Try using ps -efl & try sorting on the 10th column. That's the "resident" size which is the "truer" usage size.

HTH,
Jeff

Re: Serious Sys Mem Problem

Steven E. Protter — Thu, 29 Sep 2005 12:31:37 GMT

I'm with the bad application memory leak crowd here.

These issues are almost always caused by poor programming, though I advise against any kind of violence. :=)
Linking a good set of data collection tools.

http://www.hpux.ws/system.perf.sh

SEP

Re: Serious Sys Mem Problem

Jeff Schussele — Thu, 29 Sep 2005 12:45:53 GMT

Hey Steven,

Notice I said wave - not thump. ;~))
Shalom, my friend.

Cheers,
Jeff

Re: Serious Sys Mem Problem

Bill Hassell — Thu, 29 Sep 2005 13:36:07 GMT

From Glance, the memory is not user memory (a paltry 200megs) so ipcs and ps won't be useful. This looks like a horrible kernel module memory leak. Unfortunately, none of the normal Unix tools can map out what in the kernel space is consuming this RAM. I would strongly suspect any non-HP driver or subsystem such as SAN, commercial security and monitoring tools, anything that requires code to be added to the kernel. There is a slight chance that kernel patches are needed but without a kernel map, it will be difficult to see what's happening. Can you get a copy of kmeminfo from your sales/support rep?

Re: Serious Sys Mem Problem

Justin Willoughby — Thu, 29 Sep 2005 13:45:58 GMT

Hi Bill,

I was hoping to not open a case with HP (trying not to bother them) but sounds like I need to so I can get a copy of kmeminfo. I will do that shortly. Thanks,

- Justin

Re: Serious Sys Mem Problem

Patrick Wallek — Thu, 29 Sep 2005 13:49:29 GMT

Justin,

Never ever think twice about opening a support call with HP. If you have a support contract, that is what you pay for. You might as well get your moneys worth.

Re: Serious Sys Mem Problem

Bill Hassell — Thu, 29 Sep 2005 14:38:57 GMT

Since I was on the HP support team for several years answering calls like this, please do open a call. There are many sysadmins who do not have the luxury of emailing or calling HP for HP-UX support. And the problem you're seeing will likely take a senior engineer to resolve. It's pretty serious since growth is without bounds and your Glance memory listing shows significant paging (swap) which is severely impacting (slowing down) your applications.

Re: Serious Sys Mem Problem

Justin Willoughby — Tue, 11 Oct 2005 09:58:03 GMT

Well I had to reboot the box a week and a half ago because it ran out of memory and became unusable. I have started checking the memory since the box has been rebooted. It seems that when my backup run is when the system memory goes up. I have seen this happen with the TSM backup that we run nightly and our weekly Ignite backup. I have not seen a problem with memory going up the last few nights with TSM but I just did an Ignite backup and the system memory increased at the time the Ignite backup started by 60MB and never went down after the backup completed.

I have not placed a call to HP yet. I don't see any patches related to Ignite and memory leaks on the itrc. I am thinking I might just upgrade Ignite and see what happens. I am not sure how that could be related to the TSM nightly backup using system memory and not giving it back.

Very strange as the same version of Ignite is loaded on our other box and it's not having the same issue.

- Justin

Re: Serious Sys Mem Problem

CAS_2 — Tue, 11 Oct 2005 10:19:00 GMT

What HP-UX release is ?

Run

/usr/contrib/Q4/bin/kmeminfo

and post the output.

Q: I assume that the swapinfo output you posted is about box memory was exhausted, doesn't it?
Please, post the swapinfo output when box has no memory problem.

Re: Serious Sys Mem Problem

Justin Willoughby — Tue, 11 Oct 2005 10:26:08 GMT

CAS,

We are running HPUX 11.11 on rp4440s.

Attached is the out put from swapinfo and the kmeminfo command. There is no memory problem other then each day or at least every week more system memory is used up and not released.

- Justin

Re: Serious Sys Mem Problem

CAS_2 — Tue, 11 Oct 2005 10:41:18 GMT

I'd suggest to add a entry in the root's crontab in order to save the output of swapinfo and kmeminfo commands.

If backup might be the problem, I'd run the command

vmstat 2 count # substitue 'count' for the number of times the vmstat will print a line ENOUGH to monitor the memory for the time the backup is running. For example, if backup session last 1 hour, count should be 60 x 60 / 2 = 1800

The goal is to monitor the values of "avm", "free", "po" and "sr" columns.

Re: Serious Sys Mem Problem

Justin Willoughby — Tue, 11 Oct 2005 13:11:47 GMT

I'll run the vmstat during an Ignite backup tomorrow. What I just noticed as well is the ignite backup takes over 2 hours. On the other rp4440 in the cluster it takes about an hour to run. Very strange.... - Justin