memory, swap and mmd process

[Diane Thompson]

We have been experiencing Omniback backup errors and I am having difficulty try to rectify the problem. Some of what I see:
When we are having the Omniback issue swap memory used goes up, mmd is tops top process using up one entire CPU. We recently installed BMC Patrol Agent on this server but it is also installed on our development server and we have no problems there.

What should the ideal 'total swap PCT Used' be?

Regards,
D.Thompson

[Jeff Schussele]

Hi Diane,

The simple answer is - none.
The fact that you are using swap shows that either A) You don't have enough RAM in this server or B) It's not being allocated correctly.

Looking at that swapinfo output I'd speculate you have about 2-1/2 Gb - correct? Then looking at that top output I'd also speculate that you have not changed dbc_max_pct from it's default of 50% - correct?
If the latter is correct you definitely need to change it as well as dbc_min_pct to 16% & 10% respectively because you're paging & that's killing performance. The buffer cache is using *way* too much memory that the application obviously needs. If after you do that & you're still paging, then you probably need more RAM.

HTH,
Jeff

[Diane Thompson]

Yes, Memory=2.5 G
but dbc has been configured to:
dbc_max_pct = 20
dbc_min_pct = 5

Thanks,
Diane

[Jeff Schussele]

Hi (again) Diane,

Well my next step would be to run glance or the GUI gpm & take a look at just what's using all the memory & how. You may have a memory leak in something that's eating up all your free memory OR you may just simply need more RAM.
But when you swapping aka paging out you're going to see the vhand process become a top process & everything is going to slow way down because you are now using a disk drive for memory access & it's only...oh say....1000 times slower.
You need to determine whether you need more memory or to free some up.

Rgds,
Jeff

[Sridhar Bhaskarla]

Hi,

While the 'ideal swap PCT used' would be less than 100%, I seriously look at your memory utilization as you have around 550 MB sitting on the swap (KB Used in the dev lines of your swapinfo).

Run 'vmstat 2 20' and see if the 'po' value is constantly above 10. If so, your system is under memory presssure. Unless you have an application running on the box that has memory leaks, you will need to add more memory.

-Sri

[Diane Thompson]

Thanks Sri and Jeff for your responses.

I ran vmstat 2 20 a number of times and saw 1 appear twice.....otherwise ALL 0's. running swapinfo at the same time pct used was 70%-71%

Diane

[Sridhar Bhaskarla]

How about the swap utilization when Omniback is running. Would it be near 100%?. If so, then there is a good chance that the system is running out of swap space at that time. If swap is at 100%, system will not bring up more processes and fail with the messages such as "not enough space" or "insufficient memory".

-Sri

[Diane Thompson]

No Sri, we are not getting any kind of 'out of memory' messages. Omniback just completed a 3 hr backup process and I've got all the normal users on the system. When Omniback failes it has been unable to establish a connection with the mmd 'within 60 min' and aborts. All backups then fail until we recycle the processes. This is also when mmd is at 100% CPU. I added a cron job to stop BMC during the backups at HP's request but in looking at all of this the memory/swap stuff caught my eye and I started looking at it as well. I have been trying to convince my DBA that shared Oracle memory needs to be increased. Could that account for any of this? This is a L2000 running HPUX 11.0. We run one application (Lawson)and BMC Patrol Agent. We do not own glance.

Diane

[Ted Buis]

Patrol isn't that efficient. I have seen it consume over 10% of a system. I would turn Patrol off and see if your problem goes away. You can always use sar, iostat and vmstat. Patrol only uses the standard UNIX metrics anyway.

[Diane Thompson]

Patrol was not my choice.....It was chosen as a housewide solution. Currently Patrol is stopped, Omniback is running and a database export is running and I'm at 56% used memory (down from about 70%)

[Diane Thompson]

Just to give you an update... I have been capturing stats for about 24 hrs and finally saw some 'po' values greater than 0 in the vmstat output this morning.... 37 to 52.. down to 41, 25, 16 and tapering back down to 0 and swap was at 71% at the same time. But, all the rest of the values captured are 0.

From your responses it sounds like I additional memory is a given, but is there any kernel tuning that could help as well?

Thanks for all of your input....

Diane

[Diane Thompson]

PS- If we are running 70% utilization during peek hours and have 2.5 G of memory what is the best way to determine how much additional would be required?

Diane

[Ted Buis]

You have swapmemon=1 on your kernel. Your total swap space is what is on physical disk as well as a percentage of your RAM (somewhere between 50 to 85%). Since pseudo swap is turned on, some of your address space will be basically locked in RAM and will never physically be paged out to disk. This puts all the more pressure on the other pages in RAM if you run out of physical memory. In your case swap percentage is not that important, unless you get closer to 100%. If you get to 100%, then no new processes can start. You want to keep your swap percentage less than the ratio of RAM to the sum of Device swap plus RAM. What is important is the page out rate. Any double digit numbers over time indicate a significant issue. However, I think this is a symptom of the real problem. Try to identify the process that is using all that RAM. OB should be able to run in a 2.5GB of RAM system. If you don't know how to find RAM consumed by a process, do a search as this subject has been address on the forum before. I would also try checking for patches for either HP-UX or OB that might relate to a memory leak.

[Patrick Wallek]

When you start seeing Page Outs in vmstat, run 'swapinfo -tam' and see how much of your swap areas are actually being used.

If any of the 'dev' lines in the swapinfo output are greater than 0, then you need to add at least that much more RAM.

What type of system is this and what version of HP-UX are you running?

If you are on a reasonably new system and running 11.11, then I would likely get as much RAM as you can. If you don't like HP's prices, check out Kingston or Crucial. They have very good prices and I have never had a problem with their RAM.