memory hit 100% and no swapping occurred

[Robert Dukes]

Hi,

I am running an rp7410 with 8Gb Memory and HPUX 11i. The server ran out of memory this morning, swapinfo was showing memory used as 100%, now Ive got 2 x 4Gb swap areas set up but were never used (showed used 0%) when the system reached 100% memory utilitsation. We were not able to run hardly any commands because there was no memory. Can anyone explain why the swap areas were not used, and the system fell into a heap and had to be reloaded.

Thanks in advance for any suggestions.

[Jairo Campana]

From the looks of it you've a really loaded system, your memory is nearly exhausted. And you're CPU is heavily loaded.

What are you running on this server?
swapinfo -tam
vmstat 1 30
Are we paging.
More advanced perf data collection attached.
If you are paging into swap, its memory buy time.
The good news is you don't seem to be waiting on disk io. That may be because there is too much stuff running on the box to have time to do a lot of disk io

For paging:

vmstat -S 5 20

Look at SI and SO columns...

Rule of thumb for performance - Check Memory first, then disk, then CPU...

Your disks and CPU may be loaded because of memory pressure - fix the memory - and your CPU and disks may be fine.

Decrease the size of your buffer cache may help - what is it set to? DBC_MAX_PCT and DBC_MIN_PCT...

More memory may be your solution...may be

[Robert Dukes]

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol9
reserve - 1969 -1969
memory 6324 2415 3909 38%
total 14516 4384 10132 30% - 0 -

vmstat 1 30
------------

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 329983 887960 93 11 64 0 0 0 51 2093 12708 3717 10 3 88
1 1 0 329983 886849 241 17 149 0 0 0 0 1020 15631 1936 29 1 70
5 0 0 330584 886617 193 13 161 0 0 0 0 1128 15991 2085 30 4 66
5 0 0 330584 886385 154 10 171 0 0 0 0 1185 16759 2218 27 1 72
5 0 0 330584 886165 123 8 176 0 0 0 0 1249 16416 2236 27 2 71
5 0 0 330584 892188 109 8 141 0 0 0 0 1267 16435 2188 2 2 96
5 0 0 330584 890458 338 26 124 0 0 0 0 1220 15208 1926 14 3 83
5 0 0 330584 886910 275 20 136 0 0 0 0 1147 15807 1980 33 1 66
5 0 0 330584 886680 220 16 149 0 0 0 0 1109 15536 1949 24 3 73
5 0 0 330584 886448 176 12 165 0 0 0 0 1079 15357 1922 25 1 74

vmstat -S 5 20
--------------

procs memory page faults cpu
r b w avm free si so pi po fr de sr in sy cs us sy id
2 0 0 327674 886731 0 0 64 0 0 0 51 2090 12708 3709 10 3 88
2 0 0 327674 885337 0 0 242 0 0 0 0 1063 10469 1389 25 1 74
2 0 0 328168 886664 0 0 196 0 0 0 0 1125 10844 1450 19 2 79
2 0 0 328168 885194 0 0 238 0 0 0 0 1050 9986 1358 25 1 74
3 1 0 345618 886565 0 0 179 0 0 0 0 1158 11766 1603 20 2 78
3 1 0 345618 885355 0 0 221 0 0 0 0 1095 13138 1776 23 2 76

--------

dbc_max_pct 50
dbc_min_pct 5

I have pulled the above info off the machine. It is not running to bad at the moment, but the memory is creeping up.

[Jairo Campana]

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=743917

[Jairo Campana]

sorry too:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=605128

[Patrick Wallek]

The first thing you've GOT to do is reduce that dbc_max_pct value. 50 is WAY too high. You should set it to whatever percentage of memory would make approx. 400 MB. If you have 2GB of RAM, set dbc_max_pct to 20.

The buffer cache is SUPPOSED to be reduced automatically, down to dbc_min_pct, if the machine is under memory pressure. This doesn't always work well though.

Try this first and see if that helps. Note that that will require the kernel to be rebuilt and the machine rebooted.

[hpuxrox]

for HP-UX 11.11 and lower check the value of MAXSWAPCHUNKS. If this is set to low, you will not use your all your swap.

Thanks,

Yates

[Victor BERRIDGE]

Hi,
Is swapmem_on set at 1?


All the best
Victor

[Sridhar Bhaskarla]

Hi Robert,

Not having memory doesn't stop you from running any commands. It will just slow things down.. may be terribly if it too bad..

But not having 'enough swap' will stop the system from bringing more processes. Everytime a process starts, system will allot some space for it in the swap area and if there is not enough swap space, then the process will fail to come up.

Swap areas are *physically* used only when there is memory pressure. Otherwise they are used just for reservations.

Because HP-UX works in virtual memory realm you can always bring up processes than your actual physical memory can allow. Looks like some processes were started for which system tried to reserve swap space making it to reach 100%. You would need to find those processes. If this is reoccuring again and again, then I suggest you put a cronjob that runs every 5 mins or so that will capture the processes and the memory used by them.

UNIX95= ps -e -o 'vsz pid args'|sort -n > /tmp/memory/$(date +%Y%m%d%H%M)

Keep looking at these files to determine the rogue processes. You may find some with huge vsz values when this occurs.

-Sri

[Robert Dukes]

Hi,

Ive had the same problem again, absolutely no swap space is being used. Here is the result of a swapinfo. I really dont understand why it is not swapping...

# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol9
reserve - 8388608 -8388608
memory 6475292 6475176 116 100%

Any suggestions?

Cheers, Rob.

[Sridhar Bhaskarla · Accepted solution]

Hi Robert,

I think we need to prepare a balance sheet here to explain it better.

CREDITS: The total swap space is determined by the device swap, filesystem swap and the swap allocated from memory (pseudoswap) with the swapmem_on kernel parameter. Pseudoswap is only for calculation purposes not that it will be really used anytime to hold the paged out pages. 'swapinfo -t' gives you the total swap available.

DEBITS: As the processes come up, system will 'reserve' space for them in swap area. You can see it in 'reserve' line. Also memory locked by the processes, dynamically alloted by the kernel etc., will be deducted from the reservation and hence is shown as 'memory used' in the swapinfo line. If the system is used, then there is a good chance that 100% of the 75% of memory initially added for swap may not be available anymore as in your case. Another debit is to keep the actually paged out pages (which you are not seeing). It could be due to memory pressure or memory mapped files.

So, it is not necessary to see 'swap' utilization at 100% though there is no paging going on.

Post your above 'UNIX95' output. I suspect that there are processes with memory leaks that are simply grabbing memory but not releasing it.

-Sri

[D Block 2]

can you tell us what the kernel memory usage is compared to user process memory used ?

glance:
Sys Mem: ?
Buf Cache: ?
User Mem: ?
Free Mem:

# ps -el

what are the processes consuming memory ?

If the user procsses are not the issue, then most likely it's the kernel either (a) filling up the buffer cache (are there many files opened?), or (b) you have a memory leak in the kernel.

maybe it would be good to check your Patch List for potential memory leaks in the kernel.

--
also, I've seen before, that an external events (like your NIC card), can cause excessive packets incoming to exhaust your memory.

run glance, and look at the network statistics for posible ERRORs.

best of luck

[Robert Dukes]

Sri,

Thanks for your advice on this, sorry if I sound a little naive about this area, but Ive never seen a problem like this before. I think I understand it a little more. Here is output from swapinfo -t:-

# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol9
reserve - 8388608 -8388608
memory 6475292 6471400 3892 100%
total 14863900 14860008 3892 100% - 0 -

So you are saying that the TOTAL line at the bottom, is the total amount of real memory and device swap. Is this showing that all REAL memory and Swap may not be 100% utilised but is at least 100% reserved?

We are seeing here that 0% of swap is used, therefore Im assuming that 100% is reserved?

Many thanks for your input on this, any info is greatly appreciated.

[Sridhar Bhaskarla]

Hi Robert,

//We are seeing here that 0% of swap is used, therefore Im assuming that 100% is res
erved?//

You nailed it.

However, I wouldn't call the memory line as 'real memory'. When the system starts up, it will allot 75% of the memory as swap. As the memory is used for locking the processes or dynamically alloted by kernel, that will no longer be available for swap and hence reservation. As you can see 6475292 KB of memory initially alloted for swap is not available anymore. So, for the reservation of 8388608 KB, system will have to depend on device swap entirely hence you are seeing it 100%.

As said by Tom, post your 'memory' break-up as seen in glance's 'm' window. Also post your "UNIX95= ps -e -o 'vsz pid args' |sort -n".

As a temporary fix, you can add little more swap space but you will have to look at the root cause and you may find it using 'glance' and the above command.

-Sri

[Jeff Schussele]

Hi Rob,

Yes, that's what Sri is saying.
You're kicking off over 14GB of processes and completely exhausting your reserve space.
What you need to do now is either:
A)add another swap device
B)reduce the amount and/or size of the processes.
Are you running a lot of Java? You can start Java procs with less memory demand...usually w/o impacting their performance.

Rgds,
Jeff

[Bill Hassell]

And as mentioned, please post the setting for swapmem_on as it will dictate the minimum amount of swap space you need before any true swapping takes place. With such a large machine, you are crippled without having Glance to show you what is going on. Load the trial version from your Application CDs as soon as possible.

[Bill Hassell]

And just a note about swapmem_on. If it is 0, then the first 8Gb of swap space will never be used. You need more than RAM space in swap in order to move anything out to disk. Add another 4-8 Gb of swap space using swapon (temporary until reboot) and you'll probably see everything working again.

[Robert Dukes]

This is the current setting for swapmem_on:-

# kmtune -q swapmem_on
Parameter Current Dyn Planned Module Version
===============================================================================
swapmem_on 1 - 1

[Victor BERRIDGE]

Hi,
I would add 2 more swap devices of 4GB but this time on external disks...
Then I would change swapmem_on to 0 AND reduce:
dbc_max_pct 6
dbc_min_pct 3

And recompile the kernel and hope this will be enough

OR

Wait for Bill's advice....

All the best

Victor