server Has Excessive Memory Usage

[praveen..]

hi,
my server Has Excessive Memory Usage,
how to solve this problem?

#top
System: dipper Thu Mar 30 22:34:53 2006
Load averages: 0.01, 0.01, 0.02
593 processes: 553 sleeping, 39 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.2% 0.0% 0.8% 99.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
2 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
3 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
4 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
5 0.01 1.4% 0.0% 1.0% 97.6% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.2% 0.0% 0.2% 99.6% 0.0% 0.0% 0.0% 0.0%
7 0.01 0.2% 0.0% 0.0% 99.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.01 0.2% 0.0% 0.2% 99.6% 0.0% 0.0% 0.0% 0.0%


## vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 2450842 1831831 5 0 0 0 0 0 0 6697 21909 1921 6 2 91
1 1 0 2450842 1831752 3 0 0 0 0 0 0 3517 5354 440 0 0 100
1 1 0 2450842 1831752 2 0 0 0 0 0 0 3523 5327 443 1 0 99
1 1 0 2450842 1831752 1 0 0 0 0 0 0 3510 5288 431 0 0 100
1 1 0 2450842 1831752 0 0 0 0 0 0 0 3486 5216 415 0 0 100


Tunable Current Planned Dynamic Auto Tuning
Value Value
=============================================================================================================================
create_fastlinks 1 1 no
dbc_max_pct 10 7 yes
dbc_min_pct 5 5 yes

[praveen..]

please suggest

[Jeff Schussele]

You're showing CPU pressure with these stats - not memory. You're not out of memory, you're not paging out.

Jeff

[Jeff Schussele · Accepted solution]

I take that back....
Count the headers & data.
You're totally *idle*!
You don't have a problem.

Jeff

[praveen..]

Hi jeff,
thanks for update.


But my server is too slow.
what should i do?
how to count headers & data?

[Yogeeraj_1]

hi,

what applications are you running?
what exactly are you finding slow?

kind regards
yogeeraj

[praveen..]

when i try with glance plus utility, it shows "page out" value is "0" zero.

should i need to increase the SHMMAX value.

My server is using excessive swapping which seriously degrades the server's performance.

Please suggest

[Bill Hassell]

vmstat shows no page outs at all so you are not swapping (at the moment). Why do you think the system is swapping? What does swapinfo -tam show? When you say the system is slow, has something changed? Did it used to be faster? What does disk usage (in Glance) show?

[praveen..]

Hi Bill,
Thanks for your update.

# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 3 0 2277293 1912954 5 0 0 0 0 0 0 6686 21950 1923 7 2 91
2 3 0 2277293 1912954 22 2 7 0 0 0 0 19974 68750 13246 8 10 82
2 2 0 2434928 1912875 18 1 5 0 0 0 0 19582 67378 13271 9 8 83
2 2 0 2434928 1912875 18 1 5 0 0 0 0 19582 67378 13271 9 8 83
2 2 0 2434928 1912875 15 0 4 0 0 0 0 22780 77924 15709 8 16 76

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 20480 0 20480 0% 0 - 1 /dev/vg00/swap2
reserve - 17439 -17439
memory 32735 20005 12730 61%
total 57311 37444 19867 65% - 0 -

[praveen..]

This is the output of #glance

GlancePlus C.03.85.00 00:19:20 dipper ia64 Current Avg High
-----------------------------------------------------------------------------------------------------------------------------
CPU Util | 1% 1% 3%
Disk Util F | 1% 3% 7%
Mem Util S SU UB B | 77% 77% 77%
Swap Util U UR R | 65% 65% 65%
-----------------------------------------------------------------------------------------------------------------------------

[praveen..]

i think that Swap space is used only, when the Page Outage occurs.

and my page out is 0 zero and swap utilization is 65%

[praveen..]

i think that Swap space is used only, when the Page Outage occurs.

and my page out is 0 zero and swap utilization is 65%

what should i do?

[RAC_1]

Your understanding is not correct. The swap is not used when swapout starts. Every process that starts will reserve some swap space. This is by design, because if on account of anything it needs to swap out, it should have space. So what you see in swapinfo is OK.

[Arunvijai_4]

Hi Praveen,

You should try with SEP's system monitoring script, http://www.hpux.ws/system.perf.sh

-Arun

[Bill Hassell]

From swapinfo, you can see 0% usage of the swap area. However, from Glance, you should be seeing the R____R bar extending to 65% which is just reserved swap space, not used yet. So your server slowdown is due to something else. Now the sar and top stats show your system idle so you'll need to look at the system when it is busy. If you're seeing login delays and applications that are slow to start, you probably have DNS issues.

[praveen..]

Hi,
as i am checking the "swapinfo" & "glance" report & "top", I am getting everytime "swap utilization is 65%" ,
i dont know this page is refreshing or not?

but why its value is not being changed for a little?

[Hein van den Heuvel]

Praveen,

I truly believe that your "server is too slow", as you write.

But you really need to take 6 steps back and ask yourself (and your peers) how to define slow, and what pertinent details you can provide when the system is slow.

Slow compared to what? (expectations, promisses, an other box, what the salesman said, how it used to run yesterday,...)
Slow when doing what? (Network requests? Compiling? Serving interactive users? Creating reports froma database? What database?)

When the system is slow, what is happening at that time? Are the stats you provided here taken at a time when the system was believed to be busy?... it was not busy at all from an HPUX point of view!
Do you have application or Oracle stats to work with?

Was it always slow? What changed?

You may or you might not have a memory problem. At this point you do not appear to have a memory problem, but there is really no point wondering about it untill you get a grip on what really is happening in the system.
Why worry ( and actively diddle!) with things like dbc_max_pct, is there is no indication what-so-ever there is has any connection to percieved slowness.

Maybe your network switch has gone crazy and runs at 1mb/sec instead of 100mb/sec.
That will make the system seem very slow from the users at the other side of teh wire, but the system itself will be happily idling away.

With the information provided up to this point I think that there is as much prove (none!) that my bad-network theory is correct as that your excessive memory usage is correct.

Please consider getting help (peers, historical data, consultants,... ) in defining the problem before worrying about the solution!

>>> as i am checking the "swapinfo" & "glance" report & "top", I am getting everytime "swap utilization is 65%" ,
i dont know this page is refreshing or not?
>>> but why its value is not being changed for a little?

You can probably guess my answer for this question by now: "who cares!".
When the main problem is clearer, this minor sub-problem will magically have become clear. It's a zen thing. trust me. :-).

[ The real answer was already given by RAC and Bill: it's just as reservation which chances slowly are processes come and go].

Good luck
Hein.

[praveen..]

Thanks Hein,

but why it is showing:

# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 2320149 1921887 5 0 0 0 0 0 0 6646 21681 1903 6 2 92
1 2 0 2320149 1921794 3 0 0 0 0 0 0 2479 3230 324 0 0 100
1 2 0 2320149 1921794 2 0 0 0 0 0 0 2493 3255 333 0 0 100
1 2 0 2320149 1921794 1 0 0 0 0 0 0 2500 3346 337 0 0 100
1 2 0 2320149 1921794 0 0 0 0 0 0 0 2493 3226 334 0 0 100



cpu utilization ~100%

[Hein van den Heuvel]

It's not. It is showing 100% IDLE, doing nothing, not even waiting, while you took those stats.

The 100 is in the 'id' column.
Real CPU work is under the SY (system) or US (user) columns.
And it shows that over all, since the system was up, (counting nights and weekend) that the average use was less than 10% with a reasonable system:user proportion


Finally that vmstat shows almost as many memory pages free as in use (4:3 ratio)

Hein.