swainfo and memory usage issue...

[Hoang Chi Cong_1]

Dear all,
I have to face a trouble: current, my system's memory is always almost used.
I check in glance for a few days and the usaged very hight:
Mem Util ......... 95% 97% 100%
Swap Util ......... 80% 85% 95%

The out put of swapinfo:
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 1015236 7373372 12% 0 - 1 /dev/vg00/lvol2
reserve - 6758764 -6758764
memory 12237992 7759204 4478788 63%

The total of RAM in my system is 16GB:
(here is from cstm)
Basic Memory Description

Module Type: MEMORY
Total Configured Memory : 16384 MB
Page Size: 4096 Bytes

Memory interleaving is supported on this machine and is ON.


I think there is something wrong when setting for memory.
Could someone give me some advice?
What should I do?
Do I have modify any kernel parameter?


Any advice will much appreciated!

Regard,
Hoang Chi Cong

[Arunvijai_4]

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=791999

-Arun

[Steven E. Protter]

Collect somre more data and see it this situation is temporary.

http://www.hpux.ws/system.perf.sh

Post kmtune output to this thread.

An article:
http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000079989196

http://www1.itrc.hp.com/service/cki/enterService.do?category=c1s3ss6

I'd say your system is being stressed.

The load could be too high or it could be a poorly written application. What does the server do? What does it run?

SEP

[Devender Khatana]

Hi,

Allthough the memory is 16Gb the primary swap in only 8Gb. Is it done intentionally ?

Please post
#swapinfo -atm

Also what is the application running on the system? Even in this scenario if everything is running fine and processes does not go out of memory then everything should be fine but you should plan for memory upgrade.

HTH,
Devender

[Hoang Chi Cong_1]

Thanks to all.

S.E.P, I attached the kernel parameters...

The out put of swapinfo -atm:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 987 7205 12% 0 - 1 /dev/vg00/lvol2
reserve - 6603 -6603
memory 11951 7763 4188 65%
total 20143 15353 4790 76% - 0 -


Thanks
Rgd,
Hoang Chi Cong

[morganelan]

We normally have swap 2X RAM, unless specified by your application.You must calculate your max swap like this:

max swap = (maxswapchunks x swchunk x 1024)

I see your swap = 20 GB, I think you must have at least 36 GB swap because you have 16 GB RAM.

[morganelan]

Sorry I mistake to type, I mean that your swap = 20 GB, I think you must have at least 32 GB swap (2 x 16 GB) because you have 16 GB RAM.

[RAC_1]

Do you have glance?? glance -m

How is memeory used?? If buffer cache used optimum??
sar -b 2 5

Is write hit ration good?? Should be in the range to about 90 or more.

Your kernel settings look ok, except following. (May depend upon your apps)

nfile
msgmax
msgmnb

Watch the real time usage and modify ccordingly??

sar -v 3 4
glance -t

[morganelan]

From your kernel parameters:
When I calculate (maxswapchunks x swchunk x 1024) to get maximum swap device can be configured on your server is: 4096x2048x1024= 8589934592 == 8 GB, I think you must change your maxswapchunks and swchunk to get at least 32 GB.

[Devender Khatana]

Hi,

I suspect dbc_max_pct and dbc_min_pct values. These are set a bit high 30 & 10. Allthough in my case with similar memory the same is set to 8 & 5 respectively. But it also depends on application. What is that ?

-Devender

[Hakan Aribas]

check the following kernel parameters:

dbc_max_pct
dbc_min_pct

If you don't need file cache frequently (for example raw device instead of file system), you can decrase dbc_max_pct to 5 or 10. Decreasing dbc_max_pct will give you more memory for applications.

usage:
kcusage -l dbc_max_pct
kcusage -l dbc_min_pct

kctune dbc_max_pct

[Hoang Chi Cong_1]

Hi

Let see on the attachment. I have checked and log the out put.

My system is a banking system. We use Oracle 8i Database and Tuxedo as MiddleWare and of course, our system is a OLTP system.

Regard,
Hoang Chi Cong

Point will be submit later...

[Matthew Ghofrani]

Hi;
Try running the following command"
ps -elf|awk '{print $10, $3, $4, $15, $16}'|sort -nr|more

This gives you all the current processes accoring to the most mem hungry, then check out who is out of order. Below is a sample of my oracle8i & Tuxedo on one of my servers where the first coloum is memory by page size, in my case is 512 byte

@@@@@@@@@
124:28 /opt/ecc/exec/MHR520/mhragent
6542 indepth 23293 55:41 pss_rx
5390 indepth 23274 15:51 /indepth/java/1.4.2/JRE/bin/PA_RISC2.0/java
5263 oracle8i 16329 2:39 oracleSYS8
5247 oracle8i 15645 0:45 oracleSYS8
5231 oracle8i 17422 7:12 oracleSYS8
5179 oracle8i 17094 4:35 oracleSYS8
5023 oracle8i 15821 0:14 oracleSYS8
4931 oracle8i 16577 0:22 oracleSYS8
4923 oracle8i 7485 0:23 oracleSYS8
4923 oracle8i 15845 0:01 oracleSYS8
4859 oracle8i 24918 oracleSYS8 (LOCAL=NO)
4851 oracle8i 27639 oracleSYS8 (LOCAL=NO)
4843 oracle8i 14400 0:03 oracleSYS8
4843 oracle8i 128 oracleSYS8 (LOCAL=NO)
4675 oracle8i 15606 0:00 oracleSYS8
4551 oracle8i 9341 2:45 ora_dbw0_SYS8
4539 oracle8i 9343 2:43 ora_dbw1_SYS8
4431 oracle8i 28970 oracleSYS8 (LOCAL=NO)
4410 root 3686 11:25 /opt/ecc/exec/EGO520/egoagent
4407 oracle8i 16745 5:02 oracleSYS8
@@@@@@@

Matthew from Boston

[Sameer_Nirmal]

First thing the values of dbc_min_pct and dbc_max_pct are seems to be little high.
dbc_max_pct=30% i.e. 4.80GB of physical memory is used as system buffer. The usage of buffer seems to be OK in term of % , but doesn't necessarily to be optimum. It is not always possible to fulfill the required data
only from system buffer through logical reads and involves physical I/O to storage devices. Now its depends on how storage is configured and optimized to take care of physical I/O load and your application behaviour.

Secondly, I would advice to put on swap in term of physical device instead of using memory which is more precious . If possible add device swap equal to 16GB and remove pseudo swap by changing kernel paramter
swapmem_on=0 . This will release memory of about 7.75GB as shown in swapinfo and system will be out of swapping as well ( it is now using 12% device swap actually )

[Patrick Wallek]

Actually, setting swapmem_on to 0 will NOT release memory. HP-UX does not actually use physical memory when swapmem_on is set to 1. It is a number that HP-UX uses, approx. 75% of actual RAM size, to add to available swap space for reservation purposes. But actual physical RAM is never used.

[Ted Buis]

I would agree that it is very likely you can reduce the dbc_min_pct and dbc_max_pct, but I would suggest that you first use glance to see what the hit ratio is for the buffer cache, so that you can see if it is substantially reduced by your actions after reducing those values. Also, first look to see if you are actually doing any page-outs. If you are not, then you are simply using the memory you have purchased which isn't bad. You do have to have enough swap chunks to cover the swap size. The amount of swap you need depends on the application, but may not need to be increased unless you turn off pseudo-swap. Pseudo-swap is good as long as you never do any real page-outs. If you do page-out then the limited number of pages of RAM that can be swapped out will get hit unfairly, compared to those assigned as psuedo-swap pages which effectively locks them in memory. Since assignment of pages into psuedo-swap is arbitrary, this can mean that performance can be quite irregular if page-outs do occur. It was designed for large memory configurations where the likelyhood of page-outs was very small.