Server cannot use all the memory

[Vipin Singhal]

Hi All,
My server cannot use all the memory that is on that machine, which causes page swapping and significantly lower performance for large Transforms.

Please also see the following glance report:

GlancePlus C.03.85.00 16:22:38 warbler ia64 Current Avg High
---------------------------------------------------------------------------------------------------
CPU Util S | 1% 4% 37%
Disk Util FV V |100% 100% 100%
Mem Util S SU UB B |100% 100% 100%
Swap Util U UR R | 35% 35% 35%
---------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 7
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 1432 223374 318.2 335.7 543.7
Page In 1369 200148 304.2 300.8 394.0
Page Out 1132 147296 251.5 221.3 354.6
KB Paged In 6.5mb 973.6mb 1487.1 1498.5 2249.2
KB Paged Out 5.8mb 859.9mb 1313.7 1323.5 2656.9
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 1110 181639 246.6 273.0 485.7
VM Writes 2270 296966 504.4 446.3 906.6
Total VM : 9.4gb Sys Mem : 4.1gb User Mem: 7.4gb Phys Mem: 16.0gb
Active VM: 9.1gb Buf Cache: 4.4gb Free Mem: 61mb
Page 1 of 1
ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit

This is the output of :
# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
9 1 0 715530 1274201 12 0 18 18 2 0 168 4607 567979 669 15 1 84
9 1 0 715530 1274094 32 0 0 0 0 0 0 4257 857342 786 96 4 0
9 1 0 715530 1274075 30 0 0 0 0 0 0 4389 865638 826 96 4 0
9 1 0 713690 1274054 27 0 0 0 0 0 0 4175 845707 776 97 3 0
9 1 0 713690 1274028 26 0 0 0 0 0 0 4475 866659 846 96 4 0


Please suggest what to do?
Thanks
Vipin

[Patrick Wallek]

How much physical memory do you have? Is it 16 GB? How much swap space do you have configured? Do you have the swapmem_on kernel parameter set to 0 or 1?

If swapmem_on is set to 0, then you MUST have at least 16GB of swap space to utilize all 16GB of your RAM. If swapmem_on is set to 1, then you can get by with 4GB of swap space.

Post the output of 'swapinfo -tam' to help clarify the situation, please.

[Vipin Singhal]

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 156 3940 4% 0 - 1 /dev/vg00/lvol2
dev 20480 154 20326 1% 0 - 1 /dev/vg00/swap2
reserve - 2346 -2346
memory 16351 5066 11285 31%
total 40927 7722 33205 19% - 0 -


# machinfo
Memory = 16351 MB (15.967773 GB)
.......
.....
Please suggest

[Patrick Wallek]

It appears to me that you ARE in fact using all your memory. You have sufficient swap space (24GB) and you also have swapmem_on turned on. It looks to me like you are just running slightly short of RAM, with approx. 300MB having been paged out.

If the situation is worse at times, then I'd look at more RAM for this machine, if possible.

[A. Clay Stephenson]

Your pageouts are high. You really have 2 choices: 1) Add more memory 2) Reduce the load on the box. I suspect you are running Oracle. The first thing you need to do is reduce the size of the SGA's because the overhead of swapping is at least 100x greater than that of having to go to disk more frequently because of smaller database caches. You also may have your buffer cache set too large.

[Vipin Singhal]

Is there anyway to solve out this problem?

I am not in a condition to increase the RAM & reduce the workload.

This is Kernal Tunables:
Kernel Tunables:
----------------
physical_io_buffers 32768 (fix condor, quetzal, dipper)
eqmemsize 512 (fix condor, quetzal, dipper)
nstrtel 2048 (fix condor, quetzal, dipper)
npty 2048
nstrpty 2048
bufcache_hash_locks 4096
create_fastlinks 1
dbc_max_pct 30 (30% of main memory)
max_thread_proc 2048
maxdsiz 0xFFFFF000
maxdsiz_64bit 0x400000000
maxssiz 0x8000000
maxssiz_64bit 0x8000000
maxtsiz 0x40000000
maxtsiz_64bit 0x40000000
msgmax 32768
msgmnb 65536
msgseg 20480
msgssz 128
ncsize 32768
ninode 8192
scsi_max_qdepth 64
semmni 4096
semmns 8192
semume 512
shmmni 512
shmseg 512
vnode_cd_hash_locks 4096
vnode_hash_locks 4096


Strange (non-default) Values:
-----------------------------
maxuprc 3277 (3 <= maxuprc <= nproc - 5)
maxvgs 80 (1 <= maxvgs <= 256)
msgmap 5122 (3 <= msgmap <= min(msgseg,msgtql) + 2)
msgmni 1461 (1 <= msgmni)
msgtql 5120 ( max(1,msgmap-2) <= msgtql )
nfile 100000 ( max(2048, 2 * maxfiles_lim) <= nfile <= 2147483647 )
nswapdev 25 (1 <= nswapdev <= 25)
semmnu 4092 (1 <= semmnu <= nproc - 4)


Please suggest what to do?
Its urgent for me
Thanks
Vipin

[A. Clay Stephenson]

Your max_dbc_pct is much too high. You really want a buffer cache no larger than about 1600MB -- and that is generous. I would go to a fixed size buffer cache by setting bufpages to a non-zero value. bufpages = 307200 (307200 4K pages = 1200MB).

[Geoff Wild]

If you are running 11iv2 - then you can change max_dbc_pct on the fly - no reboot required - however - if you are at 11iv1 - you need an outage.

Rgds...Geoff

[Vipin Singhal]

Hi,
Please tell me the commands , how to make these changes, this will be very helpful to me on this my critical issue.

one thing more, everytime CPU utilization is 0%.
# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
9 1 0 709106 1279790 12 0 18 18 2 0 168 4605 572614 670 15 1 84
9 1 0 709106 1279686 26 0 0 0 0 0 0 3515 3114217 688 97 3 0
9 1 0 709106 1279686 25 0 0 0 0 0 0 3696 3168264 732 99 1 0
9 1 0 709106 1279659 23 0 0 0 0 0 0 3445 3156126 661 98 2 0
9 1 0 709106 1279636 22 0 0 0 0 0 0 3763 3098199 748 99 1 0

[Vipin Singhal]

HP-UX dipper B.11.23 U ia64 0103035585 unlimited-user license


Please tell me the step by step commands,
I will be very thankful to you
Vipin

[Geoff Wild]

Well...you can use sam -> Kernel and it will lauch the text based kcweb utility - and modify the kernel parameter dbc_max_pct.

Command line:

http://docs.hp.com/en/B2355-60105/bufpages.5.html

Rgds...Geoff

[Vipin Singhal]

Hi,
Please let me know the new parameters:

Please find the attached file
and suggest me the valuve of:
1. dbc_max_pct:
a. current value ?
b. planned value ?
2. dbc_min_pct:
a. current value ?
b. planned value ?

what about the CPU utilization?
what it is showing 0% used?

thanks
Vipin

[Vipin Singhal]

Please find the attached file and suggest me the new values?

Thanks
vipin

[john kingsley]

Try setting:
dbc_max_pct to 10%
dbc_min_pct to 5%

If you find you still need additional memory, you could probably even set dbc_max_pct lower to 5%.

With regards to CPU utilization, what is your vhand & swapper proccess activity? If these processes are active, it's possible that your other programs are waiting for pages to be moved back into memory.

[Vipin Singhal]

Please update
thanks
Vipin

[Patrick Wallek]

I would go with dbc_max at 8 and leave dbc_min at 5.

[Vipin Singhal]

in which i have to make changes?
kernal tunables or kernal usage?

is reboot required or not?

My OS is:
HP-UX dipper B.11.23 U ia64 0103035585 unlimited-user license


thanks
vipin

[Raj D.]

Hi Vipin ,

dbc_max_pct :

This determines the percentage of main memory to which the dynamic filesystem buffer cache is allowed to grow (When nbuf and bufpages are zero). The default is 50% of memory, but this is major overkill in most cases. With a huge bufcache, you are more likely to get into a situation where free memory is low and you will need to pageout or shrink the buffer cache in order to meet memory demands for active processes. You do not want to get into that situation. If you really want to use a dynamic buffer cache, start with dbc_max_pct at 25. If you have over 2 gigabytes of physical memory , start with it even smaller.

dbc_max_pct = 30 is defined , and seems very high.
You can make it max to 10 , min to 5 , & woudld be a good value , and will save some memory.

hth,
Raj.

[Vipin Singhal]

I have made the change to dbc_max_pct :8

Now the memory utilization is ok

but the CPU utilization is 100 % while using glance plus.

and cpu utilization using vmstat 1 5 command:

# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
10 1 0 712519 1274024 12 0 18 18 2 0 167 4602 583754 670 15 1 83
10 1 0 712519 1273888 32 0 0 0 0 0 0 3333 4620881 624 96 4 0
10 1 0 712519 1273870 29 0 0 0 0 0 0 3571 4612338 684 99 1 0
9 1 0 716150 1273870 27 0 0 0 0 0 0 3336 4749570 627 98 2 0
9 1 0 716150 1273849 24 0 0 0 0 0 0 3141 4724251 577 99 1 0


why this thing going to be happen?
thanks
vipin

[Geoff Wild · Accepted solution]

From your last vmstat - looks good - looks like the box isn't "swapping" anymore - IE no more deactivations....

System is just busy - how many cpus do you have?

Rgds...Geoff

[Raj D.]

Hi Vipin ,

In vmstat output check for the pi po values , ( Page in , Page out), high non zero value shows excessive paginbg activity.

In your first report page/in was showing excessive activity , (12,32,30,27,26 )

But after changing dbc_max_pct , it looks pretty good , and page/in page/out values shows 0 , thats very good.



Cheers,
Raj.

[Joel Girot]

Hi Vipin,

another kernel parameter, the vx_ninode, can consume excessive memory with the default value 0 on system with large physical memory.
The vx_ninode parameter specifies the maximum number of inodes available in the VxFS in-memory inode cache. If not explicitly specified, the value of vx_ninode is based on how much physical memory is available on the system. See the VERITAS File System Administrator's Guide for more information.
See also this thread : http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=206230

Rgds, Joel