memory,swap overflow

[mhzzang]

HI! I'm HP Server admin.. in korea.
I Haver HP superdome (PA-RISC 8700 875Mhz*20EA
40GB Memory)
asiq125 & SAS running in system
in these days,CPU utilization is 20%(Ave.),
memory util is 98%, swap util is 52%(use 13G,reserve 40G).
sometimes, when user types any command, system is shown like hang, it takes time 10~30minutes for response of commands.
kmeminfo of this system is below.
what cause of this system?
memory ?

Summary of processes memory usage:

List sorted by physical size, in pages/bytes:

virtual physical swap
pid ppid pages / bytes pages / bytes pages / bytes command
13434 1 7095727 27.1g 5622189 21.4g 7110339 27.1g asiqsrv12
16829 1 22923 89.5m 12845 50.2m 12828 50.1m midaemon
5605 1 27517 107.5m 10330 40.4m 15320 59.8m scopeux
13471 1 9224 36.0m 8833 34.5m 8881 34.7m csmgr
23696 19244 10112 39.5m 8614 33.6m 9020 35.2m bgscollect

Total: 5662811 21.6g 7156388 27.3g




--------------------------------------------------------
Summary of processes memory usage:

List sorted by swap size, in pages/bytes:

virtual physical swap
pid ppid pages / bytes pages / bytes pages / bytes command
13434 1 7095727 27.1g 5622193 21.4g 7110339 27.1g asiqsrv12
7620 7602 210044 820.5m 190132 742.7m 209711 819.2m sas
4819 1 33240 129.8m 3501 13.7m 30969 121.0m vxsvc
17353 17338 26079 101.9m 7864 30.7m 15363 60.0m glance
5605 1 27517 107.5m 10334 40.4m 15320 59.8m scopeux
28981 3975 25475 99.5m 7460 29.1m 14758 57.6m glance
16829 1 22923 89.5m 12845 50.2m 12828 50.1m midaemon
23309 5414 17284 67.5m 6869 26.8m 10435 40.8m egsagent
23696 19244 10112 39.5m 8614 33.6m 9020 35.2m bgscollect
13471 1 9224 36.0m 8833 34.5m 8881 34.7m csmgr

Total: 5878645 22.4g 7437624 28.4g






asiqì ë í Memory ì ¬ì ©í í ©
----------------------------------------------------------------------
Process's memory regions (in pages):

Process "asiqsrv12", pid 13434, 64bit, R_SHARE_MAGIC:

type space vaddr ref virt phys swap
TEXT 0x77d1000.0x4000000000000000 2 39 9 1
UAREA 0x001f400.0x400003ffffff0000 1 8 8 9
UAREA 0x0028000.0x400003ffffff0000 1 8 8 9
.................................
................
MMAP 0xe39c400.0x800003ff7aae3000 1 257 5 259
MMAP 0xe39c400.0x800003ff7abe4000 1 257 8 259
MMAP 0xe39c400.0x800003ff7ace5000 1 257 7 259

SHLTEXT 0xbd5a400.0xc000000000380000 5 36 31 1
SHLTEXT 0xbd5a400.0xc000000000400000 9 281 200 2
SHLTEXT 0xbd5a400.0xc000000000540000 2 137 34 1
SHLTEXT 0xbd5a400.0xc000000000600000 2 151 0 1
SHLTEXT 0xbd5a400.0xc0000000006c0000 2 74 0 1
SHLTEXT 0xbd5a400.0xc000000000740000 2 129 0 1
SHLTEXT 0xbd5a400.0xc000000000800000 2 1452 761 5
SHLTEXT 0xbd5a400.0xc000000000dc0000 2 90 78 1
SHLTEXT 0xbd5a400.0xc000000001000000 2 3799 1950 13
total 7095

[Dietmar Konermann]

Please check if PHKL_28695 and PHKL_28410 are installed. If not, then installation of these would be my 1st advise.

[malay boy]

Did you twig your kernel parameter ??? If yes,please re-check and if can rechange to the default.

regards
mB

[Steve Steel]

Hi


Look at
ftp://eh:spear9@hprc.external.hp.com/memory.htm

It may help but this sort of thing is normally sybase tuning if the patch level is up to date.

Steve Steel

[Sridhar Bhaskarla]

Hi,

Looks like you are definitely under memory pressure. It may be due to memory leak or you really are using it.

I would suggest you run the following periodically and see if there is any difference in the vsz portion of the static processes.

UNIX95= ps -e -o 'vsz pid ppid ruser args'

Pipe the output of the above command into a file every hour. After a day or so compare the first and last file and see if any process has it's size grown. That may be a possible culprit.

You may also check your dbc_max/min_pct kernel parameters. dbc_max_pct is set to 50% by default. It should shrink down when there is a memory pressure and it does on well on 11i but check how much it is used. If this parameter is 50% and if you are using even 5-10 GB of it, then it's a waste of memory. You will need to get it reduced to 500-700MB.

-Sri

[Ted Buis]

I have seen systems appear to be unresponsive to commands, when there were network problems. Sometimes half/full duplex problems on a 10/100BaseT with a switch can make things seem very unresponsive.

Use Glance to see the number of page outs and then look at your swap devices to see if you are saturating any of those disks.

What is your buffer cache? If it is too large, that isn't good.

Lastly, if you are at 98% RAM utilization, you most likely problem is that you are out of virtual memory. You need at the least to create additional swap space on disk.