Memory Usage

[Ralph Grothe]

Hello,

I'm a bit embarrassed to ask you this.
But it seems I know too little of the fundamentals of Unix memory management to give a convincing explanation when asked by someone.

The box is a K460 running HP-UX 11.00.

It is fitted according to the memory inventory that you may find as an stm dump attached to this posting (i.e. it has 1 GB RAM where all banks are enabled and no DIMMs marked as failed)

Unfortunately we haven't got a glance license for this box so that I have to use vanilla Unix tools (viz. mainly vmstat).

Below you can see a vmstat dump over 6 intervalls of 20 secs. (just as an example).

Also you may find the dump of swapinfo.

The problem of "insufficient" memory arose when DBAs installed yet another Oracle 9i instance while there was an Oracle 8.1.7 instance already running.

Because they weren't able to start this new Oracle 9i instance I suspected that it was too memory hungry and needed to reserve more swap space than was available at the time of start attempt.
I've seen many SAP/Oracle startup difficulties that simply resulted from too little swap space.
Thus I donated yet another swap device, but that didn't help either.

Swap space now is this

# swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1126400 302848 823552 27% 0 - 1 /dev/vg00/lvol2
dev 1048576 0 1048576 0% 0 - 2 /dev/vgbs11/lvol2
reserve - 574308 -574308
memory 720044 609624 110420 85%
total 2895020 1486780 1408240 51% - 0 -


This is what vmstat reported (during a randomly picked short interval)

# vmstat 20 6
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 0 0 11767 3032 3 2 0 0 2 0 2 1223 1098 149 4 1 94
0 1 0 77748 2920 0 0 0 0 0 0 0 1042 154 50 0 0 99
0 0 0 75172 2920 0 0 0 0 0 0 0 1038 103 45 0 0 100
0 0 0 12378 2922 0 0 0 0 0 0 0 1033 91 44 0 0 100
0 0 0 75114 2922 0 0 0 0 0 0 0 1037 140 45 0 0 100
0 0 0 78633 3012 0 0 0 0 0 0 0 1039 114 48 1 1 98


Only for this very short sampling period the active virtual memory is 55135 pages averaged, while free memory is 2954 pages averaged.

If I recall correctly the page size is 4 KB.
This would yield some 215 MB avm and only 11.5 MB free.

Now I wonder what the phys. RAM is used for.

Therefore I did this crude calculation,
i.e. summing up all pages used by processes.

# UNIX95= ps -e -o sz=|awk '{s+=$1};END{printf"%10.2f\n",s*4/1024}'
1337.13

which is beyond the phys. RAM size.
Thus, I assume that probably many pages are currently paged out to the swap devices (e.g. of long runners).

Are my assumptions correct?

What other tools, apart from Glance, can I use to broaden my view of memory usage?

What can I suggest the DBAs how to overcome their Oracle 9i problems?

Rgds.
Ralph

[Karthik S S · Accepted solution]

Find the attached memory management whitepaper which will answer your basic questions on memory allocation in HP-UX. As far as the oracle instance is concerned I think it is something to do with the kernel configuration. Check kernel parameters like maxswapchunks etc.

-Karthik S S

[Stefan Farrelly]

Your assumptions are pretty good. Basically, the device line from swapinfo shows over 300MB swapped out - so youre pretty brave trying to install another oracle instance when you are so short of memory (300MB+ short). Sounds like a shared memory problem to me - it needs to be a contiguous chunk of memory which it just cant get. Is the new instance 32bit or 64bit ? I would shutdown a currently running instance or get the users off, then your new instance install should work.

[Jdamian]

Locate kmeminfo in /usr/contrib/Q4/bin and run it.

It will report memory usage. I think it can be useful.

[Jdamian]

I also suggest to check the shared memory regions because this kind of memory is not swapped. Run

ipcs -ma

[Ralph Grothe]

Karthik,

many thanks for the whitepaper!
This will educate me more on memory issues.

Swap chunk Nos. and size should, though not terribly high, allow me to set up 8 GB of swap space (swapon has claimed it without complaints anyway).

# kmtune -q swchunk -q maxswapchunks
Parameter Value
===============================================================================
swchunk 2048
maxswapchunks 4096


Stefan,

yes, I also doubt that they will be able to, with the resources given, bring up that Oracle 9 beast as long as they aren't willing to free resources occupied by the other instance.

I assume they're going for a 64 Bit build.
Probably that's what made them upgrade?

How can I find out?
Yeah, I should ask them...

Damian,

is kmeminfo a harmful command?
Though, it doesn't sound like.

However, I found these docs in the Q4 tree

# compress -dc /usr/contrib/Q4/doc/Q4Docs.tar.Z |tar tvf -
r--r--r-- 3031/171 56490 Dec 2 19:05 1997 Q4BeginnersGuide.ps
r--r--r-- 3031/171 215246 Dec 2 19:05 1997 Q4ProgrammersReference.ps
r--r--r-- 3031/171 37747 Dec 2 19:05 1997 Q4KernelDevelopersGuide.ps
r--r--r-- 3031/171 154211 Dec 2 19:05 1997 Q4ReferenceManual.ps

But by superficial glance at those in gv I couldn't find a reference for kminfo.


The shared mem segs look like this,
with procs/threads attached to all.
(I anonymized the users, not to offend anyone ;-)

# ipcs -mbo
IPC status from /dev/kmem as of Tue Feb 17 15:18:19 2004
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c173c --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 2 31040
m 2 0x41200035 --rw-rw-rw- root root 2 8192
m 9731 0x00000000 D-rw------- root root 7 1052672
m 2052 0x00000000 D-rw------- www other 7 184324
m 50181 0x3ff0e178 --rw-r----- oracle dba 19 93814784
m 1542 0x30204df4 --rw-rw-rw- mem leakers 7 73728
m 1543 0x3020518c --rw-rw-rw- mem leakers 7 14462976
m 1544 0x302050be --rw-rw-rw- mem leakers 7 122880000

[Steve Steel]

Hi

I was given this url once

ftp://eh:spear9@hprc.external.hp.com/memory.htm

It helped me a lot.


Steve Steel

[Ralph Grothe]

Steve,

thank you for another valuable source.

I hope they won't change the login because I suspect this was given to you under the premise of non-disclosure?