Glance showing sys mem very high - whats using it ?

[Stefan Farrelly]

Hi all,

We have a new production server running 11i with Oracle Financials. Were already suffering memory creep (or leak). The amount of Sys(tem) memory as shown in glance keeps going up, despite us shutting down compeltely the application nightly for backup (this reduces user memory usage back to the same level after a reboot so there is no leak here but Sys memory usage does not fall back to its value after a reboot).

As an example;
1. our old Financials server running 10.20 only shows Sys memory usage in glance as 200Mb.
2. Other N class servers running 11.0 and large Oracle databases show Sys memory usage in glance of around 200 Mb also.
3. Our new 11i Nclass shows Sys memory usage in glance of over 1Gb !! (this starts at 500Mb after a reboot, but keeps going up every night when we shut down the app for a backup. Once we shutdown our app the Sys memory total does not fall back to the value before the app was started, and after restarting the app the Sys memory total grows again).

Ive checked everything. Buffer cache is the same (this is shown separately in glance anyway so its not part of the Sys total), kernel table values are almost identical on all servers, semaphores/message queues the same, shared memory usage is again separate - not part of the Sys total).. Were patched up to the Mar2003 bundles.

I guess my main question is how do you get a breakdown of the Sys memory total from glance.

Anyone know how this total is compiled?

[Steve Steel]

Hi

Go to

http://rc.rose.hp.com/RRC/pcc/calltemplate/unsupported_utilities.htm

Get procsize which will give you a broken down usage.


See the manpage

If the url does not access

procsize (documentation)is available at???



External ftp site:

System: hprc.external.hp.com (192.170.19.51)

Login: eh
Password: spear9

ftp://eh:spear9@hprc.external.hp.com/

ftp://eh:spear9@192.170.19.51/

ftp://eh:spear9@hprc.external.hp.com/procsize.sh

ftp://eh:spear9@hprc.external.hp.com/procsize.README.txt


procsize.README.txt

oo transfer (FTP) from PC to UNIX in ASCII mode or ^M's will appear and

cause problems during the un-shar'ing.

oo extract with ???sh procsize.sh???

oo procsize requires a parameter, either ???f for a full report on all processes

or???p pid for a report on a specific process.

(e.g. './procsize -f|more' or './procsize ???f >/tmp/procsize.txt')

oo example:

# ./procsize -f

pid Comm UAREA TEXT DATA STACK SHMEM IO MMAP Total

2916 getty v 4 5 6 4 0 0 349 369

2287 prm3d v 68 6 671 513 0 0 37212 38471

oo Here are some other options for procsize:

#./procsize ???fc <-- Shows common memory regions for all PID's

# ./procsize ???fcn <-- As above but only show shared regions once

#./procsize ???fcRV <-- As above but break out VSS and RSS information

oo Since procsize is not a supported utility there is no documentation for it, so we are left
to educated conjecture regarding what the columns mean. Luckily they are pretty self-explanatory???


pid = process id

Comm = name of the command/process

UAREA =User Area

TEXT = The size of the resident memory occupied by text regions of this process.

DATA = The size of the resident memory occupied by data regions of this process.

STACK = The size of the resident memory occupied by stack regions of this process.

SHMEM = The size of the resident memory occupied by shared memory regions of this process.

IO = (see iomap(7) for information about this one; I've never seen any value besides 0.)

MMAP = The size of the resident memory occupied by memory mapped regions of this process.

Total = The total size of the memory occupied by this process.



Steve Steel

[James R. Ferguson]

Hi Stefan:

One consideration is, is 'glance' misreporting(?).

There is a good Knowledge Base paper (#KBRC00011764) which discusses how to analyze memory usage. In particular, the WTEC 'kmeminfo' tool is mentioned as most useful when analyzing system memory.

Regards!

...JRF...

[Zeev Schultz]

kmeminfo is ok,only need good explanations/docs
on it. :(

[James R. Ferguson]

Hi (again) Stefan:

Knowledge Base document #4000027449 notes that with 11.11 'glance' now accounts for a larger fraction of memory allocation as "system" rather than "user". Thus, comparisons of 11.11 versus pre-11.11 'glance' metrics is misleading from this standpoint alone.

Regards!

...JRF...

[Zeev Schultz]

this is a nice one covering all tools (shminfo,kmeminfo,procsize & glance):
KBRC00011764

[Stefan Farrelly]

Thanks James and Zeev but I cannot find either document you mentioned. I even registered at the US ITRC site and tried searching there, neither can be found. Can you please let me know where I can find these documents or post them here as a reply ?

Thanks,

Stefan

[Zeev Schultz]

Stefan
here you go

Zeev

[Zeev Schultz]

Stefan
here you go

Zeev

[Stefan Farrelly]

Thanks Zeev and Alex - Ive read the documents and can now see 2 things;

1. Glance is now accurately reporting Sys memory usage under 11i (didnt in 10.20/11.0) - which explains the large increase in 11i over 11.0.

2. Buffer cache is now *included* in the Sys mem total from Glance, but on the memory line glance shows Buf Mem separately - whereas its actually part of Sys Mem (according to kmeminfo).

However, we still have our problem. Each night with our application completely closed for backup the Sys Mem total does not reduce to previous level, and on application restart it has grown by another couple of hundred Mb. I will delve into this further....

[Tom Danzig]

I seem to recall a memory leak issue related to snmp (/usr/sbin/mib2agt?) at one time. A quick check for 11.11 patches revealed this:

PHSS_25978:
1.snmpdm doesn't respond properly for large invalid packets
2.snmpdm doesn't respond properly in case of malformed PDUs
3.There is a memory leak associated in snmpdm with an improperly encoded packet.

May or not be related.

[Steven E. Protter]

Stefan,

This problem is not limited to Oracle financials. It can occur with other Oracle products based on the same technologies(ias).

Some relief to this issue occurred by putting together a script that blew away shared memory resouces after the oracle product was shut down.

It seemed that certain oracle processes were staying up even after shutdown and hanging shared memory resources.

So we came up with something that ran through the ipcs display and issued ipcrm commands of the right user owned the resource.

Eventually we got a patch from Oracle and the process hang stopped happening.

I'd upload the script, but I'm not near my office and have no access.

Good Luck,

SEP

[Bill Hassell]

Make sure that you have the latest version of Glance and midaemon. 11.11 changed a lot of the structures that midaemon uses to obtain its data and some unrelated patches to the kernel can affect the accuracy of midaemon/Glance reports.

[Stefan Farrelly]

Hi Bill,

Thanks for replying. Yes, downloading the latest version of glance now. However, Ian from the HPRC has kindly sent me the latest kmeminfo which seems to work much better and gives more detailed output, the problem is it now reports Sys Mem usage up from 1.7Gb (old glance) to 2.2Gb !! another 500Mb unaccounted for. Kmeminfo output follows, glance output to follow soon.

Question is where on earth is 2.2Gb used by the System ?? The totals above dont seem to add up.

Loading symbols from /stand/vmunix
Kernel TEXT pages not requested in crashconf
Will use an artificial mapping from a.out TEXT pages
kmeminfo (3.77)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Tue Jun 3 20:36:59 BST 2003
boot: Tue Jun 3 21:06:17 2003
dump: Thu Jun 5 13:14:10 2003

----------------------------------------------------------------------
Pfdat processing:

Scanning 2042248 pfdat entries (be patient) ...


----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physmem = 2092544 8.0g 100% Physical memory
Freemem = 681688 2.6g 33% Free physical memory
Used = 1410856 5.4g 67% Used physical memory
System = 570308 2.2g 27% By kernel:
text = 2486 9.7m 0% text
data = 427 1.7m 0% data
bss = 1206 4.7m 0% bss
Static = 98281 383.9m 5% for text/static data
Dynamic = 363208 1.4g 17% for dynamic data
Bufcache = 104627 408.7m 5% for buffer cache
Eqmem = 46 184.0k 0% for equiv. mapped memory
SCmem = 4146 16.2m 0% for critical memory
User = 845509 3.2g 40% By user processes:
Uarea = 5336 20.8m 0% for thread uareas
Disowned = 8 32.0k 0% Disowned pages

----------------------------------------------------------------------
Kernel dynamic memory usage (in page/byte/percent):

Dynamic = 363208 1.4g 17% Kernel dynamic memory
Arenas = 338044 1.3g 16% Kernel arenas
vx_buffer_cache = 101632 397.0m 5%
vx_inode_cache = 94786 370.3m 5%
vx_global_pool = 89531 349.7m 4%
M_SPINLOCK = 10945 42.8m 1%
ALLOCB_MBLK_LM = 8802 34.4m 0%
M_TEMP = 4647 18.2m 0%
M_SWAP = 4206 16.4m 0%
M_LVM = 3010 11.8m 0%
KMEM_ALLOC = 2874 11.2m 0%
M_REG = 2790 10.9m 0%
M_PREG = 2695 10.5m 0%
VFD_BT_NODE = 2426 9.5m 0%
M_VXVM = 1367 5.3m 0%
M_IOSYS = 1330 5.2m 0%
ALLOCB_MBLK_DA = 760 3.0m 0%
Other = 6243 24.4m 0% Other arenas...
Kalloc = 25109 98.1m 1% kalloc()
SuperPagePool = 6803 26.6m 0% Kernel superpage cache
BufcacheBufs = 11071 43.2m 1% Buffer cache bufs
BufcacheHash = 5120 20.0m 0% Buffer cache hash heads
Other = 2115 8.3m 0% Other...
Eqalloc = 55 220.0k 0% eqalloc()

[Stefan Farrelly]

I now have the latest version of Glance for 11i installed; C.03.70.00

It now reports;

Total VM : 5.00gb Sys Mem : 1.70gb User Mem: 3.25gb Phys Mem: 7.98gb Active VM: 2.29gb Buf Cache: 408.7mb Free Mem: 2.63gb

System Memory as 1.7Gb, same as old version reported. But new kmemnifo reports Sys Mem as 2.2Gb ! Im even more confused and even more memory short than I was when I started this...

Any bright ideas anyone ?

[Stefan Farrelly]

It appears we are getting somewhere.

The latest glance, C.03.70, for 11i is installed by same result so no help there.

The new kmeminfo is revealing and we have found the following issues from it;

1. kernel value vx_ninode is set to 0, which means the kernel dynamically sets it, and with more ram its set larger. This is a possbile cause. I will hard set it (guideline is 10% less than ninode) and reboot.

2. We're running JFS B.03.50.0, which the HPRC reports has known memory and performance problems, and we should consider reverting back to 3.3. We will investigate doing this.

[Sridhar Bhaskarla]

Hi Stefan,

The glance sysmem metric does not include buffer cache. As the help says
//
On HP-UX 10.20 and 11.0, this metric
does not include some kinds of
dynamically allocated kernel memory.
This has always been reported in
the GBL_MEM_USER* metrics.
On HP-UX 11i and beyond, this metric
includes some kinds of dynamically
allocated kernel memory.//

While there is no documentation provided by HP giving the information about "some kinds of", I agree with you that it is because of the dynamic setting of vx_ninode cache. However on 11i, there is no vx_noifree kernel parameter to control the freeing up of inode cache unlike on 11.x. But on 11.x, you don't have this issue :-).

Did you check the user memory usage on your 10.20 or 11.0 systems?

Though I observed the same trend on my systems, I am still living with them as our 11i systems have plenty of memory.

-Sri

[Sridhar Bhaskarla]

Hi Stefan,

The glance sysmem metric does not include buffer cache. As the help says
//
On HP-UX 10.20 and 11.0, this metric
does not include some kinds of
dynamically allocated kernel memory.
This has always been reported in
the GBL_MEM_USER* metrics.
On HP-UX 11i and beyond, this metric
includes some kinds of dynamically
allocated kernel memory.//

While there is no documentation provided by HP giving the information about "some kinds of", I agree with you that it is because of the dynamic setting of vx_ninode cache. However on 11i, there is no vx_noifree kernel parameter to control the freeing up of inode cache unlike on 11.x. But on 11.x, you don't have this issue :-).

Did you check the user memory usage on your 10.20 or 11.0 systems?

Though I observed the same trend on my systems, I am still living with them as our 11i systems have plenty of memory.

-Sri

[Jim Barden]

This is not relegated to Oracle products. We have a sybase data warehousing server with extreme memory demands, which also exhibits the same symptoms. Sys memory keeps increasing after a reboot, and free memory keeps going down. We have tried everything, but don't have an explanation. Even HP backline engineers did not have an explanation for this.

[Martin Johnson]

You do not mention the version of Oracle you are running. Some versions have known memory leaks. Make sure you have the latest oracle patches installed.

HTH
Marty

[Stefan Farrelly]

Just as an update to this problem;

setting vx_ninode to 90% of ninode (ie. a non-zero value) fixed our problem. With this now set our system memory usage was over 1GB Lower!

So my advice is, set vx_ninode (dont leave it set to 0) - it will save you lots of RAM.

[Bill Hassell]

ninode is a real problem parameter. It defaults to a very obsolete formula that assumes all filesystems are HFS (not VxFS) and vx_ninode is based on ninode's value unless explicitly set (as you discovered). ninode is the basis for vx_ninode, maxcdnodes, ncsize and the directory name lookup cache (DNLC). ninode defines the HFS cache and indirectly defines the VxFS inode cache, NFS mode cache, the CacheFS cnode cache and the DNLC cache.

The recommendation is to reduce ninode by elimating the formula and setting it to a small fixed value, perhaps 1024. Then explicitly set ncsize to set the DNLC size. DNLC is a cache of current and recently opened filenames. When opening a file, HP-UX first looks in the DNLC cache to see if the location is already in memory. If it is, no directory search is needed and the file can be opened in a few milliseconds rather than the thousands of milliseconds needed for a physical disk search.

There are a number of decisions to make about ncsze, ninode, vx_ninode and maxcdnodes. The best reference for these values is found in the NFS performance book: "Optimizing NFS Performance" by Dave Olker. It has excellent information on why 11i is so much better for buffer cache performance as well as improving NFS performance.

[Steven E. Protter]

We also recently solved oracle performance problems by dealing with inode parameters.

There were other issues, but our product set ran better after that fix.

Oracle 9ias 1.0.2.2 PL 12
Oracle Server EE 8.1.7.4.0

rp5450 Server running 2 GB RAM

Two weeks later one of the two CPU's went on holiday.

There is a good document on Oracle/General performance that was given to me by one of HP's better Oracle performance people.

UPERFKBAN00000726

I know you might not be able to see that being in Europe, but I'm trying to find a link that will work for anyone.

http://www1.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchString=UPERFKBAN00000726&search.x=28&search.y=8&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty

Will this work for Europe? I don't have a European ID so I can't test. Its a great doc, worth tracking down.

SEP