memory usage - how to identify the process

[Mariusz Grabowski_2]

Hi,

I have a system with high cpu memory usage, and wondering, how to identify the process, that causes this.

first of all - it's 2gb, where ~.5 gb goes for system/buffer cache, the other 1.5 should be available for others. but during the night, when oracle (oracle db runs there; sga = ~400mb) is down for backup, memory usage should be limited to ~.5gb, which is 25%, but it stays on a level of 1gb, which is 50%.

I\ve idenfied a shm area owned by root (using ipcs), but I'm unable to find any process, that is bound to that area (ipcs shows 2 processes are there; unftly, LPID/CPID do not exist, so I'm unable to find out..)

anyone has an idea how to address this? thanks in advance
mariusz

[Steven E. Protter]

Best utility for the job is lsof.

It may be that the process that allocated the memory is dead and the correct course of action is ipcrm

ipcrm -m

-m changes based on the type of memory on the ipcs display.

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.74/

SEP

[Mariusz Grabowski_2]

SEP,

unftly - lsof is not available for me (don't ask why) :(

the mem segment is still in use (if it wasn't, ipcs'd show '0' as nattach)
thus - running ipcrm might harm this :)

thanks, m

[Kent Ostby]

Well, a lot of times, I find that the top memory user is also a top CPU user (though not always).

ps -ef | cut -c42-80 | sort -nr | head

repeat and see who is chewing up memory.

Of course there is also the russian roulette method, kill what you think is using up memory and re-check memory.

Finally, if you have an HP contract, you could do a TC on the box and ask them to tell you which processes are taking up memory.

They have a utility that can pull that data from a crash dump.

[Geoff Wild]

LSOF here:

http://hpux.ee.ualberta.ca/hppd/hpux/Sysadmin/lsof-4.74/

Rgds...Geoff

[Mariusz Grabowski_2]

thanks again, but the server is running (in general) ok, I just wonder what is the memory consumer. doing tc would be fine, if i'd had a production issue - but it's not the case this time.

and - cpu consumer does not correspond to memory this time :(

thanks again - some other ideas?

cheers, m

[Geoff Wild]

Try this script:

# cat memtop
#!/bin/sh
# memtop - show top memory users and pid
# VSZ is in KB
echo "VSZ(KB) PID RUSER COMMAND"
UNIX95= ps -e -o 'vsz pid ruser args' |sort -nr|head -30


Rgds...Geoff

[Vincente Fernandes]

Look like Oracle doesn't clear the shared memory/semaphores.
When oracle is down for backup use
ipcs -m |grep oracle
ipcs -s | grep oracle
or write a little script and schedule it under cron to find out the shared memory/semaphores in use after oracle is down for backup.
use ipcrm -m id and ipcrm -s id to clear the shared memory/semaphores.
ps -ef at that time will also tell you the processes running on server

[Bill Hassell]

The best tool is of course Glance. It will identify great details about specific processes as well as identify the type of memory being consumed. Standard Unix tools are fairly useless for this. Note that if anyone uses kill -9 for database prcesses, this will definitely leave orpahned shared memory areas (as well as corrupted indexes and bad data records). The major memory areas are: kernel, processes (local memory), shared memory and buffer cache. Your buffer cache is probably still set to the (very bad) default of 50% max. So when processes terminate, the buffer cache will grow automatically. For 2Gb RAM, the maximum buffer cache should be about 20% (kernel param = dbc_max_pct).

To see your shared memory area for 32bit programs, get a copy of shminfo from:

ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

[Mariusz Grabowski_2]

thanks Bill & Vinc. I've tried to analyze the situation with glance, but RSS in glance do not give a full picture...

i have a dbc_max_pct set to 15%, so 300mb is max for my system.

cheers, mariusz

[Bill Hassell]

The resident set size (RSS) is the portion of the program which is currently in memory. You can have a 500meg program with a 400meg shared area but have only 200megs in RAM. As noted in the Glance help for RSS, the measurement depends on whether the oprocess has consumed any CPU time during the measurement period.