performance issue HPUX 11

[Adam Noble]

We currently have a major perfromance issue on one of our servers. Effectively whenever a user logs into an application and effectively connects to the database a process is generated with a resident set size of around 16MB. We have over 100 users and simply not the memory to satisfy them. The question I have is via Glance does the RSS include shared memory regions. I drill down and can see that the process is using shared memory segments but you cannot quantify this.

I need to be able to go back to users and say yes each process is using 16MB, however I can't tell what % (if any) of that figure is shared memory.

Any help would be appreciated.

[Robert Salter]

Adam,

Did you try using ipcs? Do a man on it to get all the options, but ipcs -m will show the memory segments used. It sounds like the database needs to be tuned a bit, if it's allocating 16mb per connection, unless that's a default. Your DBA may be able to reconfigure their parameters. Or per chance the application has some tuneable parameters. Or did you check your system tuneable parameters to make enough resources available?

Just some thoughts on where to look. Good luck.

Bob

[A. Clay Stephenson]

If you use Glance Reports->Process List (Select Process) then Reports->Process Memory Regions, you will see a detailed breakdown of text, data, stack, and shared memory for each process. That should give you enough data to address your problem.

[rick jones · Accepted solution]

After you've followed the good glance advice, you may also want to check with chatr the data page size being requested by the appliction(s) launched for each user. That, and examine the settings of the vps_* kernel tunables. Someone may have attempted to tune the application performance by increasing the page size. This can increase performance - particularly by decreasing time spent dealing with TLB misses, but it can increase the memory usage as the memory gets given to processes in page-sized quantities.

If the drilling-down doesn't help, one thing you might do to see how much is shared versus not is load-up a few users, see what the memory usage is on the system as a whole, then eliminate those users (gently please :) and check the memory again, then add users back and check and such. If as you increase users, you see memory usage increase by 16 MB a time, them you can presume it is all unshared.

[Adam Noble]

I think Rick appreciates my dilemma, I have been able to drill down and see where the memory is being allocated, however for example it gives a shared memory value of 479RSS 545 VSS. Therefore I can't quantify how much of the 16MB of RSS is shared memory. Through rough estimates like Rick suggests I'm guessing each connection is actually only around 4MB. I can see this through limiting user logins like suggested. Anyway I guess there is no exact way of seperating the process like pmem on Solaris.

[Don Morris_1]

Adam -- you can use the pstat_getprocvm() interface to do this. I'm attaching a program I wrote for this purpose in an 11i v2.0 environment a little while back... fair warning, I haven't tested it on 11i or 11.0 (though I believe I've made the change needed to handle it... PS_RSESTACK shouldn't exist on those releases, but everything else should work) -- and there may still be minor bugs, this was mainly for my own use debugging a large virtual size question for a system daemon. Feel free to rewrite it as you see fit.

[Don Morris_1]

Oh... forgot to add -- the output is in pages, not bytes. So you'll want to multiply appropriately (usually 4k... but if you feel
the need, sysconf() with _SC_PAGESIZE will of course tell you).

[Don Morris_1]

Missed that pst_swap isn't available on earlier releases. Here's a revised version of the code to handle that.

[Mike Stroyan]

The glance RSS total actually tries to report the amount of shared memory a process is responsible for. It does that by taking the total RSS for each shared segment and dividing that by the number of processes that segment is mapped into. That means that glance will report more shared memory RSS for a process when it has fewer peers using the same segments.

I have attached a program that uses pstat_procvm and reports the same RSS numbers as glance.