topic A trouble with program performance in Operating System - HP-UX

A trouble with program performance

xinm — Mon, 22 Dec 2008 07:00:05 GMT

Dear friends:
When I run the same C++ program in machine A and B and C.First,A and B completed in 7 hours,C completed in 8 hours.But after about 10 days,A and B used 14 hours or more then,and C still completed in 8 hours.
After about 10 days,restarted B,then B will complet about in 7 hours.But run the program tomorrow,the trouble still happened.
This program used maccloc() frequently.
So I checked program’s config file,check the program’s size,check the program version,check .profile,all same!
--------------------------------------------
A,B machinfo

CPU info:
Number of CPUs = 8
Clock speed = 1100 MHz
Bus speed = 400 MT/s

Cache info:
L1 Instruction: size = 16 KB, associativity = 4
L1 Data: size = 16 KB, associativity = 4
L2 Unified: size = 256 KB, associativity = 8
L3 Unified: size = 4096 KB, associativity = 16

Memory = 24562 MB (23.986328 GB)

OS info:
sysname = HP-UX
release = B.11.23
version = U (unlimited-user license)
machine = ia64
vmunix _release_version:
@(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
----------------------------------------
C machinfo
CPU info:
Number of CPUs = 8
Clock speed = 1595 MHz
Bus speed = 532 MT/s

Cache info (per core):
L1 Instruction: size = 16 KB, associativity = 4
L1 Data: size = 16 KB, associativity = 4
L2 Instruction: size = 1024 KB, associativity = 8
L2 Data: size = 256 KB, associativity = 8
L3 Unified: size = 9216 KB, associativity = 9

Memory = 16350 MB (15.966797 GB)

OS info:
sysname = HP-UX
release = B.11.23
version = U (unlimited-user license)
machine = ia64
vmunix _release_version:
@(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
-----------------------------------------
sar –ud 2 1000
c9t0d0 1.99 0.50 53 385 0.00 0.69
10:02:59 15 23 47 14
c2t1d0 100.00 65.20 428 21985 80.39 16.69
c9t0d0 4.00 0.50 62 565 0.00 0.94
10:03:01 16 15 52 18
c2t1d0 100.00 103.83 516 23574 68.59 14.48
c9t0d0 4.02 0.50 52 365 0.00 1.00
10:03:03 12 26 34 28
c2t1d0 99.50 107.61 392 15385 111.80 16.61
c9t0d0 3.00 0.50 53 398 0.00 0.85
10:03:05 14 32 33 22
c2t1d0 94.97 9.07 238 17148 25.69 24.20
c9t0d0 5.03 0.55 59 712 0.22 2.40
10:03:07 16 24 46 13
c2t1d0 99.50 78.81 480 26205 73.25 14.05
c9t0d0 1.99 0.50 49 381 0.00 0.66
--------------------------------------------------------------------------

I hope somebody can help me.
Thanks!

Re: A trouble with program performance

Dennis Handly — Mon, 22 Dec 2008 12:51:23 GMT

>This program used maccloc()

What does it do?

>But after about 10 days, A and B used 14 hours or more then,

Does C take longer and longer too?
Do you have a memory leak such that these processes take more and more memory?

Will it be faster if you just restart the application, rather than the whole machine?

Re: A trouble with program performance

James R. Ferguson — Mon, 22 Dec 2008 13:02:35 GMT

Hi:

> This program used maccloc() frequently.

If you meant 'malloc()' then you should probably look for memory leaks.

Regards!

...JRF...

Re: A trouble with program performance

xinm — Mon, 22 Dec 2008 14:16:12 GMT

Thanks all!
I am sorry for typing "maccloc()",it is malloc().
to Dennis Handly:
C doesn't take longer and longer time.
If only restart program,the trouble is still.
to James R. Ferguson:
If memory leak,why C is ok,there are the same program on three machines(A,B,C).

Re: A trouble with program performance

James R. Ferguson — Mon, 22 Dec 2008 14:36:17 GMT

Hi (again):

> If memory leak,why C is ok,there are the same program on three machines

Are the three machines identical in terms of patches, particularly in regard to 'libc' patches?

Regards!

...JRF...

Re: A trouble with program performance

Dennis Handly — Wed, 24 Dec 2008 14:52:02 GMT

>If only restart program, the trouble is still.

Still there after a while? Or right away?
Hmm, I guess you wouldn't know that unless you monitored the memory size in top.

Re: A trouble with program performance

Bill Hassell — Wed, 24 Dec 2008 17:10:24 GMT

Is this program primarily compute-bound, that is, it does perform a lot of disk or network activity? If there is a lot of disk activity, then the first are to look at is disk performance on the problem server.

The second has to do with installed memory. A and B have 24GB but C has only 16GB. Since the problem doesn't seem to occur on the C machine, what else do you have running on the A and B machines at the same time? Since C is smaller, is it less busy?

Re: A trouble with program performance

Mike Stroyan — Wed, 14 Jan 2009 17:06:34 GMT

One problem that can affect a system over time is fragmentation of memory. That can happen over a period of minutes, hours, or days, depending on how the system is being used. It can be much worse if the dbc_max_pct and dbc_min_pct kernel tunables are at different values. Perhaps your system "C" has those tunables set to the same value. That would make it would be relatively immune to the effect in contrast to systems "A" and "B".

There can be a performance penalty from having dbc_max_pct different than dbc_min_pct. The dynamic buffer cache has a tendency to use 4K pages which are scattered around physical memory. That can interfere with applications which are trying to use large pages. A large page setting such as "chatr +pd 1M" or "ld +pd 1M" can really improve the performance of programs that jump around accessing data in a large memory region. It reduces the number of TLB misses.

If you are running large processes you should consider using a fixed buffer cache size, and possibly setting the chatr +pd attribute on programs which do not already have it set. (Large pages can increase the amount of RAM a program needs by rounding up to bigger pages, so don't go too far with it.)