Performance issues with our HPUX 11.11 boxes

[Christina Wang]

Hi, Experts,

I'm new to HPUX system admin stuff. We got a very strange problem with our HPUX 11.11 boxes. We have two HPUX boxes. One has 4-1000mhz 9000/800/rp3440 CPUs with 8gb RAM and 4 gb swap. The second one has 2-800mhz 9000/800/rp3410 CPUs with 4gb RAM and 4 gb swap. They are all installed with HPUX B.11.11. We need to build the same large scale project using GNU gcc 4.1 on both machines. With the same compiler, the first machine is much slower(about 10 times slower) than the second one although it has faster CPUs. If I use "top" to monitor the processes while building the project, I can see that the CPU usage rate is very low on the first machine -- also about 10 times lower than the second one. It seems the slow machine spends a lot of time on file I/O and the CPU idle time is always around 98%-99%.

I'm wondering what kind of system configurations could cause this slowness? What fields should we look into? All the files are on local disks. Thanks in advance!


Christina

[A. Clay Stephenson]

First, compare the kmtune outputs of the two machines and look for any obvious differences. You are making the classic mistake of thinking that one machine has 4 1000MHz processors and the other has 2 800 MHz processors and therefore one box should be able to run ~2.5X as fast as the other. That would only apply to multi-threaded largely CPU-bound processes. In your case, I would expect compiles to be roughly 25% faster; all other things being equal --- which they aren't. I assume that 10x slower means that the actual elapsed time for two otherwise identical builds takes 10 minutes on one box and 1 minute on the other --- or some similar ratio.

One of the things that might explain your observed behavior is a very small buffer cache. This would trigger lots of delays waiting of i/o. The default dynamic buffer cache (dbc_max_pct) of 50% is almosy always ludicrous but someone may have really tuned it down to a tiny value. If bufpages it set to a non-zero value then the buffer cache is fixed so check that setting first. You want a bugger cache no smaller than about 100MiB and no larger than about 1600MiB --- for general purpose boxes with your amount of memory under 11.11 somewhere between 800-1200MiB would be about right.

Top is pretty much worthless as a diagnostic tool. You need to use Glance if possible. If you do not have Glance installed (use the swlist command to find out; just "swlist" will suffice), then you can install a 60-Day Trial version from any Applications CD set. Glance will allow you to pinpoint the performance bottlenecks in short order. If you have an X terminal or emulator such as Reflection X you can run the Motif version of glance, gpm, and get nice graphs.

[Christina Wang]

Hi Clay,

Thanks for your reply. You are right that "10x slower means that the actuale lapsed time for two otherwise identical builds takes 10 minutes on one box and 1 minute on the other --- or some similar ratio."

As what you suggested, I used "kmtune" to compare the outputs on these two boxes. The slower machine gave two extra fields that are not on the output of the faster machine:

dmp_max_msecs 10000 - 10000
dmp_pathsw_bshift 10 - 10

Other fields that are different are:

from the slower machine:

ncsize 6344 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
nfile 2107 - (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
ninode 1224 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 1808 - (((NPROC*7)/4)+16)
nproc 1024 - (768+8*MAXUSERS)
nsysmap 2048 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 2048 - ((NPROC)>800?2*(NPROC):800)
shmmax 0x8000000 Y 0x8000000

From the faster machine:

ncsize 5596 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
nfile 910 - (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
ninode 476 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 499 - (((NPROC*7)/4)+16)
nproc 276 - (20+8*MAXUSERS)
nsysmap 800 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 800 - ((NPROC)>800?2*(NPROC):800)
shmmax 0x4000000 Y 0X4000000


Anything that seems wrong here?

Thanks!

Christina