HPUX 11.22 Process switching CPU problem

[HP-UX SYSTEM MANAGER]

We just installed a new analysis server, running dual Itanium 900MHz CPUs and HPUX 11.22. When we benchmark it, we found that a job that took 2.5 hours on a C3600 (one CPU, 1gig RAM) took 7.5 hours on the dual 900MHz Itanium with 4gig RAM.

We notice that as the process runs, it continuously switcehs back and forth from CPU 0 to CPU 1. In the course of about 3 hours, we have over 60,000 switches! Is this an attempt to load-balance the dual CPUs? On our old system (running 2 PA-RISC CPUs) the process stayed on 1 CPU until completion.

Any help on this is greatly appreciated.

[Jeff Schussele]

Hi,

What is the kernel parameter
timeslice
set to?
If it's 1 CHANGE IT 10.
That alone could cause massive context switching.

Rgds,
Jeff

[doug hosking]

There are many possible variables here.
If you think it is related to the presence
of the 2nd CPU, you can try a simple
experiment to disable the second processor
temporarily by booting with the '-P1' option.

HP-UX Boot Loader for IA64 Revision 1.713

Press Any Key to interrupt Autoboot
\EFI\HPUX\AUTO ==> boot vmunix
Seconds left till autoboot - 10
Type 'help' for help

HPUX> boot vmunix -P1

Using the 'P1' option as shown above will
temporarily restrict the system to using
only one of the available processors. This
will get undone automatically on the next
reboot.

This should help you tell whether the slowness
is related to (needless?) switching between
processors. Keep in mind that the IPF
systems may save and restore much more state
than PA systems do when doing a context switch.
A future release of HP-UX will contain some
optimizations to that process.

There are other possible explanations for
the extra time. How was the application
compiled for the rx2600? Are you running
a PA or an IPF version of the binary?
Although PA binaries will generally work
on IPF systems they will generally run
somewhat slower than native IPF code would.
It's best to recompile them native if you can.

Can you describe the nature of the job in
question? (Does it do a lot of I/O ? Use a lot
of memory? Do a lot of system calls ? A lot of floating point computations?)

[John Bolene]

I also noticed this behavior when doing benchmarking seti runs on a superdome. With 15 processors idle and 1 at 100%, the load would switch processors about every 10 seconds. It was like the idle ones were hungry and wanted something to do.

It must be some kind of optimization that is done with 11.22 as it was not happening with 11.0 on the last multiprocessor system I tested.

It is much more efficient to keep the process in the same cpu.

[doug hosking]

Keep in mind that 11.22 is the IPF version of HP-UX. The PA releases have had many years for tuning of the machine dependent parts of the system. The IPF versions are still undergoing that tuning. It still seems premature to assume that the problem is due solely to context switch overhead. 60,000 context switches should not take anywhere near several hours. I strongly suspect something else is affecting the results, even if the context switching code isn't optimal. I'd be very interested to see answers to the previously asked questions about the nature
of the application.

[HP-UX SYSTEM MANAGER]

To anser all your questions:
Timeslice is set to "10" already; that's the first thing we checked. Our old system was set to 1, and we didn't have this problem.

We cannot reboot the machine right now; it's in production. However, I can answer the questions about the actual program.

The program in question is an engine analysis program, which was compiled on/for PA-RISC. It is mainly floating point ops, very CPU intensive but not particularly disk or i/o heavy.

Also, I didn't mean to say that the cause of the slowdown was due to the switching; just that this is a new behavior that we didn't see in our old system. If possible we'd like to disable it without disabling the CPU. I was hoping for further insight into why the analysis application would run so much slower.

I'll see if we can get a recompile on it.

[Pete Randall]

That's extremely interesting that the program is heavy on floating point - the IPF architecture is supposed to excel at floating point!! Hmmm!


Pete

[doug hosking]

I wonder if what you are seeing is overhead from aries. You say you are running a PA binary on the IPF system. While supported, and a great way to help in migration from PA to IPF, nobody claims that aries is the optimal way to run applications on IPF systems. Is it possible to recompile the application native? It would not surprise me at all to see a very significant improvement in performance when you do.

[rick jones · Accepted solution]

60000 CPU switches in three hours is epsilon - small enough as not likely to matter. However, if you want to lock the process to a given CPU, you can use the mpsched command - either by launching the process from mpsched, or after the process is running.

I think that Doug's suggestion to try to get a native IPF version of the binary is a very good one. There may also be some useful intformation to be had from Caliper or perhaps prospect. http://www.hp.com/go/prospect and http://www.hp.com/go/caliper. If not now, certianly once you have a native IPF binary.