load average on itanium

[Satya_6]

Hi,

got an itanium with Hp-UX 11.22, it just becomes very very slow after 2 or 3 days of uptime and someone does a console login. the load average becomes 6.5 to 7 at that time. Do a reboot, it would be normal again,

TIA
satya

[Paula J Frazer-Campbell]

Hi

Use glance/ top and monitor processes - 60 trial of glance in on HP cds.

What other software / apps do you have running?

Paula

[harry d brown jr]

So what kind of applications are you running?

Do you have measureware/glance installed?

What is the configuration?

Have you searched to see if you need any patches?

live free or die
harry

[Paula J Frazer-Campbell]

Please See


http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xf98719434a69d711abdc0090277a778c,00.html


Paula

[Ramkumar Devanathan]

Satya,

can you attach the output of the kmtune command on this machine? maybe it'd help.

For my part, I'd faced a similar problem, on some hp-ux 11.0 machines and thereafter increased the maxusers parameter to 512 from about 32 or some number close to that.

Thereafter the machine seemed to perform much better.

Also, see if there is enough swap space allocated. If not allocate more swap. Also turn on filesystem swap... you may use sam to do this.

maxusers has a cascading effect on most other kernel parameters. So, the system allocates more resources to start with and so, most processes run more efficiently.

setting maxusers to 512 is definitely not going to harm the working in any way - maybe there'd be a benefit - You may try this.

Was about to suggest using glance to analyse the problem... (if it is yet available for hp-ux 11.22) but i believe that'd be like taking coal to newcastle... since I know you would have thought of glance much much earlier than me.

- ramd.

[Satya_6]

Hi,

the m/c configuration is as follows

processor: 900 MHz
hardware model: ia64 workstation zx6000

Physical memory: 2042.0 MB
Swap: 4096 MB

OS: B 11.22 U
OS Kernel width : 64

Have hardly any apps running on the system and when this problem occurs it is the 'X'
which is taking all the CPU
(100 %).

Ram:

I have attached the kmtune output.


TIA
satya

[Paula J Frazer-Campbell]

Hi

Can you post sar outputs and a ps -ef taken during a high load.

If you do ps -ef > /tmp/highload then you should capture it all.
also a top output.

Paula

[Satya_6]

Hi,

As I had mentioned it is the 'X" server which takes the complete CPU. I have attached the ps and top output

TIA
satya

[Ravi_8]

Hi satya

from your top command, it shows process 'X' consuming lots of CPU time, can you restart that daemons since it is running from a very long time (851 hrs)

[Satya_6]

Hi Ravi,

something wrong there, actually the system's uptime is only 2 or 3 days;

whenever this problem occurs, I am just rebooting or restring the X server. but it keeps happening every 2 or 3 days.

I think I have to update some patches for this. I will go through the patch DB.

TIA
satya

[Rick Beldin]

Are you using the system as a server? That is, do you not have a mouse and keyboard installed?

If so, there is a known problem (that will be addressed in an upcoming CDE patch) whereby dtlogin tries 'too hard' to start the Xserver when no mouse and keyboard are present. If you are running such a configuration, a simple fix is to edit /etc/dt/config/Xservers and comment out the line that says:

* Local local@console /usr/bin/X11/X :0

As I said, this will be addressed in a upcoming patch.

The problem stems from the fact that the Xserver will not 'normally' start without a keyboard and mouse. When the Xserver fails, dtlogin tries to start it again... and again...

Now, if you *are* running this system with a keyboard and mouse, I would be suspicious of some of other problem, possibly an application problem.

Some things to note:

- how long are logged into your CDE session before this happens? Minutes? Hours? Days?

- are you running any applications continuously?

[Satya_6]

Hi Rick,

I am not using the m/c as a server. It has got keyboard and the mouse installed.


The problem generally happens after 2 or 3 days of CDE session. (that's defintely in terms of hours).

There are no applications runnning on the m/c.

TIA
satya

[Bill Douglass]

From your top outpuit, it looks like you're using a load of virtual memory.

Memory: 147184K (64356K) real, 13885464K (2528104K) virtual, 1153224K free

Also, you have about 40% CPU idle.

I wonder if you machine is paging itself to death when you start up a memory intensive app like X.

Can you post some vmstat statistics from before and during the slow-down?

Also, your bufpages and nbufs is set to zero. I wonder if your machine in trying to allocate excessive memory to the buffer cache, resulting in too memory pressure when you start up X.

[Satya_6]

Bill,

I don't think it is the problem with memory. I was thinking it is to do with the CPU load.

I have attached both the 'top' and 'vmstat' output during the problem and at nornal condition.

TIA
satya

[Mike Stroyan · Accepted solution]

You should apply patch PHSS_26650 for the X server and patch PHSS_26651 for OpenGL. (The OpenGL patch is required to make OpenGL work with the X server patch.) It seems that the X server has gone into an infinite loop, which does not really match any of the defect fixes in that patch. Still, it is a starting point. If the X server still goes to CPU-bound, you could use gdb as root from a network login to see what the stack trace looks like in the X server when it is CPU-bound. Is the console login working when the X server is acting like this?

[Bill Hassell]

The X server can go into intensive CPU usage when there are network problems. But these problems are not bad cables, it's usually due to improper logout procedures (a common problem when PCs connect using some emulator and then the PC crashes). Another possibility is that logins are turning off traps for common signals (especially SIGHUP) and once a remote user disconnects, the server gets confused as to why there are strange responses (from a non-existant client).

However, to avoid the instability of addign just a few patches that might help, add one of the most recent patch bundles. Xwindows is critically dependent on X-patches as well as networking and many kernel patches. Check the ITRC patch bundle pages for the current patch sets.

[Rick Beldin]

Going back a few responses - does the cpu load decrease if you logout out of CDE and then log back in? You indicated that you were logged in for several days.

Are there any applications (non HP) that are displaying on the graphics display that are up and running for the duration of your login? If so, does the cpu load decrease if you terminate them?



We might have a problem with what Mike said to try. I tried to do this on a fairly heavily patched ZX2000:

/opt/langtools/bin/gdb /usr/bin/X11/X 3568
Detected PA executable.
Invoking /usr/ccs/bin/gdbpa.
HP gdb 3.0.06 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 3.0.06 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..(no debugging symbols found)...
/root/3568: No such file or directory.
Attaching to program: /usr/bin/X11/X, process 3568

Aries LIMITATION: [pid=4100]
Aries cannot proceed emulation because
Encountered ttrace syscall with request "TT_PROC_ATTACH".
Attaching debugger to a running process is currently
NOT supported under ARIES.

[Mike Stroyan]

Bill Hassell- There is no support plus pack released for 11.22 yet.
Rick Beldin- It seems that you started gdb on the PA-RISC wrapper at /usr/bin/X11/X.
The big CPU-bound process should be the IPF native program at /usr/bin/X11/Xf86. gdb should be able to attach to it.

[Satya_6]

Hi Mike,

I am not finding any process for Xf86.

ps -aef | grep Xf86 gives nothing.
So I am not able to attach that process to gdb.
(However I could attach the process X to adb but it is not giving me any trace).

Should I run Xf86 instead of X
(shall I change
Local local@console /usr/bin/X11/X :0 to
Local local@console /usr/bin/X11/Xf86:0 in the
/etc/dt/config/Xserver file ?)

TIA
satya

[Mike Stroyan]

You should use Xf86 as the program to debug, but then attach to the process id that ps showed as /usr/bin/X11/X. That worked for me,
(as root).

# ps -fp 1616
%CPU PID S USER VSZ COMMAND
0.08 1616 S root 77804 /usr/bin/X11/X -fp tcp/:7000 :0 -auth /var/opt/gnome/gdm/:0.

# /opt/langtools/bin/gdb /usr/bin/X11/Xf86
HP gdb 3.1.1 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 3.1.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..(no debugging symbols found)...
(gdb) attach 1616
Attaching to program: /usr/bin/X11/Xf86, process 1616
...
(gdb) bt
...

[Satya_6]

Hi Mike,

I just updated the patches and it looks like it did the trick. I am finding the system up and working for the whole week without any problem.

Thank you very much

regards
satya