topic Re: High Load Queue, High Idle Time, Java in Operating System - HP-UX

High Load Queue, High Idle Time, Java

Pedro Cirne — Fri, 06 Jan 2006 06:07:59 GMT

Hi,

I've a hp-ux 11i running mainly java 1.3 and I'm experiencing a "strange" situation, top shows high load queue, but idle time is also high!:

Atached find some top/sar statistics.

Any ideia on how can I improve this situation? The system response is strange, sometimes whe typing comands on shell things freeze for 2 or 3 seconds....

Thks,

Pedro

Re: High Load Queue, High Idle Time, Java

Arunvijai_4 — Fri, 06 Jan 2006 06:12:43 GMT

Hi Pedro,

Do you have required patches for Java installed on your server ? http://h18012.www1.hp.com/java/patches/index.html

Try upgrading to JDK1.4 or 1.5 since 1.3 is going to be dead on March'06.

-Arun

Re: High Load Queue, High Idle Time, Java

Pedro Cirne — Fri, 06 Jan 2006 06:17:14 GMT

Hi,

Yes I have the latest patche bundles installed:

root@poru9011:/tmp# /usr/sbin/swlist | grep -e QPK -e GOLD -e BUNDLE11i
BUNDLE11i B.11.11.0102.2 Required Patch Bundle for HP-UX 11i, February 2001
GOLDAPPS11i B.11.11.0509.429 Applications Patches for HP-UX 11i v1, September 2005
GOLDBASE11i B.11.11.0509.429 Base Patches for HP-UX 11i v1, September 2005

Upgrading to more recent Java versions is not an option :((

Pedro

Re: High Load Queue, High Idle Time, Java

Arunvijai_4 — Fri, 06 Jan 2006 06:26:07 GMT

Hi Again,

Can you try using HPJmeter to see what exactly happens ? Since this is java, it requires exhaustive analysis

http://www.hp.com/products1/unix/java/hpjmeter/index.html

-Arun

Re: High Load Queue, High Idle Time, Java

Clifford Chi — Fri, 06 Jan 2006 07:16:59 GMT

Hi Pedro,

May I suggest the following after I have reviewed the attachment:

1. Install PHCO_32466. This is a "top" cumulative patch. Need to make sure the "top" is reporting correct figures.

2. From your "top" output, I am seeing username "transfer" has multiple Java processes. Is this normal?

Note: there is a zombie process reported in your "top" too. Use "ps -elf" and check on "S" column for process that has "Z" status, this is a zombie process. Kill it.

Re: High Load Queue, High Idle Time, Java

Mark Nieuwboer — Fri, 06 Jan 2006 07:35:15 GMT

Hi predro,

can you give the output of kmtune -l or sysdef. This problems can happen when your kernel is not compiled the right way for java.
For java you must increase a lot of kernel parameters such as.
max_thread_proc.

grtz. Mark

Re: High Load Queue, High Idle Time, Java

Eric Antunes — Fri, 06 Jan 2006 07:38:32 GMT

Hi Pedro,

Can you tell me your swap situation and timeslice parameter value?

Best Regards,

Eric Antunes

Re: High Load Queue, High Idle Time, Java

Pedro Cirne — Fri, 06 Jan 2006 07:39:00 GMT

Hi Mark,

root@poru9011:/# sysdef
NAME VALUE BOOT MIN-MAX UNITS FLAGS
acctresume 4 - -100-100 -
acctsuspend 2 - -100-100 -
allocate_fs_swapmap 0 - - -
bufpages 26214 - 0- Pages -
create_fastlinks 0 - - -
dbc_max_pct 20 - - -
dbc_min_pct 5 - - -
default_disk_ir 0 - - -
dskless_node 0 - 0-1 -
eisa_io_estimate 768 - - -
eqmemsize 23 - - -
file_pad 10 - 0- -
fs_async 0 - 0-1 -
hpux_aes_override 0 - - -
maxdsiz 503866 - 0-503866 Pages -
maxdsiz_64bit 262144 - 1024-1073479679 Pages -
maxfiles 2048 - 30-2048 -
maxfiles_lim 2048 - 30-2048 -
maxssiz 2048 - 0-98048 Pages -
maxssiz_64bit 2048 - 4-262144 Pages -
maxswapchunks 2048 - 1-16384 -
maxtsiz 16384 - 0-503859 Pages -
maxtsiz_64bit 262144 - 1024-1073741823 Pages -
maxuprc 200 - 3- -
maxvgs 10 - - -
msgmap 39 - 3- -
nbuf 21216 - 0- -
ncallout 7235 - 6- -
ncdnode 150 - - -
ndilbuffers 30 - 1- -
netisr_priority -1 - -1-127 -
netmemmax 0 - - -
nfile 30010 - 14- -
nflocks 200 - 2- -
ninode 4796 - 14- -
no_lvm_disks 0 - - -
nproc 4116 - 10- -
npty 60 - 1- -
nstrpty 60 - - -
nswapdev 10 - 1-25 -
nswapfs 10 - 1-25 -
public_shlibs 1 - - -
remote_nfs_swap 0 - - -
rtsched_numpri 32 - - -
sema 0 - 0-1 -
semmap 63 - 4- -
shmem 0 - 0-1 -
shmmni 200 - 3-1024 -
streampipes 0 - 0- -
swapmem_on 1 - - -
swchunk 2048 - 2048-16384 kBytes -
timeslice 10 - -1-2147483648 Ticks -
unlockable_mem 16664 - 0- Pages -

I already run the HPjconfig from HP to tune the kernel.

Thks.

Re: High Load Queue, High Idle Time, Java

Sameer_Nirmal — Fri, 06 Jan 2006 11:00:13 GMT

Hi,

The load average values shown in top could be verified using
sar -q
Glance
uptime
w
# adb /stand/vmunix /dev/kmem
adb>avenrun/3F

There is a patch for load avarage
PHKL_33336.

Since the system responce is not normal, there is a resource problem for sure on the system. The attached report show swapping is being done.

I would do complete system check up using
sar , vmstat , iostat, netstat , Glance
syslog, demsg , swapinfo -tam, bdf etc.

For Java side, I would check the java processes using q4/p4 to know what they are doing and resources being used could be known as well. Verifying java specific kernel parameters such as maxssize,max-thread_proc, nkthread and ulimit.

Re: High Load Queue, High Idle Time, Java

Bill Thorsteinson — Fri, 06 Jan 2006 11:04:41 GMT

Given the number of transfer proceses
I suspect you are running into heavy
swapping. This will kill performance
in the manner you are experiencing.

Java garbage collection will cycle most
of the pages into memory each run. If
there are a lot of objects being created
and discarded this can occur every few
seconds. Starting transfer will smaller
memory limits could help.

Can you nice or stop most of the transfer
processes?

Re: High Load Queue, High Idle Time, Java

Pedro Cirne — Fri, 06 Jan 2006 11:11:45 GMT

Hi Bill,

The server is for running those java processes with username "transfer", that's what I want the server to do so no use to nice or stop them.

Thks,

Pedro

Re: High Load Queue, High Idle Time, Java

Mark Nieuwboer — Mon, 09 Jan 2006 04:42:06 GMT

Hi Pedro,

Here is a link wich can help to tune the os for java.
http://www.hp.com/products1/unix/java/infolibrary/prog_guide/java1/index.html.

grtz. Mark

Re: High Load Queue, High Idle Time, Java

RAC_1 — Mon, 09 Jan 2006 05:09:39 GMT

Let me tell you one thing. The load average is not a very correct way of deciding how system is doing. The laod average is basically no. of jobs waiting in queue to run over a gievn period of time.

the important que global priority run queue.
You can check it with glance.
Prepare a file as follows. (call it glance.syntax"
print "global priority queue", gbl_pri_queue

Do you have a very high priority queue??

Also is system experiencing any kind of bottleneck?? -disk i/o, memory, swap, cpu??
disk i/o may show the behaviour you are experiencing now. Check global wait states in glance. glance -B, glance -i, iostat, sar -d

Also not to forget java patches.

Re: High Load Queue, High Idle Time, Java

Eric Antunes — Mon, 09 Jan 2006 05:18:30 GMT

Hi Pedro,

It seems that those java processes are eating lots of memory.

See "who" are they with the attached script (usage: mostra_procs_pesados.sh ).

It may be a bug...

Best Regards,

Eric Antunes

Re: High Load Queue, High Idle Time, Java

Eric Antunes — Mon, 09 Jan 2006 05:19:33 GMT

Re: High Load Queue, High Idle Time, Java

Pedro Cirne — Mon, 09 Jan 2006 05:36:47 GMT

Hi RAC,

Thank you for your reply, attached find sar/glance/iostat stats, would you have a look?

Thank you very much,

Pedro

Re: High Load Queue, High Idle Time, Java

RAC_1 — Mon, 09 Jan 2006 05:49:56 GMT

Apart from memeory usage, all seems fine.
Post vmstat 1 5 also.

Check what processes are usig most of the memory. With java processes, you can control what heap size (initial/max) they can use. java --h for details. That you can control some java sizes.

UNIX95= ps -ef -o pid,ruser,vsz,args|sort -nrk3

Re: High Load Queue, High Idle Time, Java

Pedro Cirne — Mon, 09 Jan 2006 06:20:10 GMT

Hi RAC,

Attached find the vmstat statistics.

Thks,

PEdro

Re: High Load Queue, High Idle Time, Java

Arunvijai_4 — Mon, 09 Jan 2006 06:26:00 GMT

Hi Petro, here is a link to a good guide "Performance tuning Javaâ ¢: kernel parameters affecting Javaâ ¢ performance"

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1616,00.html

(You should have DSPP membership in order to view this page, its free!)

-Arun

Re: High Load Queue, High Idle Time, Java

Arunvijai_4 — Mon, 09 Jan 2006 06:27:35 GMT

One more link, Performance tuning Javaâ ¢: training & tutorials

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1607,00.html

-Arun