High Load Queue, High Idle Time, Java

[Pedro Cirne]

Hi,

I've a hp-ux 11i running mainly java 1.3 and I'm experiencing a "strange" situation, top shows high load queue, but idle time is also high!:

Atached find some top/sar statistics.

Any ideia on how can I improve this situation? The system response is strange, sometimes whe typing comands on shell things freeze for 2 or 3 seconds....

Thks,

Pedro

[Arunvijai_4]

Hi Pedro,

Do you have required patches for Java installed on your server ? http://h18012.www1.hp.com/java/patches/index.html

Try upgrading to JDK1.4 or 1.5 since 1.3 is going to be dead on March'06.

-Arun

[Pedro Cirne]

Hi,

Yes I have the latest patche bundles installed:

root@poru9011:/tmp# /usr/sbin/swlist | grep -e QPK -e GOLD -e BUNDLE11i
BUNDLE11i B.11.11.0102.2 Required Patch Bundle for HP-UX 11i, February 2001
GOLDAPPS11i B.11.11.0509.429 Applications Patches for HP-UX 11i v1, September 2005
GOLDBASE11i B.11.11.0509.429 Base Patches for HP-UX 11i v1, September 2005

Upgrading to more recent Java versions is not an option :((

Pedro

[Arunvijai_4]

Hi Again,

Can you try using HPJmeter to see what exactly happens ? Since this is java, it requires exhaustive analysis

http://www.hp.com/products1/unix/java/hpjmeter/index.html

-Arun

[Clifford Chi]

Hi Pedro,

May I suggest the following after I have reviewed the attachment:

1. Install PHCO_32466. This is a "top" cumulative patch. Need to make sure the "top" is reporting correct figures.

2. From your "top" output, I am seeing username "transfer" has multiple Java processes. Is this normal?

Note: there is a zombie process reported in your "top" too. Use "ps -elf" and check on "S" column for process that has "Z" status, this is a zombie process. Kill it.

[Mark Nieuwboer]

Hi predro,

can you give the output of kmtune -l or sysdef. This problems can happen when your kernel is not compiled the right way for java.
For java you must increase a lot of kernel parameters such as.
max_thread_proc.

grtz. Mark

[Eric Antunes]

Hi Pedro,

Can you tell me your swap situation and timeslice parameter value?

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi Mark,

root@poru9011:/# sysdef
NAME VALUE BOOT MIN-MAX UNITS FLAGS
acctresume 4 - -100-100 -
acctsuspend 2 - -100-100 -
allocate_fs_swapmap 0 - - -
bufpages 26214 - 0- Pages -
create_fastlinks 0 - - -
dbc_max_pct 20 - - -
dbc_min_pct 5 - - -
default_disk_ir 0 - - -
dskless_node 0 - 0-1 -
eisa_io_estimate 768 - - -
eqmemsize 23 - - -
file_pad 10 - 0- -
fs_async 0 - 0-1 -
hpux_aes_override 0 - - -
maxdsiz 503866 - 0-503866 Pages -
maxdsiz_64bit 262144 - 1024-1073479679 Pages -
maxfiles 2048 - 30-2048 -
maxfiles_lim 2048 - 30-2048 -
maxssiz 2048 - 0-98048 Pages -
maxssiz_64bit 2048 - 4-262144 Pages -
maxswapchunks 2048 - 1-16384 -
maxtsiz 16384 - 0-503859 Pages -
maxtsiz_64bit 262144 - 1024-1073741823 Pages -
maxuprc 200 - 3- -
maxvgs 10 - - -
msgmap 39 - 3- -
nbuf 21216 - 0- -
ncallout 7235 - 6- -
ncdnode 150 - - -
ndilbuffers 30 - 1- -
netisr_priority -1 - -1-127 -
netmemmax 0 - - -
nfile 30010 - 14- -
nflocks 200 - 2- -
ninode 4796 - 14- -
no_lvm_disks 0 - - -
nproc 4116 - 10- -
npty 60 - 1- -
nstrpty 60 - - -
nswapdev 10 - 1-25 -
nswapfs 10 - 1-25 -
public_shlibs 1 - - -
remote_nfs_swap 0 - - -
rtsched_numpri 32 - - -
sema 0 - 0-1 -
semmap 63 - 4- -
shmem 0 - 0-1 -
shmmni 200 - 3-1024 -
streampipes 0 - 0- -
swapmem_on 1 - - -
swchunk 2048 - 2048-16384 kBytes -
timeslice 10 - -1-2147483648 Ticks -
unlockable_mem 16664 - 0- Pages -

I already run the HPjconfig from HP to tune the kernel.

Thks.

[Sameer_Nirmal]

Hi,

The load average values shown in top could be verified using
sar -q
Glance
uptime
w
# adb /stand/vmunix /dev/kmem
adb>avenrun/3F

There is a patch for load avarage
PHKL_33336.

Since the system responce is not normal, there is a resource problem for sure on the system. The attached report show swapping is being done.

I would do complete system check up using
sar , vmstat , iostat, netstat , Glance
syslog, demsg , swapinfo -tam, bdf etc.

For Java side, I would check the java processes using q4/p4 to know what they are doing and resources being used could be known as well. Verifying java specific kernel parameters such as maxssize,max-thread_proc, nkthread and ulimit.

[Bill Thorsteinson]

Given the number of transfer proceses
I suspect you are running into heavy
swapping. This will kill performance
in the manner you are experiencing.

Java garbage collection will cycle most
of the pages into memory each run. If
there are a lot of objects being created
and discarded this can occur every few
seconds. Starting transfer will smaller
memory limits could help.

Can you nice or stop most of the transfer
processes?

[Pedro Cirne]

Hi Bill,

The server is for running those java processes with username "transfer", that's what I want the server to do so no use to nice or stop them.

Thks,

Pedro

[Mark Nieuwboer]

Hi Pedro,

Here is a link wich can help to tune the os for java.
http://www.hp.com/products1/unix/java/infolibrary/prog_guide/java1/index.html.

grtz. Mark

[RAC_1]

Let me tell you one thing. The load average is not a very correct way of deciding how system is doing. The laod average is basically no. of jobs waiting in queue to run over a gievn period of time.

the important que global priority run queue.
You can check it with glance.
Prepare a file as follows. (call it glance.syntax"
print "global priority queue", gbl_pri_queue

Do you have a very high priority queue??

Also is system experiencing any kind of bottleneck?? -disk i/o, memory, swap, cpu??
disk i/o may show the behaviour you are experiencing now. Check global wait states in glance. glance -B, glance -i, iostat, sar -d

Also not to forget java patches.

[Eric Antunes]

Hi Pedro,

It seems that those java processes are eating lots of memory.

See "who" are they with the attached script (usage: mostra_procs_pesados.sh ).

It may be a bug...

Best Regards,

Eric Antunes

[Eric Antunes]

Hi Pedro,

It seems that those java processes are eating lots of memory.

See "who" are they with the attached script (usage: mostra_procs_pesados.sh ).

It may be a bug...

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi RAC,

Thank you for your reply, attached find sar/glance/iostat stats, would you have a look?

Thank you very much,

Pedro

[RAC_1]

Apart from memeory usage, all seems fine.
Post vmstat 1 5 also.

Check what processes are usig most of the memory. With java processes, you can control what heap size (initial/max) they can use. java --h for details. That you can control some java sizes.

UNIX95= ps -ef -o pid,ruser,vsz,args|sort -nrk3

[Pedro Cirne]

Hi RAC,

Attached find the vmstat statistics.

Thks,

PEdro

[Arunvijai_4]

Hi Petro, here is a link to a good guide "Performance tuning Javaâ ¢: kernel parameters affecting Javaâ ¢ performance"

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1616,00.html

(You should have DSPP membership in order to view this page, its free!)

-Arun

[Arunvijai_4]

One more link, Performance tuning Javaâ ¢: training & tutorials

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1607,00.html

-Arun

[RAC_1]

Java is a problem here. You most certainly have problems on account of java processes, who are eating a lot of memory and resulting in swapping.
Yes, you are swapping. Denoted by "po" column in vmstat. Also look at swapinfo -mat


Check how java processes are using the memory.

ps -efx | grep -i [j]ava
check particularly the the arguments passed to them. I am sure that you will see high figures. These options control how much memory they will grab. Unless you make them of reasonable value, your problem will not go away. (these values are max heap sizes, initial heap size and such things. Again java --h will give you details about what these options and values should be)

Also make sure you are up to date on java patches.

[Alzhy]

Pedro,

What you're experiencing is not "strange" - you simply are just running too many java processes that your system can handle. You should already have found out by this time that Java is "hog" -- CPU and memory (specially).

We have java 1.4 and 1.5 and the story is unchanged. java is still is a resource hog that you need to carefully plan your resources for.