Process locking

[Helen French]

Hi,

I have two servers (K and D), running same application (JAVA). When the application runs on K class server, it is locking all other user processes and the application runs faster. But in D class, everything looks fine even though the application run slower. The Kernel parameters are the same, the top and sar outputs confirms that K class has more free resources !

Any idea what is happening?

Thanks!
Shiju

[Krishnan Viswanathan]

Is the RAM and virtual memory same on both K and D class ?

[A. Clay Stephenson]

There is just not enough data here to make any intelligent response.

1) Are the two boxes running the same versions of HP-UX and at the same patch level?

2) Are the Java versions/patches the same?

3) Are there other applications running on the K-box that are not running on the D-box?

4) Have you at least narrowed down the bottleneck? I/O, memory, CPU

5) Have you adjusted the priorities of any processes? You might try renicing the Java processes.

6) I would also compare the timeslice values on the two boxes.

7) Have you compared the run queues on both boxes using sar -q?

You probably need to install the Trial Version of Glance on your K-box; that will help you find the problem.

Regards, Clay

[Dan Carter_1]

Hi Shiju,

I've just gone through this myself. This was caused by a HotSpot issue. I'm assuming you're on HPUX 11.00 K Class multi processor mixed thread mode. The symptom is that the java process will take all CPU time and not release for other until it finishes.

As root, do "java -version"
Make sure on the k-class you're at least running java version "1.3.1.00-release

Example: Ours is ksh$ java -version
java version "1.3.1.00-release"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1.00-release-010719-
23:15-PA_RISC1.1)
Java HotSpot(TM) Server VM (build 1.3.1 1.3.1.00-release-010720-01:51-PA_RISC2.0
PA2.0, mixed mode)

Then you need to install a minimum battery of patches (listed below), tune the kernel using http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1602,00.html

Java Specific Patch List (as of 10/01 for HP 9000 KClass)
Patch bundle 9905 atexit shared library mutex
REQUIRED to support NFS over TCP/IP. Make sure it is
installed BEFORE you install PHNE_20344 and
PHNE_20371
PHCO_17556 fsck_vxfs(1M) Cumulative Patch
PHCO_17792 (Dependency for PHKL_18543)
PHCO_18229 libc cumulative header file patch
PHCO_19491 (Superceded by PHCO_27330 01April2001)
PHCO_19666 HP Java 1.3
PHCO_22269 (Dependency for PHKL_18543)
(Recalled by HP-Use PHCO_21845)
PHCO_23770 HP Java 1.3
PHCO_23561 (Dependency for PHKL_18543)
PHKL_14750 Fix pthread_cond_timedwait(3T) error
PHKL_17038 (Dependency for PHKL_18543)
PHKL_18543 HP Java 1.3
PHKL_19800 (Dependency for PHKL_18543)
PHKL_20016 (Dependency for PHKL_18543)
PHKL_20202 Fix pthread error return, nfs/tcp panic
PHKL_20079 (Dependency for PHKL_18543)
(Superceded by PHKL_24027 10May2001)
PHKL_21392 (VxFS performance, hang, icache DPFs)
PHKL_22145 signal, threads, spinlock, scheduler, IDS fix
PHKL_22266 s700_800 11.00 LVM Cumulative patch
PHKL_22432 (Dependency for PHKL_18543)
(Superceded by 24027)
PHKL_22562
PHKL_22589 (Dependency for PHKL_18543)
PHKL_23002 HP Java 1.3
PHKL_23406 (Dependency for PHKL_18543)
PHKL_24027 VxFS memory mapped truncating
PHKL_24116 probe,sysproc,PM,VM,PA-8700 cumulative patch
PHNE_19616 (Dependency for PHSS_19748)
(Superceded by PHNE_22086 15Sep2000)
PHNE_22086 (Dependency for PHSS_23517)
PHNE_20316 Cumulative STREAMS Patch
PHSS_14582 (Superceded by PHSS_16931 26Jan1999)
PHSS_15853 (Superceded by PHSS_22614 07Mar2001)
PHSS_16849 LIBCL patch
PHSS_17535 HP Java 1.3 AWT
PHSS_18066 C Preprocessor cumulative
PHSS_18110 (Superceded by PHSS_22809 09Mar2001)
PHSS_18430 (Dependency for PHSS_19748)
PHSS_18649 Assembler patch
PHSS_19178 (Superceded by PHSS_22217 21Aug2000)
PHSS_19593 (Superceded by PHSS_23823 17Apr2001)
PHSS_19178 s700_800 11.00 +O4/PBO compiler cumulative patch
PHSS_19748 (Superceded by PHSS_23797 03May2001)
PHSS_19866 (Superceded by PHSS_22514 27Feb2001)
PHSS_20544 (Superceded by PHSS_21046 29Mar2000)
PHSS_20863 (Dependency for PHSS_23517)
PHSS_21947 11.00 ld(1), dld.sl(5), and linker tools cumulative patch
PHSS_22614 HP Java 1.3 AWT
PHSS_22946 HP Java 1.3 AWT
(Dependency for PHSS_23517)
(Superceded by 23823 03MAY2001)
PHSS_22543 HP-UX SDK for the JavaTM 2 Platform applications and applets using GUIs
PHSS_23517 HP Java 1.3 AWT
(Superceded by PHSS_23797 03MAY2001)

-Dan Carter

[Helen French]

Hi,

Iyer: RAM and VM are more on K class.

Clay: Thanks for the response. Here are the answers for your questions,

1)K class is running on 10.20, D class with 11.0 (32 bit version). This is the only major difference with the systems.
2)The Java versions are:
K class - C.01.18.00
D class - C.01.18.01
Tha patches are up to date in both machines.
3) All the applications running on K box is same as D. One difference is K using a Netscape and D with iplanet.
4)Yes, I checked CPU, memeory and I/O. In D, these resources usage is very high (95% used). But in K box, i could see almost all resources are free ! Also K has 2 CPUs
5)I have not adjusted priorities of any process. Infact I was trying to do that but it is not possible, because the same process will be running by many users at the same time.
6)The timeslice value is same (10)
7)The run queues are almost same through SAM.
8) I have glance running on both and didn't help me with any more details.

Waiting for your reply ..thanks!


Dan: I am looking at the patches you are listed now, and will come back with details. Please note that your version and my java version are different. Does that make any changes ? Thanks for the link ..

Shiju

[Dan Carter_1]

Yes, although the patch dependencies begin with the same base, they do branch off to specific platforms, 32 v/s 64 bit processing and 10.x and 11.x dependencies. You have to be be very careful and maticulous about each of those, and then following the new patches that superceede the older patches.

-Dan

[Carl Erhorn]

Did you ever solve this? I wanted to point out that there can be major differences in the threading models used on 10.2 and 11.0. That may be the reason for very different behaviour on the systems.

I haven't run the 10.x systems for some years now, but I recall they used a totally different threading model in the JRE.

Hope this helps.

[Helen French]

Hi Carl,

Thanks for the reply.

No. The problem still exists. Unfortunately, this issue got a lower priority, as I was bound with some other issues and some application testing/upgrade was going on these systems. I checked all other possibilities except the OS upgrade and failed.

I will be doing the OS upgrade ( cold install 11.x) soon and let you know if that helps !

Thanks
Shiju

[Helen French]

Hi All,

I know it is too late, but thought of updating. This problem is resolved now, after the OS update ( from 10.20 to 11.0 ). Thanks to Clay, Dan and Carl.

Thanks for all your advices !

Shiju

[Dan Carter_1]

No problem, Glad to help.
RIght now I'm looking at the 1.3.1.05 release and finding out somje interesting things about that right now. Be careful on heap size assignments if you upgrade...-
Check out the release notes...
Dan