High cpu util and sleep.

[Steven M Evans]

Hello All,

I have a java process that generally takes 40-110% CPU, yet glance reports that it is sleeping 96% in the event column. This is a problem, since performance tanks with a load over 3 (which happens frequently). It's a home-brewed app with Sybase running on the same rp8400. How it have both high CPU and sleep?

Glance output:
Wait States PID: 18020, java PPID: 18013 euid: 270 User: titan

Event % Blocked On %
--------------------------------------------------------------------------------
IPC : 0.0 Cache : 0.0 CPU Util : 61.0
Job Control: 0.0 CDROM IO : 0.0 Wait Reason: SLEEP
Message : 0.0 Disk IO : 0.0
Pipe : 2.4 Graphics : 0.0
RPC : 0.0 Inode : 0.0
Semaphore : 0.0 IO : 0.0
Sleep : 96.0 LAN : 0.0
Socket : 0.0 NFS : 0.0
Stream : 1.2 Priority : 0.1
Terminal : 0.0 System : 0.0
Other : 0.0 Virtual Mem: 0.0


TIA,
Steve Evans

[Mel Burslan]

Do not take this as a gospel but in my sysadmin life I have seen my fair share of badly written homebrew java applications, taking servers down to its knees.

Make sure that you have a fairly recent pthreads patch on your system. You can find it using :

swlist -l product | grep -i pthread

and research this patch on the ITRC patch database and see if it is proceeded by a better patch and if it had any critical warnings on it. If so, upgrade at your earliest convenience.

Again, from the previous experience, java processes, especially badly written ones, do not clean up after themselves, causing memory leaks, which can tank your performance in no time.

Unfortunately, there is no surefire way (at least for me) to tell you like "do this and do that and kill the java programmer while you are at it". Mine is just collection of anectodal evidence and hope it helps.

[Steven E. Protter]

Shalom Steve,

Your java process is waking up a lot.

Kind of like me (2:20 a.m.).

There are a couple of possible solutions:

1) OS patches for java. It will install without these but these problems can happen.
2) http://www.hp.com/go/java . Geta newer version of java, if your application and database support it, this can really help.
3) Bad developer code. That has to be dealt with by identifying the code and that always means sysadmin work 'cause programmer just copy code, they don't work ;-)

SEP

[Michael Steele_2]

Processes go to sleep and are swapped out from the run queue when they are waiting for resources. No CPU usage is consumed when out of the run queue and sleeping. When the resouce frees up, the process goes back into the run queue to use CPU time. It might wait in the run queue for 20 nanoseconds or whatever, but waiting in the run queue is still better then sleeping outside of it. So the question is, what resource is bottlenecked? CPU for sure, and if so then you've probably got some terribly looping code nearing infinity and needs to be re-written. What are your disks and other resources doing while this is happening?

sar -u 5 5
vmstat 5 5
swapinfo tam
sar -d 5 5
netstat

[Steven E. Protter]

Information collector:

http://www.hpux.ws/system.perf.sh

SEP

[Steven M Evans]

Thanks for the responses, gents.

Mel - you may get a 10 just for anecdotal evidence of killing your java programmers. pthread is the latest.

$ swlist -l product | grep -i pthread
PHCO_27633 1.0 Pthread.h fix and new enhancement
PHCO_30544 1.0 Pthread enhancement and fixes

Steven - go back to sleep, this problem isn't worth you waking up. Hope you're not close to the border. Java is fairly well updated, I'm 0.0.0.02 behind.

Jdk14 1.4.2.08.02 Java2 1.4 SDK
Jre14 1.4.2.08.02 Java2 1.4 RTE

Michael - I'm with ya on the 'terribly looping code'. I'd like to be able to say, "Fix ______." I'll be doing Mr. Protter's perf.sh later this afternoon. Plus I'll send you the MeasureWare reports I stare at.

More info on the system:
16-way w/ 48gb memory. Sybase 12.5 w/ 6 engines - 5tb db. 100tb Centera, 30tb of Symmetrix. (sar -d will be huge!) Using PRM.

The Centera has a perlfs gateway and is for data - Monday we had a failed node, which hung operation. This is being fixed in the cabinet and the code. Typically not an issue unless broken. EMCs powermt against Symmetrix shows Q-IOs < 10, unless we are doing backups. Memory usage is static; disk cache in 1-2% range and hit 98%.

The reason I'm asking about the java is this: Sybase is responding internally very well. But when load goes up, it's like we can't get responses out of Sybase. Doing and sp_who gives internal timing of 0 or 1 sec. time shows 3-5 seconds. Yesterday was freaky because we had 90 root zombies hanging around taking CPU. Because they are root, PRM treats them extra special and skews resources to other things (as far as I can tell). That wasn't part of the java bit, and has been fixed. But it really got me wondering about why a java process would be taking so much cpu while sleeping and how to get it to cooperate.

So much for the book for now. I'll send you more heaping mounds of data later (probably tommorrow).

[Steven M Evans]

Output from system info attached sans disk info that put the compressed file over 1MB .

More staring at the process and it's 200 threads and the system calls revealed it's spending 90% of it's time in ksleep. According to this article - http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1618,00.html#CPU - it seems there is some lock contention going on. There were patches for 11.0 systems, which don't seem to apply to 11.11.

Any comments on this or from the system info provided?

[Steven M Evans]

Well,

I've sent info over to the developers, along with tusc info to see what they can do...

[Steven M Evans]

.

[Dennis Handly]

Have you looked at the java web pages for performance tools?
http://www.hp.com/go/java
o HPjtune
o HPjmeter
o HPjconfig