Error in syslog: Java out of memory

[Igor Klinaev]

Hello,
Please help to figure out the problem.

We have a TWS8.3 running with websphere on HP i64 11.23

Websphere app server hung at around 21:18 and the following messages appeared in syslog:

Feb 27 21:18:48 syslog: Java out of memory messages are marked with pid:8990 in /var/adm/syslog/syslog.log.
Feb 27 21:18:48 syslog: pid: 8990 4Kb 4Kb 4Kb PCT START/ 4K
b DEV NO./
Feb 27 21:18:48 syslog: pid: 8990 TYPE AVAIL USED FREE USED LIMIT RESERV
E PRI FILE NAME
Feb 27 21:18:48 syslog: pid: 8990 dev 1048576 0 1048576 0.00% 0
- 1 64 (major) 2 (minor)
Feb 27 21:18:48 syslog: pid: 8990 reserve - 2421209 -2421209
Feb 27 21:18:48 syslog: pid: 8990 memory 4177160 1258876 2918284 30.14%
Feb 27 21:18:48 syslog: pid: 8990 Dynamic libraries:
Feb 27 21:18:48 syslog: Java out of memory messages are marked with pid:8990 in /var/adm/syslog/syslog.log.
Feb 27 21:18:48 syslog: pid: 8990 4Kb 4Kb 4Kb PCT START/ 4K
b DEV NO./
Feb 27 21:18:48syslog: pid: 8990 TYPE AVAIL USED FREE USED LIMIT RESERV
E PRI FILE NAME
Feb 27 21:18:48 syslog: pid: 8990 dev 1048576 0 1048576 0.00% 0
- 1 64 (major) 2 (minor)
Feb 27 21:18:48 syslog: pid: 8990 reserve - 2421209 -2421209
Feb 27 21:18:48 syslog: pid: 8990 memory 4177160 1258876 2918284 30.14%
Thank you.

[TTr]

Igor,

How much memory does your server have?

How active is your websphere environment?

How many concurrent connections do you see on your server?

What kind of connections are they? Do many of them use JAVA? Java processes are known to use a lot of memory.

[Igor Klinaev]

Hi TTr,

How much memory does your server have?
Memory: 4617792K (2274684K) real, 7791600K (4448596K) virtual, 5345872K free

How active is your websphere environment?
used only for TWS

How many concurrent connections do you see on your server?
[What kind of connections are they? Do many of them use JAVA? Java processes are known to use a lot of memory
ps -ef|grep java
root 1873 1 0 Nov 18 ? 94:43 /opt/java1.3/jre/bin/../bin/IA64/native_threads/java -DRunnerValuePairFile=/var/opt/runner/data/valuePairs -cp /opt/runner/java
root 2291 2273 0 Nov 18 ? 212:02 /opt/IBM/SCM/client/../_jvm/jre/bin/../bin/IA64/native_threads/java -Xint -Xmx64m -Djlog.logCmdPort=1953 -Dsun.net.inetaddr.ttl
root 7482 1 0 10:43:52 ? 4:05 /u01/app/tws/twsadmin/appserver/java/bin/IA64W/java -XX:MaxPermSize=128m -Dwas.status.socket=60159 -Djava.net.preferIPv4Stack=f

[TTr]

It looks like you have at least 3 different JAVA environments here, one is the runner, the second is IBM-SCM and the third is TWS.

Can you tell which one ran out of memory in the syslog?

Also the syslog is complaining about what looks like swap space in
> Feb 27 21:18:48 syslog: pid: 8990 dev 1048576 0 1048576 0.00% 0 - 1 64 (major) 2 (minor)

Do you have glance installed? It shows a better picture of the memory status of the server, including virtual and swap.

Some things to check are
1. the size of the swap volume
2. the kernel parameters for dbc_pct_max and dbc_pct_min, by default they are set to 50% and 5%. If based on your statement above your server has 4.5GB physical memory the dbc_max_pct and min should be set to 4% and 2%.
3. Check with the JAVA that has the memory issue if there are any memory leaks in the code.

[Igor Klinaev]

1. the size of the swap volume
swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 16384000 317820 16066180 2% 0 - 0 /dev/vg00/lvol10
reserve - 8594688 -8594688
memory 16708640 4983892 11724748 30%

2. the kernel parameters for dbc_pct_max and dbc_pct_min,
dbc_min_pct 5 5 yes
dbc_max_pct 15 15 yes

[Igor Klinaev]

It was WebSphere App server (pid 8990) running in Tivoli Workload Scheduler TWS 8.3

Feb 27 21:18:48 syslog: pid: 8990 dev 1048576 0 1048576 0.00% 0 - 1 64 (major) 2 (minor)

[TTr]

It looks like you have enough resources, memory and swap.

I don't understand however why in your syslog, pid 8990 complains about the swap space and that swap space is shown to be only 1GB!?

> Feb 27 21:18:48 syslog: pid: 8990 dev 1048576 0 1048576 0.00% 0 - 1 64 (major) 2 (minor)

Any other messages in /var/adm/syslog/syslog.log at about the same time TWS hang?

It could also be a parameter setting in the TWS configuration. Unfortunately I am not familiar withe TWS.

Keep looking for more clues.