jar processes consumsing 16GB memory hpux 11.31

[chindi]

Hi,

We have rx2660 HPUX 11.31  which has 16GB RAM & 4core CPU ,

When checked through glance it shows 100% physical memory used.

root #/tmp >./kmeminfo
tool: kmeminfo 10.56 - libp4 9.845 - libhpux 1.494 - HP CONFIDENTIAL
unix: HP-UX B.11.31 64bit Montvale 9140M on host "dcwbinf1"
core: /dev/kmem live kernel!
link: Fri Nov 07 17:08:54 IST 2014
boot: Thu Nov 27 13:01:59 2014
time: Tue Feb 16 10:32:22 2016
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4188971 16.0g 100%
Free memory = 38576 150.7m 1%
User processes = 2129316 8.1g 51% details with -user
Detached SHMEM = 1 4.0k 0% details with -shmem
System = 2021079 7.7g 48%
Kernel = 1867754 7.1g 45% kernel text and data
Dynamic Arenas = 1318333 5.0g 31% details with -arena
UFS_MISC_ARENA = 371722 1.4g 9%
spinlock_arena = 125822 491.5m 3%
vx_inode_kmcach = 110308 430.9m 3%
reg_fixed_arena = 107185 418.7m 3%
vx_global_kmcac = 91234 356.4m 2%
Other arenas = 512062 2.0g 12% details with -arena
Super page pool = 158118 617.6m 4% details with -kas
Emergency pool = 8754 34.2m 0% system critical reserve
Unused Text ITR = 7013 27.4m 0%
PDK malloc pool = 17304 67.6m 0% Free PDK malloc pool
Arena Lobj HDRs = 10267 40.1m 0% Arena large object headers
UAREA's = -81632 16777216.0t 1610612736% For active and cached threads
Overflow pte's = 33011 128.9m 1%
Static Tables = 51230 200.1m 1% details with -static
vhpt = 32768 128.0m 1%
text = 9370 36.6m 0% vmunix text section
bss = 6807 26.6m 0% vmunix bss section
data = 2124 8.3m 0% vmunix data section
svhpt = 64 256.0k 0%
Other tables = 95 380.0k 0% details with -static
Pfdat = 204804 800.0m 5%
Other kernel = 140552 549.0m 3% Other kernel data structures
File Cache = 153325 598.9m 4% details with -ufc

root #/tmp >ps -ef |grep -i java |wc -l
40

We have Jar processes running count is 40

[Steven Schweda]

> We have Jar processes running count is 40

   Perhaps I'm missing something obvious, but where is your evidence
that these "Jar processes" are the ones which are using too much memory?
Or are those the only user processes running on your system?

   I'm more of a Zip+UnZip guy, so I know nothing, but I would not
expect "jar" to use much memory.

[chindi]

Hi Steven,

root #/tmp >UNIX95= ps -eo "vsz,user,pcpu,args" | sort -rn |pg
1915456 tomcat 0.01 /opt/java6/bin/IA64N/java -Djava.util.logging.config.file=/opt/hpws22/tomcat/conf/logging.properties -Dfile.encoding=UTF-8 -Xms
1909192 impsweb 0.06 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar KVG_Server_6011.jar
1749448 impsweb 0.16 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar GSC_Server_6000.jar
1565124 impsweb 0.05 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar UKG_Server_6021.jar
1556936 impsweb 0.05 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar SRG_Server_6022.jar
1556932 impsweb 0.04 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar A_Server_6001.jar
1520068 impsweb 0.08 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Ae_Server_6012.jar
1511880 impsweb 0.09 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Me_Server_6024.jar
1484124 impsweb 4.11 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar /imps/Gateway/Gateway.jar
1438152 impsweb 0.10 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Ce_Server_6025.jar
1413572 impsweb 0.08 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Ue_Server_6030.jar
1411896 tomcat 3.12 /opt/java6/bin/IA64N/java -jar NACH_CBS_SAL_DOWNLOAD.jar
1401284 impsweb 0.07 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Ae_Server_6029.jar
1362564 impsweb 0.11 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Mer_Server_6026.jar
1362552 tomcat 0.30 /opt/java6/bin/IA64N/java -jar NACH_2.0_CBS_ACHCR.jar
1352132 impsweb 0.08 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Pe_Server_6023.jar
1346180 impsweb 0.12 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Te_Server_6028.jar
1337988 impsweb 0.11 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar Mer_Server_6027.jar
1337972 tomcat 0.25 /opt/java6/bin/IA64N/java -jar NACH_2.0_ECS_DR.jar
1336388 yesprod 0.87 /opt/java6/bin/IA64N/java -jar fantailp.jar agentconfig.xml
1335012 infdmz 1.60 /opt/java6/bin/IA64N/java -jar AV_Option2.jar
1333876 tomcat 0.35 /opt/java6/bin/IA64N/java -jar NACH_2.0_ACH_DR.jar
1330916 infdmz 1.92 /opt/java6/bin/IA64N/java -jar AV_Option1.jar
1330912 infdmz 1.62 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_DownloadUpload.jar
1330400 infdmz 0.30 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_DownloadReport.jar
1330400 infdmz 0.20 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_ProcessAadhar.jar
1330396 infdmz 0.11 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_DownloadReport.jar
1329740 infdmz 0.18 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_ProcessAck.jar
1326816 infdmz 1.86 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_DownloadUpload.jar
1325684 tomcat 0.24 /opt/java6/bin/IA64N/java -jar NACH_2.0_ECS.jar
1325644 infdmz 0.47 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_ProcessAck.jar
1308136 infdmz 1.86 /opt/java6/bin/IA64N/java -jar /infsftpdmz/AXIS-CH/Jar/AxisReverseFeed.jar
1306912 infdmz 0.04 /opt/java6/bin/IA64N/java -jar /infsftpdmz/AXIS-CH/Jar/AxisCHOutward.jar
1305204 tomcat 0.10 /opt/java6/bin/IA64N/java -jar NACH_2.0_CBS_DBTL_option1.jar
1301112 tomcat 0.35 /opt/java6/bin/IA64N/java -jar NACH_2.0_CBS_DBTL_option2.jar
1298856 microweb 0.00 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar ADCB_Web.jar
1280584 infdmz 0.01 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_GenerateReturn.jar
1277148 infdmz 0.02 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_ProcessInward.jar
1277148 infdmz 0.00 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_ProcessInward.jar
1276492 infdmz 0.01 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/APB-CR_Jars/APB_EncryptDecrypt.jar
1276488 infdmz 0.05 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_EncryptDecrypt.jar
1276488 infdmz 0.00 /opt/java6/bin/IA64N/java -jar /infsftpdmz/APBSH2H/Jars/ACH-CR_Jars/ACH_GenerateReturn.jar

[Dennis Handly]

>We have java processes running count is 40

So what's the problem?  These java processes will just use as much as they can before garbage collecting.

[chindi]

Problem is Physical memory is almost 100% used and its swapping .

Got fork error few days back.

HP Action Plan:

===========

1. Please suggest the customer to tune the following kernel parameter.

1. a) kctune -h vxfs_ifree_timelag=-1
2. b) kctune -h vx_ninode=30000  Its online , so was thinking of applying it .

Just wanted to take opinion from you guys.

[Dennis Handly]

>HPE Action Plan:

This may help on the kernel's ~7.7 GB usage.

[chindi]

Above settings Didn't help.

[Steven Schweda]

> We have Jar processes running count is 40

> [...]
> 1909192 impsweb 0.06 /opt/java6/bin/IA64N/java -Dfile.encoding=ISO-8859-1 -jar KVG_Server_6011.jar
> [...]

   Those are Java processes, not "jar" processes.  Who wrote them?  And
what, exactly, are they doing (which wants all that memory)?

[chindi]

Hi Steven,

Have asked Application team to check it.