TOMCAT on HP-UX B.11.11

[Marvin Dungao]

Hi, I just need a help with our application. Currently we are using a content management tool as a catalog application and the application server is TOMCAT, the TOMCATs OS is HP UX B.11.11, unfortunately when we do large loads TOMCAT goes down after an hour or 2, do you have any idea on how to solve the issue?

We already tried increasing the MAXFILES paramater from 2048 to 4096, unfortunately it stil goes down after loading large file to the content management environment THanks!

[skt_skt]

Do you see in any log file telling that it reached a limit 4096?there could be some thing wrong in app side too. Is it something new you are implementing or any recent changes.

[Marvin Dungao]

Hi Sir..Thank you so much for your reply.. previously we have an error "too many open files" and one of our 3rd level adviced to increase the number of maxfiles parameter in UNIX..that is why we set it up to 4096..

now the too many open file error was already eliminated..and still tomcat goes down after a few hours of loading a large file to the application, we are using Requisite's emerge for content management im not sure if you are familiar with the product

anyway here is sample of the log files hope you can help

[Dennis Handly]

>anyway here is sample of the log files hope you can help

Not that this is meaningful to me:
21 May 2007 13:11:26,762 DEBUG - Checking: Validate Picture URL=class com.requisite.ems.rules.RuleValidateURL
21 May 2007 13:11:26,767 DEBUG - Unable to connect to URL:http://bdhp4508.na.pg.com:8081/emerge/images/US_10009933/noimage-21.jpg
21 May 2007 13:11:26,767 FATAL - java.net.SocketException: Invalid argument (errno:22)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

[Steven E. Protter · Accepted solution]

Shalom,

Its true there may be a kernel file limitation at play here. Don't forget to increase maxuprc the maximum number of processes for a non-root user. That could be a problem.

It's worth making sure that the java patches recommended at http://www.hp.com/go/java are installed becasue this can cause problem as well.

I recommend maxfiles and nfiles and maxfiles_lim all be increased and leave yourself some margin of error. These parmeter work with one and other and they all must be looked at in this situation.

SEP

[Marvin Dungao]

Hi,

I checked our current kernel paramaters. Here is one below. I am not a UNIX person, pls bear with me. Can you recommend some values to the kernel paramters you have mentioned?

Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 8 - 8
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dnlc_hash_locks 512 - 512
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 31232 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 2304 - (MAXUSERS*3)
maxdsiz 384000000 - 384000000
maxdsiz_64bit 0x40000000 - 0X40000000
maxfiles 4096 - 4096
maxfiles_lim 2048 Y 2048
maxqueuetime - - 0
maxssiz 0x800000 - 0X800000
maxssiz_64bit 0x800000 - 0X800000
maxswapchunks 16384 - 16384
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 2048 Y 2048
maxusers 768 - 768
maxvgs 200 - 200
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 1026 - (2+MSGTQL)
msgmax 32768 Y 32768
msgmnb 32768 Y 32768
msgmni 50 - 50
msgseg 32750 - 32750
msgssz 8 - 8
msgtql 1024 - 1024
nbuf 0 - 0
ncallout 27392 - (2*((NPROC*7/4+16)*2))
ncdnode 150 - 150
nclist 12388 - (100+16*MAXUSERS)
ncsize 9960 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 8808 - (2*NPROC+1000)
nflocks 200 - 200
nhtbl_scale 0 - 0
ninode 4840 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 4608 - (2*MAX_THREAD_PROC)
nni - - 2
no_lvm_disks 0 - 0
nproc 3904 - (MAXUSERS*5+64)
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 7808 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 7808 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
select_enh 0 Y 0
sema 1 - 1
semaem 16384 - 16384
semmap 1538 - (SEMMNI+2)
semmni 1536 - 1536
semmns 1664 - 1664
semmnu 640 - 640
semmsl 2048 Y 2048
semume 30 - 30
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 268435456 Y 268435456
shmmni 512 - 512
shmseg 512 Y 512
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 1 - 1
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 65536 - 65536
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024

[Dennis Handly]

As SEP says, nfiles.

I also noticed that maxdsiz and maxdsiz_64bit are small and may need to be increased. You might as well do it with the same kernel regen.
You have maxdsiz 384 Mb and maxdsiz_64bit 1 Gb.

You may want to set maxdsiz to 512 Mb, 0x20000000.

[Marvin Dungao]

Hi Dennis and SEP

Thank you so much for the recommendation. What do you mean by kernel regen anyway, sorry as I am not familiar with the terms I only know basic unix.

Can I set the paramters in both Hex and Decimal values? again thanks in advance

[Dennis Handly]

>What do you mean by kernel regen anyway, sorry as I am not familiar with the terms I only know basic unix.

To change these values on 11.11, you have to create a new kernel and reboot. Sam can do that for you.

>Can I set the parameters in both Hex and Decimal values?

I believe so. One or the other. Use leading 0x for hex.

[Marvin Dungao]

Hi,

I am planning to change the following paramters today as you have recommended. I just have some questions

maxfiles 4096 - 4096
maxfiles_lim 2048 Y 2048
nfile 8808 - (2*NPROC+1000)
maxdsiz
maxdsiz_64bit
maxuprc

1.) All paramters as 2 values one current and one pending. Whats the difference? should they always be the same? whats the effect of it?

2.) in the nfile, i have read that this formula should be used to get the nfile

nfile=2(maxfiles_lim). My question is, should I set both current and pending to the result of the formula?

What are your recommended values to the other paramaters?

Thank you so much

[skt_skt]

So in order to increase the maxfiles , the nfile also need to be increased.Both are dependant

Find out the mem requierment from Oracle/App team and update following kernel parms which need a reboot. I had couple of suggetion to increase this values as it will define the stack size and data size respectively.

maxssiz
maxssiz_64
maxdsiz
maxdsiz_64

shmmax can be done dynamically later depending on requierment.

[Dennis Handly]

>1) All parameters as 2 values one current and one pending. What's the difference?

Pending is the value until you reboot, except for dynamic tunable. So you can only change the pending value.

>2) in the nfile, i have read that this formula should be used to get the nfile

No, you should use no formulas, just set it to a large value, 20000.

If you search for nfile in ITRC, you see many suggestions why.

[Marvin Dungao]

Hi First of all I would like to thank you all for the help you have given. The problem in our test system has been solved. How ever when I tried to put it in Production the same problem happens. Both QA and Prod has the same kernel parameter settings.

I had an error again of "TOO Many Open Files" Is it still possible to increase the MAXFILES? or do you have any other solution

The main difference between QA and Prod is that in QA HP UX is being used by 4 applications unlike in Prod where its only used by 1. Thank you so much

[Dennis Handly]

>I had an error again of "TOO Many Open Files" Is it still possible to increase the MAXFILES?

I would think that is nfiles.

[Marvin Dungao]

ic...so can I increase the Nfiles? currently as suggested the Nfiles was set to 20000 can I still increase it?

[Dennis Handly]

Oops, perhaps that was with maxfiles. See following link.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=838392

Nobody mentioned that maxfiles_lim must be >= maxfiles:
maxfiles 4096 - 4096
maxfiles_lim 2048 Y 2048

You can use: sar -v 2 20
to check out your nfile limit:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=382992

[Marvin Dungao]

HI,

Actually I changed the maxfiles and maxfiles_lim to have the same value 4096 since I saw in my reading that the value should be (as you have said) maxfiles_lim >= maxfiles.

I have also checked the usage of the nfile and its just around 10percent..

Would you know how can I check for the usage of the maxfiles?

[Dennis Handly]

>Would you know how can I check for the usage of the maxfiles?

There is glance of course. You must purchase it. There is a trial version on your CDs.

4096 is a lot of open files. I suppose you could just make it 8K.

[Marvin Dungao]

Thank you so much for this... is there another way to check on the usage of maxfiles like the way we checked on nfile using sar -v 2 20?

[Marvin Dungao]

I jsut checked.... i have a glance program installed!... do you know whats the syntax to use it? so that I check the current usage of the maxfiles?

[Dennis Handly]

>do you know whats the syntax to use it? so that I check the current usage of the maxfiles?

Well I would use gpm, for the graphical display. From the CPU window, you want a process report. Then you select the process and ask for open files. Then if you have more than 4K, you would have to laboriously count them up.

One of the reports is a system call, and it would have the count of opens. Then subtract the number of closes.