ERROR: Error assigning connection request, can't create any more handlers

[Pradeep Sadanapalli]

We faced application connectivity issues with the application server recently and when I checke dthe log files, it showed the below error messages

" OLT_CAT:1008: "ERROR: Could not establish listening address on network 0x0002233286306318"
JOLT_CAT:1083: "ERROR: Error assigning connection request, can't create any more handlers" "

Looking up these errors they are the result of one of the following (From BEA's support site):

"The problem could be that the number of processes allowable by a user (ulimit) has been reached. If this is the case, consult with your System Administrator to increase this limit."
or
"The system is out of network addresses, check with the system administrator to increase the number of addresses to use."

On that particular day, the application load was high due to more users accessing the application.

I checked the kernel parameters - maxuprc(value=2048), maxfiles(value=maxfiles), maxfiles_lim(value=20480). We are very sure that these limits are not hit. Also I checked #netstat -p tcp and I see :

0 connect requests dropped due to full queue
12738 connect requests dropped due to no listener

I am not able to figure out what else could result in this connectivity issue. Are there any other parameters that cause the limitation on number of connections? Please help.

Our application server runs on HP-UX11i v1

[Steven E. Protter]

nfile
nproc

Both need to be checked.

maxfiles is not equal to maxfiles. Please re-check.

dmesg

vi /var/adm/syslog/syslog.log

Look for clues.

SEP

[Pradeep Sadanapalli]

I am sorry. The values are as below.

maxfiles=4096
nfile=29249
nproc=16020

These values look to be sufficient enough. Any more suggestions please.

[Victor BERRIDGE]

Hi,

the message "12738 connect requests dropped due to no listener" is maybe your issue, is there a oracle DB behind? can you stop restart the listener, look at the listener logs...
did you have some issues with your network (netstat -a shows loads of TIMEWAIT etc...)?


All the best
Victor

[Pradeep Sadanapalli]

Yes, we have an oracle DB behind the app server. When I checked for netstat -a|grep TIME_WAIT , I got around 50. I am not able to restart the listner at this moment, but do you think this error might be due to a limitation at the Oracle listener end?

[Victor BERRIDGE]

Not limitations, but listener gone bezerk...
I have had this happening quite often during a period, and only way to bypass was a restart once a day by cron at 2AM then it disappeared, I believe it was a bug in an application module , that was corrected later (?) although questioning the developpers led to nothing (no, nobody touched the box, or updated the application...)


All the best
Victor

[Pradeep Sadanapalli]

The error messages eventually stopped
and the application started working fine
after few hours without admin intervention.

So I guess the oracle listener is fine. This
connectivity issues were faced only during
the load time. But after that everything was
working fine. So I believe this has something
to do with limitations, as we faced this error
only during heavy traffic.

[Victor BERRIDGE]

Other possibilities:
Run out of sockets (lots of old not cleaned...)
/var/tmp was full at a moment...


All the best
Victor