UDP connections increase continually

[yuyuguang]

Hi All,

Our business is running two jobs which recv and send messages via web service, but after some days we have error ora-12540:
internal limit restriction exceeded.

We found the udp connections will not be released, at the beginning we thought maybe our business cause this problem, but udp connections was not increased when we moved two jobs in a new oracle which is installed on windows.

We trace the udp connections on router, but found nothing, that means udp are from local to localhost.

[yuyuguang]

training#[/]date
Mon Nov 1 17:23:31 EAT 2010
training#[/]netstat -a | wc -l
1337
training#[/]netstat -a | wc -l
1335
training#[/]netstat -a | grep tcp | wc -l
120
training#[/]netstat -a | grep udp | wc -l
1096


training#[/]date
Mon Nov 1 17:54:56 EAT 2010
training#[/]netstat -a | grep udp |wc -l
1125
training#[/]netstat -a | wc -l
1365
training#[/]netstat -a | grep tcp | wc -l
123


training:/cosys06/cosys/edi>date
Tue Nov 2 14:25:55 EAT 2010
training:/cosys06/cosys/edi>
training:/cosys06/cosys/edi>netstat -a | grep udp |wc -l
2209
training:/cosys06/cosys/edi>netstat -a | wc -l
2409
training:/cosys06/cosys/edi>netstat -a | grep tcp | wc -l
92

training#[/]date
Tue Nov 2 15:45:41 EAT 2010
training#[/]
training#[/]netstat -a | grep udp |wc -l
2235
training#[/]netstat -a | wc -l
2427
training#[/]netstat -a | grep tcp | wc -l
87



training:/cosys06/cosys/edi>date

Tue Nov 2 16:55:24 EAT 2010

training:/cosys06/cosys/edi>netstat -a | wc -l

2496

training:/cosys06/cosys/edi>netstat -a | grep udp | wc -l

2310

training:/cosys06/cosys/edi>netstat -a | grep tcp | wc -l

83

[yuyuguang]

from yesterday we stopped two jobs on UNIX. udp connections are not increased and not released.

training#[/tmp/hp]model
9000/800/rp3410
training#[/tmp/hp]uname -a
HP-UX training B.11.11 U 9000/800 3112192028 unlimited-user license


attched file is log of command "nettl -tn all -e ns_ls_udp -f /var/adm/nettl", you can open it with wireshark

[yuyuguang]

Oracle Version
1 Oracle9i Enterprise Edition Release 9.2.0.1.0 - 64bit Production
2 PL/SQL Release 9.2.0.1.0 - Production
3 CORE 9.2.0.1.0 Production
4 TNS for HPUX: Version 9.2.0.1.0 - Production
5 NLSRTL Version 9.2.0.1.0 - Production

Attached file is the readable file of nettl log

[S. Ney]

Hello what is the output of:
ndd -get /dev/ip ip_ire_gw_probe
swapinfo -tam
kctune -v nproc
kctune -v nfile
kctune -v maxfiles
kctune -v maxuprc
as oracle ulimit -a

Here are some possible relevant threads for you:
http://www.freelists.org/post/oracle-l/ORA12540-TNSinternal-limit-restriction-exceeded,3

http://www.dba-village.com/village/dvp_forum.OpenThread?ThreadIdA=12745

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1118793

[yuyuguang]

Hi S.Ney

maxusers = 250
maxuprc = 651
nproc = 814
nfile = 14258
maxfiles = 2048
semmni = 4070
semmns = 8140
semmsl = 2048

training#[/]swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 14 2034 1% 0 - 1 /dev/vg00/lvol2
dev 3072 15 3057 0% 0 - 1 /dev/vg00/pswap
reserve - 2257 -2257
memory 1578 454 1124 29%
total 6698 2740 3958 41% - 0 -


training#[/]ndd -get /dev/ip ip_ire_gw_probe
1


SQL> select name, value from v$parameter where name like '%processes%';

NAME VALUE
---------------------------------------------------------------- --------------------------------------------------------------------------------
processes 500
db_writer_processes 1
log_archive_max_processes 2
job_queue_processes 15
aq_tm_processes 1

SQL> SELECT resource_name, limit_value FROM v$resource_limit;

RESOURCE_NAME LIMIT_VALUE
------------------------------ --------------------
processes 500
sessions 555
enqueue_locks 6780
enqueue_resources UNLIMITED
ges_procs 0
ges_ress UNLIMITED
ges_locks UNLIMITED
ges_cache_ress UNLIMITED
ges_reg_msgs UNLIMITED
ges_big_msgs UNLIMITED
ges_rsv_msgs 0
gcs_resources 34936
gcs_shadows 34936
dml_locks UNLIMITED
temporary_table_locks UNLIMITED
transactions UNLIMITED
branches UNLIMITED
cmtcallbk UNLIMITED
sort_segment_locks UNLIMITED
max_rollback_segments 123

RESOURCE_NAME LIMIT_VALUE
------------------------------ --------------------
max_shared_servers 20
parallel_max_servers 6

22 rows selected

[S. Ney]

I am not an expert in kernel parameters however the threads I've linked in my previous post indicate that you could adjusting some of the posted parameters. Hopefully someone with more experience tuning will be able to provide more insight.

Another thought is your dead gateway detection. We have our servers set to zero due to network requirements.

Lastly here is a thread with the same OS and oracle level as yours.
http://www.freelists.org/post/oracle-l/ORA12540-TNSinternal-limit-restriction-exceeded,2