System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Intermittent no outgoing connections possible

Han Barnat
Occasional Advisor

Intermittent no outgoing connections possible

Hi,

We have an RP8420 with 4 vpars HP-UX 11v1, patched up to December 2008 plus some critical patches up to August 2009, with Oracle 10.2.0.4 and intermittently on one of the vpars, always in the afternoon on different times of the day, new outgoing connections can not be made.



We only see errors in the applicationlog, but while testing we could see the errors also with a ssh and telnet session from this vpar to other nodes. This error also occurs intermittent with a ssh localhost.

# ssh localhost

Can't assign requested address



There are no errors reported in /var/adm/syslog/syslog.log.



On a testsystem we can not reproduce the errors although system settings are identical.



Aug 25 host005.gps.ro.mj.nl java.net.BindException: bind() failed,

errno = 227

Aug 25 host005.gps.ro.mj.nl at java.net.PlainSocketImpl.socket

Bind(PlainSocketImpl.java)

Aug 25 host005.gps.ro.mj.nl at java.net.PlainSocketImpl.bind(P

lainSocketImpl.java)

Aug 25 host005.gps.ro.mj.nl at java.net.Socket.bind(Socket.java)

Aug 25 host005.gps.ro.mj.nl at java.net.Socket.(Socket.java)

Aug 25 host005.gps.ro.mj.nl at java.net.Socket.(Socket.java)

Aug 25 host005.gps.ro.mj.nl at HTTPClient.HTTPConnection.getSo

cket(HTTPConnection.java)

Aug 25 host005.gps.ro.mj.nl at HTTPClient.HTTPConnection.doCon

nect(HTTPConnection.java)

Aug 25 host005.gps.ro.mj.nl at HTTPClient.HTTPConnection.sendR

equest(HTTPConnection.java:3007)

Aug 25 host005.gps.ro.mj.nl at HTTPClient.HTTPConnection.handl

eRequest(HTTPConnection.java)



It seems this error is a result from the error above:

Aug 25 12:55:43 host005.gps.ro.mj.nl gps 2009-08-25 12:55:43,094 KOEGELBJ01

(1-69351315) ERROR hfa_hfapi_p.perform_login: error ORA-29260: network error: Connect failed because target host or object does not exist

Aug 25 12:55:43 host005.gps.ro.mj.nl gps 2009-08-25 12:55:43,095 KOEGELBJ01

(1-69351315) ERROR hfa_hfapi_p.handle_network_error: network error ->DO disconnect



Also this one:

Aug 25 host005.gps.ro.mj.nl java.net.SocketException: Can't assign requested address



Several searches on the internet and ITRC pointed us in the direction of a lack of sockets but we can not determine if that is truly the case.

Netstat -an | wc -l never shows more than 4200 sockets in use.



A sar -v at the time of the error shows:

14:48:00 N/A N/A 1293/4096 0 2361/34816 0 38365/63498 0



CPU and memory usage is below 50%.



host005:/var/adm/sa# netstat -ian

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll

lo0 4136 127.0.0.0 127.0.0.1 3769745 0 3769745 0 0

lan13:1 1500 10.32.20.0 10.32.20.22 80 0 0 0 0

lan15 1500 10.32.4.192 10.32.4.195 2 0 1 0 0

lan13 1500 10.32.20.0 10.32.20.20 63473125 0 120101929 0 0

lan16 1500 10.32.5.224 10.32.5.229 1 0 1 0 0



host005:/var/adm/sa# netstat -rn

Routing tables

Destination Gateway Flags Refs Interface Pmtu

127.0.0.1 127.0.0.1 UH 0 lo0 4136

10.32.20.20 10.32.20.20 UH 0 lan13 4136

10.32.20.22 10.32.20.22 UH 0 lan13:1 4136

10.32.5.229 10.32.5.229 UH 0 lan16 4136

10.32.4.195 10.32.4.195 UH 0 lan15 4136

10.32.4.192 10.32.4.195 U 2 lan15 1500

10.32.5.224 10.32.5.229 U 2 lan16 1500

10.32.20.0 10.32.20.20 U 3 lan13 1500

10.32.20.0 10.32.20.22 U 3 lan13:1 1500

127.0.0.0 127.0.0.1 U 0 lo0 4136

default 10.32.20.1 UG 0 lan13 1500



All suggestions are very welcome and will be rewarded.
9 REPLIES
Turgay Cavdar
Honored Contributor

Re: Intermittent no outgoing connections possible

>>intermittently on one of the vpars, always in the afternoon on different times of the day, new outgoing connections can not be made.

If outgoing connections cant be done there may be a possibility of lack of anonymous ports on the system. The defaults are 49152 through 65535. You can try to set it to lower values and give a try:
ndd -set /dev/tcp tcp_smallest_anon_port ...

>>Netstat -an | wc -l never shows more than 4200 sockets in use.

Did you take this value at the time when problem occurs?
Han Barnat
Occasional Advisor

Re: Intermittent no outgoing connections possible

Thanks for your response.

The settings are:
ndd -get /dev/tcp tcp_smallest_anon_port
49152

Turgay Cavdar
Honored Contributor

Re: Intermittent no outgoing connections possible

Cant get the answer:
>>Netstat -an | wc -l never shows more than 4200 sockets in use.

Did you take this value at the time when problem occurs?
Han Barnat
Occasional Advisor

Re: Intermittent no outgoing connections possible

Yes!

Re: Intermittent no outgoing connections possible

Hi,

I'm working with Han on this issue.

Anybody else with suggestions?

Also we would like to know how to determine the number of sockets in use.

netstat -an | wc -l
or /usr/contrib/bin/lsof /dev/ip | wc -l
or ?

We know we can expand the range 49152-65535 with lowering tcp_smallest_anon_port but do we need that?

Regards,
Roland
Turgay Cavdar
Honored Contributor

Re: Intermittent no outgoing connections possible

If you have more than 16383 outgoing connection then you have to tune the value. When the problem occurs just check the number of outgoing connections. Get the netstat -na output and load them to excel and remove incoming connections/LISTEN entries and calculate the connections(conncetions of all states:ESTABLISHED,CLOSE_WAIT,TIME_WAIT etc..)
Han Barnat
Occasional Advisor

Re: Intermittent no outgoing connections possible

Netstat does not show the "BOUND" status.

HP's tool "xportshow" does show BOUND states.

We are currently monitoring the use of the ports and calculating the totals.
Here's an example.
The amount of BOUND state session is rapidly increasing over time. The peak is expected in about 2 hours from now (11:00am).

time total establ listen finwt2 timewt dynamc total dynamc client tcpudp dynamc bound bound total
08:58:52 1717 1195 77 6 438 759 557 244 734 2274 1003 3242 0 5516
08:59:28 1777 1233 77 6 460 815 556 243 735 2333 1058 3225 0 5558
09:00:05 1809 1279 77 5 447 857 562 246 737 2371 1103 3233 0 5604
09:00:41 1783 1291 77 5 409 808 562 247 741 2345 1055 3265 0 5610
Laurent Menase
Honored Contributor

Re: Intermittent no outgoing connections possible

typically this should be raised to hpux support.
- do you use XTI connections?
- do you use access right send over unix sockets?

Han Barnat
Occasional Advisor

Re: Intermittent no outgoing connections possible

The problem of the bound status ports is caused by a Microsoft Windows IIS server. This server has a keepalive setting which causes the http connections to be open. After a timeout (120s) IIS will close the inactive connections. The database server will see this as open sockets. (status STSO).
The solution is to switch off the keepalive option on the IIS server.
The amount of tcp bound status ports is now only two instead of thousands.