System error: [10061] Connection refused

[Ali Bajunaid]

Some time when the system (HP-UX) is very loaded and clients connecting from Windows to host thru TCP/IP socket they get this error:

System error: [10061] Connection refused

dbc_max_pct 2
ksi_send_max 256
max_async_ports 2000
max_thread_proc 1024
maxdsiz 0X80000000
maxdsiz_64bit 0XF0000000
maxfiles 2048
maxfiles_lim 4096
maxssiz 134217728
maxssiz_64bit 1073741824
maxswapchunks 16384
maxtsiz 0XF000000
maxtsiz_64bit 0X40000000
maxuprc (NPROC-8)
maxusers 1664
maxvgs 40
shmmax 4294967296
nproc 13332

tcp_largest_anon_port:

Largest anonymous port number to use. [1024, 65535]
Default: 65535

Can some one help me on this.

Thanks

[Robert-Jan Goossens]

Hi,

Loaded with users at those moments ?

Are you using NIS ?

Robert-Jan.

[Ali Bajunaid]

Yes, loaded whit users.

We are not using NIS

Thanks

[Steven E. Protter]

check out npproc and nfiles parameters in the kernel.

Run the attached script to collect data while this is going on.

check swapinfo -tam

The problem could be anywhere. If you have lots of users you need to know whether its the network, the cpu, the kernel or what thats doing this.

If its some users all the time, look additionally at settings in /var/adm/inetd.sec

If its telnet, look and see how many ptty devices you have set up.

Does it happen at high load factors or all the time.

If this post helps, please assign points.

SEP

[Michael Steele_2]

1) This usually is a port issue so what is the client server application between HP-UX and Windows? Omniback? Network Node Manager? From this we can id the ports for you to check.

2) Verify DNS and WINS domain names and name resolutions.

Here is a Forum hit:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc0fa5dc05a7ad711abdc0090277a778c,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc8f8d211e18ad5118ff10090279cd0f9,00.html

3) From what I've gleaned from these threads and your problem it sounds like the network is too loaded and dropping packets. So check for dropped packets with netstat from both HP-UX and Windows.

netstat -m | grep -i -e dropped -e collision

[Ron Kinner]

I am at home now so can't check it but there is an ndd command - which controls the size of the new connection queue - which can be changed to help here. do
ndd -h |grep tcp
to see the list of ndd parameters. You can check
netstat -s
first - I believe there is a statistic there about the number of dropped new connection attempts - if it is building up during this time then you may be able to help it somewhat by lengthening the queue. Of course, if the CPU is overloaded or out of memory you might get the same problem. There is also another parameter in HPUX about the maximum number of files open at once. A TCP connection is considered an open file so you might be reaching this limit.

Ron

[Ali Bajunaid]

Please find the output

[Michael Steele_2]

Hi Ali:

Something's not right here. Zero inode? This is not possible.

I'd suggest an immediate reboot to start with, and then patch updates. But you'll also have to reduce some kernel parameters:

ninode
nfile

You probably have some system memory fragmentation issues since you're only using 124/2020, or 6%.

15:06:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
15:06:16 N/A N/A 3387/13332 0 0/108704 0 12407/202038 0
15:06:17 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0
15:06:18 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0
15:06:19 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0

############################################

Please attach these commands:

lvlnboot -v
pvdisplay -v /dev/dsk/c0t8d0 | head -50
pvdisplay -v /dev/dsk/c2t8d0 | head -60

This is also odd:

15:06:33 c0t8d0 100.00 0.50 4 14 0.81 533.74
c2t8d0 100.00 0.50 5 26 3.63 389.77

[RolandH]

See waht I have found in the technical knowledge db.

PROBLEM: After each update of a client from an HP-UX Installation Server, Omniback is changing the inet port in /etc/services for the Sun Solaris servers to 5550 instead of the default one 5555.

The following errors can be found in the session messages from the update:

[Warning] [110:103] Command execution on client failed!
remshd: Login incorrect.
[Normal] Update of client system marte-bp.inversis.com completed.

[Critical] [110:101] Client import failed!
[12:1625] Import host failed.
System error: [10061] Connection refused

The port 5550 was the default port in the previous Omniback installation.


CONFIGURATION
Operating System - HP-UX, Sun Solaris
Version - 3.10, 3.50, 4.0, 4.10
Product - omniback II

RESOLUTION: When an Inet port different than the default (5555) is being used in an Omniback cell, this value is stored in the /etc/opt/omni/customize/socket file.

After removing this file, Omniback started to use the default port (5555) in the installations with no problems.

For troubleshooting this type of problem, where the issue appears to come from a configurable parameter and could be taken from a file, the following unix command can be used:

#find /etc/opt/omni -name '*' -exec grep -l 5550 {} \;

In this case the output of the command returned the file: /etc/opt/omni/customize/socket

[Ali Bajunaid]

Our problem is not related to OMNI BACK, we are using an application running on WIN 2000 accessing HP-UNIX thru TCP/IP Socket

[RolandH]

Sorry Ali,

but can you do this for us

# ndd -get /dev/tcp tcp_largest_anon_port
# ndd -get /dev/tcp tcp_ip_abort_interval
# ndd -get /dev/tcp tcp_ip_abort_cinterval
# ndd -get /dev/tcp tcp_ip_notify_interval
# ndd -get /dev/tcp tcp_ip_notify_cinterval

results?



Roland

[Ali Bajunaid]

Results:

ndd -get /dev/tcp tcp_largest_anon_port
65535

ndd -get /dev/tcp tcp_ip_abort_interval
600000

ndd -get /dev/tcp tcp_ip_abort_cinterval
75000

ndd -get /dev/tcp tcp_ip_notify_interval
10000

ndd -get /dev/tcp tcp_ip_notify_cinterval
10000


Regards

[RolandH]

These ndd values are all default and seems to be OK for me.

But your nstrpty values seems a little bit low with a value of 60 if your clients connect with telnet. What is your npty (Default:60)value?
That is very low for a default value. If you have it already increased - have you increased the device files for that, too? This will not be done by default if your system reboots with a increased npty/nstrpty value. You must do it by yourself.

To check the number of telnet sessions you have available, on the box refusing
the connection do:

ls /dev/pty |wc -w
ls /dev/ptym | wc -w

1. Use SAM
Kernel
Configurable Parameters
npty
nstrpty
nstrtel (11.x only)
NOTE:
Set NPTY, NSTRPTY and NSTRTEL to 512, save and reboot. These are
the desired number of telnet sessions.

2. Rebuild the special device files at the command prompt as follows:

insf -d ptys -n 512 512 being the number you set NPTY in SAM
insf -d ptym -n 512
insf -d pts -s 512 -e -v (for 11.0 only )

Following the reboot, the maximum telnet sessions will now increase to 512.

HTH
Roland

[Ali Bajunaid]

The client application not using telnet, the way how it is work as following:

- At host we have aprogram listening on specific port (1335).

- Client Application establish a connection to that port number.

Please note this not Oracle listener.

Regards

[U.SivaKumar_2]

Hi,

Provided you have enough system memory resources.

increase this parameters.

#ndd -set /dev/tcp tcp_conn_request_max 2000

Edit /etc/rc.config.d/nddconf to make the changes permanent even after system reboot.

Increasing maxfiles to 4096 and nfiles + 1000 is also recommended.

regards,

U.SivaKumar

[Ali Bajunaid]

Hi,

The current value for

tcp_conn_request_max = 4096

maxfiles 2048
maxfiles_lim 4096
nfile 202028 - (15*NPROC+2048)
ninode 108704 - (8*NPROC+2048)
nproc 13332 - (20+8*MAXUSERS)
maxusers 1664


Regards

[Zigor Buruaga]

Hi,

Maybe the program listening on port 1335 has some sort of parameter, such as max sessions or something like that.
When the problem occurs, how many sessions are established? and what happens if you kill an established client session and start again a previously failed connection?
Have you checked ( with GPM ) the use of kernel parameters when the problem occurs?

Only an idea, hope it helps.
Regards,
Zigor

[Ali Bajunaid]

Hi,

When the problem occurs, how many sessions are established?
- Around 3000 sessions.

and what happens if you kill an established client session and start again a previously failed connection?

- This can't controled.

Have you checked ( with GPM ) the use of kernel parameters when the problem occurs?

- No

Regards

[Ali Bajunaid]

Hi,

From GPM, we are getting allarm :

CRT : Network Bottleneck Probability= 100.00%

End of Network Bottleneck Alert


Regards

[Ali Bajunaid]

Hi,

From dmesg, we getting these messages:

0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.
0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.
0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.


Regards

[U.SivaKumar_2]

Hi,

Then set tcp_conn_request_max = 6000 reload your network application.

regards,

U.SivaKumar

[U.SivaKumar_2]

Hi,

Also set
/usr/sbin/ndd -set tcp_syn_rcvd_max 6000

Make necessary changes in /etc/rc.config.d/nddconf and reload ur network application.

regards,

U.SivaKumar

[Zigor Buruaga]

Hi,

About "dmesg" check this:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x28ec35067c18d6118ff40090279cd0f9,00.html

And about the connections problems check this:

"tcp_conn_request_max:

Maximum number of outstanding inbound connection requests.
[1, - ] Default: 20 connections

This is also known as the maximum depth of the "listen queue." The actual maximum for any given TCP endpoint in the LISTEN state will be
the MINIMUM of tcp_conn_request_max and the value the application passed-in to the listen() socket call.

For this parameter to take effect for a given application, it must be set BEFORE said application makes its call to listen(). So, if you use ndd to set this value after the application has started, it will have
no effect unless you can get the application to recreate its LISTEN endpoint(s).

You can see if tcp_conn_request_max MIGHT be too small by looking at the output of either "netstat -s" or "netstat -p tcp" and looking for the line displaying the number of connection requests dropped due to
full queue. If the number of drops is zero, the value of tcp_conn_request_max is fine. If the value is non-zero, either tcp_conn_request_max is too small, or the values the applications are using in their calls to listen() are too small.

Setting this value higher has no particular downside."

Regards,
Zigor

[Ali Bajunaid]

Hi,

Please find the output of nestat -p :

tcp:
1611103417 packets sent
1472820724 data packets (1057007467 bytes)
3331895 data packets (1722654504 bytes) retransmitted
138286252 ack-only packets (30983030 delayed)
88 URG only packets
2527 window probe packets
59281 window update packets
29424723 control packets
1789434185 packets received
1307838279 acks (for 824711219 bytes)
10203175 duplicate acks
5534 acks for unsent data
1490572308 packets (2431811244 bytes) received in-sequence
141 completely duplicate packets (159949 bytes)
1317 packets with some dup, data (1375081 bytes duped)
136054 out of order packets (175873447 bytes)
2833 packets (2269116429 bytes) of data after window
285 window probes
457188476 window update packets
55474 packets received after close
5646 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
657745 connection requests
12003603 connection accepts
12661348 connections established (including accepts)
14575766 connections closed (including 304184 drops)
5635 embryonic connections dropped
1293035916 segments updated rtt (of 1293035916 attempts)
2246452 retransmit timeouts
105784 connections dropped by rexmit timeout
2527 persist timeouts
1637996 keepalive timeouts
1636315 keepalive probes sent
187 connections dropped by keepalive
145 connect requests dropped due to full queue
1223887 connect requests dropped due to no listener

[Zigor Buruaga]

Hi,

"145 connect requests dropped due to full queue".
It seems you need to increase that parameter. I would follow the recomendations from U.Sivakumar.

Kind regards,
Zigor