topic Re: System error: [10061] Connection refused in Operating System - HP-UX

System error: [10061] Connection refused

Ali Bajunaid — Sun, 03 Aug 2003 07:51:15 GMT

Some time when the system (HP-UX) is very loaded and clients connecting from Windows to host thru TCP/IP socket they get this error:

System error: [10061] Connection refused

dbc_max_pct 2
ksi_send_max 256
max_async_ports 2000
max_thread_proc 1024
maxdsiz 0X80000000
maxdsiz_64bit 0XF0000000
maxfiles 2048
maxfiles_lim 4096
maxssiz 134217728
maxssiz_64bit 1073741824
maxswapchunks 16384
maxtsiz 0XF000000
maxtsiz_64bit 0X40000000
maxuprc (NPROC-8)
maxusers 1664
maxvgs 40
shmmax 4294967296
nproc 13332

tcp_largest_anon_port:

Largest anonymous port number to use. [1024, 65535]
Default: 65535

Can some one help me on this.

Thanks

Re: System error: [10061] Connection refused

Robert-Jan Goossens — Sun, 03 Aug 2003 07:58:03 GMT

Hi,

Loaded with users at those moments ?

Are you using NIS ?

Robert-Jan.

Re: System error: [10061] Connection refused

Ali Bajunaid — Sun, 03 Aug 2003 08:11:54 GMT

Yes, loaded whit users.

We are not using NIS

Thanks

Re: System error: [10061] Connection refused

Steven E. Protter — Sun, 03 Aug 2003 11:34:39 GMT

check out npproc and nfiles parameters in the kernel.

Run the attached script to collect data while this is going on.

check swapinfo -tam

The problem could be anywhere. If you have lots of users you need to know whether its the network, the cpu, the kernel or what thats doing this.

If its some users all the time, look additionally at settings in /var/adm/inetd.sec

If its telnet, look and see how many ptty devices you have set up.

Does it happen at high load factors or all the time.

If this post helps, please assign points.

SEP

Re: System error: [10061] Connection refused

Michael Steele_2 — Sun, 03 Aug 2003 13:28:58 GMT

1) This usually is a port issue so what is the client server application between HP-UX and Windows? Omniback? Network Node Manager? From this we can id the ports for you to check.

2) Verify DNS and WINS domain names and name resolutions.

Here is a Forum hit:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc0fa5dc05a7ad711abdc0090277a778c,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc8f8d211e18ad5118ff10090279cd0f9,00.html

3) From what I've gleaned from these threads and your problem it sounds like the network is too loaded and dropping packets. So check for dropped packets with netstat from both HP-UX and Windows.

netstat -m | grep -i -e dropped -e collision

Re: System error: [10061] Connection refused

Ron Kinner — Sun, 03 Aug 2003 17:59:17 GMT

I am at home now so can't check it but there is an ndd command - which controls the size of the new connection queue - which can be changed to help here. do
ndd -h |grep tcp
to see the list of ndd parameters. You can check
netstat -s
first - I believe there is a statistic there about the number of dropped new connection attempts - if it is building up during this time then you may be able to help it somewhat by lengthening the queue. Of course, if the CPU is overloaded or out of memory you might get the same problem. There is also another parameter in HPUX about the maximum number of files open at once. A TCP connection is considered an open file so you might be reaching this limit.

Ron

Re: System error: [10061] Connection refused

Ali Bajunaid — Mon, 04 Aug 2003 11:35:14 GMT

Please find the output

Re: System error: [10061] Connection refused

Michael Steele_2 — Mon, 04 Aug 2003 11:51:12 GMT

Hi Ali:

Something's not right here. Zero inode? This is not possible.

I'd suggest an immediate reboot to start with, and then patch updates. But you'll also have to reduce some kernel parameters:

ninode
nfile

You probably have some system memory fragmentation issues since you're only using 124/2020, or 6%.

15:06:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
15:06:16 N/A N/A 3387/13332 0 0/108704 0 12407/202038 0
15:06:17 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0
15:06:18 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0
15:06:19 N/A N/A 3386/13332 0 0/108704 0 12402/202038 0

############################################

Please attach these commands:

lvlnboot -v
pvdisplay -v /dev/dsk/c0t8d0 | head -50
pvdisplay -v /dev/dsk/c2t8d0 | head -60

This is also odd:

15:06:33 c0t8d0 100.00 0.50 4 14 0.81 533.74
c2t8d0 100.00 0.50 5 26 3.63 389.77

Re: System error: [10061] Connection refused

RolandH — Mon, 04 Aug 2003 12:22:55 GMT

See waht I have found in the technical knowledge db.

PROBLEM: After each update of a client from an HP-UX Installation Server, Omniback is changing the inet port in /etc/services for the Sun Solaris servers to 5550 instead of the default one 5555.

The following errors can be found in the session messages from the update:

[Warning] [110:103] Command execution on client failed!
remshd: Login incorrect.
[Normal] Update of client system marte-bp.inversis.com completed.

[Critical] [110:101] Client import failed!
[12:1625] Import host failed.
System error: [10061] Connection refused

The port 5550 was the default port in the previous Omniback installation.

CONFIGURATION
Operating System - HP-UX, Sun Solaris
Version - 3.10, 3.50, 4.0, 4.10
Product - omniback II

RESOLUTION: When an Inet port different than the default (5555) is being used in an Omniback cell, this value is stored in the /etc/opt/omni/customize/socket file.

After removing this file, Omniback started to use the default port (5555) in the installations with no problems.

For troubleshooting this type of problem, where the issue appears to come from a configurable parameter and could be taken from a file, the following unix command can be used:

#find /etc/opt/omni -name '*' -exec grep -l 5550 {} \;

In this case the output of the command returned the file: /etc/opt/omni/customize/socket

Re: System error: [10061] Connection refused

Ali Bajunaid — Mon, 04 Aug 2003 12:50:09 GMT

Our problem is not related to OMNI BACK, we are using an application running on WIN 2000 accessing HP-UNIX thru TCP/IP Socket

Re: System error: [10061] Connection refused

RolandH — Mon, 04 Aug 2003 12:59:36 GMT

Sorry Ali,

but can you do this for us

# ndd -get /dev/tcp tcp_largest_anon_port
# ndd -get /dev/tcp tcp_ip_abort_interval
# ndd -get /dev/tcp tcp_ip_abort_cinterval
# ndd -get /dev/tcp tcp_ip_notify_interval
# ndd -get /dev/tcp tcp_ip_notify_cinterval

results?

Roland

Re: System error: [10061] Connection refused

Ali Bajunaid — Mon, 04 Aug 2003 13:06:01 GMT

Results:

ndd -get /dev/tcp tcp_largest_anon_port
65535

ndd -get /dev/tcp tcp_ip_abort_interval
600000

ndd -get /dev/tcp tcp_ip_abort_cinterval
75000

ndd -get /dev/tcp tcp_ip_notify_interval
10000

ndd -get /dev/tcp tcp_ip_notify_cinterval
10000

Regards

Re: System error: [10061] Connection refused

RolandH — Mon, 04 Aug 2003 13:20:56 GMT

These ndd values are all default and seems to be OK for me.

But your nstrpty values seems a little bit low with a value of 60 if your clients connect with telnet. What is your npty (Default:60)value?
That is very low for a default value. If you have it already increased - have you increased the device files for that, too? This will not be done by default if your system reboots with a increased npty/nstrpty value. You must do it by yourself.

To check the number of telnet sessions you have available, on the box refusing
the connection do:

ls /dev/pty |wc -w
ls /dev/ptym | wc -w

1. Use SAM
Kernel
Configurable Parameters
npty
nstrpty
nstrtel (11.x only)
NOTE:
Set NPTY, NSTRPTY and NSTRTEL to 512, save and reboot. These are
the desired number of telnet sessions.

2. Rebuild the special device files at the command prompt as follows:

insf -d ptys -n 512 512 being the number you set NPTY in SAM
insf -d ptym -n 512
insf -d pts -s 512 -e -v (for 11.0 only )

Following the reboot, the maximum telnet sessions will now increase to 512.

HTH
Roland

Re: System error: [10061] Connection refused

Ali Bajunaid — Tue, 05 Aug 2003 03:51:06 GMT

The client application not using telnet, the way how it is work as following:

- At host we have aprogram listening on specific port (1335).

- Client Application establish a connection to that port number.

Please note this not Oracle listener.

Regards

Re: System error: [10061] Connection refused

U.SivaKumar_2 — Tue, 05 Aug 2003 04:26:12 GMT

Hi,

Provided you have enough system memory resources.

increase this parameters.

#ndd -set /dev/tcp tcp_conn_request_max 2000

Edit /etc/rc.config.d/nddconf to make the changes permanent even after system reboot.

Increasing maxfiles to 4096 and nfiles + 1000 is also recommended.

regards,

U.SivaKumar

Re: System error: [10061] Connection refused

Ali Bajunaid — Tue, 05 Aug 2003 04:45:52 GMT

Hi,

The current value for

tcp_conn_request_max = 4096

maxfiles 2048
maxfiles_lim 4096
nfile 202028 - (15*NPROC+2048)
ninode 108704 - (8*NPROC+2048)
nproc 13332 - (20+8*MAXUSERS)
maxusers 1664

Regards

Re: System error: [10061] Connection refused

Zigor Buruaga — Tue, 05 Aug 2003 06:16:40 GMT

Hi,

Maybe the program listening on port 1335 has some sort of parameter, such as max sessions or something like that.
When the problem occurs, how many sessions are established? and what happens if you kill an established client session and start again a previously failed connection?
Have you checked ( with GPM ) the use of kernel parameters when the problem occurs?

Only an idea, hope it helps.
Regards,
Zigor

Re: System error: [10061] Connection refused

Ali Bajunaid — Tue, 05 Aug 2003 06:30:05 GMT

Hi,

When the problem occurs, how many sessions are established?
- Around 3000 sessions.

and what happens if you kill an established client session and start again a previously failed connection?

- This can't controled.

Have you checked ( with GPM ) the use of kernel parameters when the problem occurs?

- No

Regards

Re: System error: [10061] Connection refused

Ali Bajunaid — Tue, 05 Aug 2003 06:38:58 GMT

Hi,

From GPM, we are getting allarm :

CRT : Network Bottleneck Probability= 100.00%

End of Network Bottleneck Alert

Regards

Re: System error: [10061] Connection refused

Ali Bajunaid — Tue, 05 Aug 2003 06:46:21 GMT

Hi,

From dmesg, we getting these messages:

0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.
0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.
0/0/14/0/0: Unable to access previously accessed device at nport ID 0x21255.

Regards