TCP Port becomes unusable

[Randy W. Suhrbier]

OpenVMS Alpha 7.3-2
TCPIP 5.4 ECO5

Problem:
Every few months a process appears to be unable to open its predefined listening port. We reboot to solve the problem, we have not yet tried just restarting TCPIP.

Situation:
We have a weekly data deployment cycle that results in all the vendor's application processes being restarted. The vendor's environment includes a library around TCPIP, part of which supports opening listening ports. If the library is unable to open a listening port it will go into a loop, making up to 24 attempts, once every 5 seconds. After the last attempt fails the process aborts. Each atempt is logged to SYS$OUTPUT. We do not have access to the vendor's code, but debug output appears to indicate the looping only occurs if the return status is SS$_DUPLNAM.

It is quite common to see a few failed attempts on a restart even though the BGDevice shows the REUSEADR option.

Every 4 months or so a process will always fail all 24 attempts, every time we restart it. TCPIP SHOW DEVICE/PORT= shows nothing. The latest occurance lasted for a whole day and we left the process down for extended periods of time.

I have looked through the ECO 6 & 7 release notes and haven't noticed anything relavent.

Randy S.

[Robert Gezelter]

Randy,

If you are rebooting to resolve the problem, please force a crash next time so that the dump can be examined.

The question is: Is this a quirk of IP or is it some form of hung process connection.

Certainly, while the dump is useful, it would be interesting to attempt just a restart of the IP stack. Please remember that such a restart must be done from:

- a direct connection,
- a DECnet remote terminal,
- a LAT session, or
- a batch job.


In other words, the restart cannot be done from a telnet or ssh session (which would use the IP stack).

- Bob Gezelter, http://www.rlgsc.com

[Steven Schweda]

> [...] looping only occurs if the return
> status is SS$_DUPLNAM.

The status returned from what? (Is this a
process creation problem, or an IP-related
problem?)

[Randy W. Suhrbier]

IP related problem. Status from in IOSB for bind.

[Jon Pinkley]

Randy S.

Have you talked to the vendor of the application? Is there a supplied shutdown procedure for their product? What are you doing to restart the vendor's application processes?

Here is what the manual has to say about this status:

http://h71000.www7.hp.com/doc/73final/6529/6529pro_019.html#tcppmch05_15

SS$_DUPLNAM Programming error. The port being bound is already in use. An attempt to bind the socket to an address and port failed.

Jon

[labadie_1]

Could you post twice a
$ ucx sh dev bgxxx:/fu
for you bg device, when
1) it works fine
2) it is hung

It could help understand what goes on.

[Robert Gezelter]

Randy,

While it is easier to postmortem a system from a crash dump (which would be not a disruption since you are already rebooting), you can use SDA on the running system to display the relevant data structures to see what the status of the relevant BG device is.

In an unrelated situations, over the years, I have encountered numerous situations where an application component failed to exit when so instructed, and the restart procedure would fail. Looking at the output of SHOW SYSTEM or going into SDA normally identified the culprit in fairly short order. Manually fixing the problem (e.g., in most cases, terminating the malfunctioning processs) allowed the restart to occur without the need to resort to a system restart.

Long time forum user symposia and forum readers know that I loath rebooting systems unnecessarily.

- Bob Gezelter, http://www.rlgsc.com

[Randy W. Suhrbier]

Hi,

1. We stop and start the processes per vendor supplied procedures.

2. We have not seen any evidence that the process has not exited.

3. When the problem exists, UCX SHOW DEVICE/PORT= does not show any devices using the known port.

4. Does anyone have direct experience with the REUSEADR option?

[Arch_Muthiah]

Randy,

When we look into help and Jon's response...
> SS$_DUPLNAM Programming error.

you say $ ucx sho device/port=nnn shows nothing, no service and process assigned to that port. But the docu continues as
>The port being bound is already in use.
It won't be correct.

I doubt this may be the prog. error only.
OR as Jon's info, the application/service assigned on that port not properily shutdown. Because once the service is started on a specific port, simply disabling the service or stop/id the process will stop the process, but ends up with unpredicatable result and will be a problem again when we want to use that port. The vendor program may want to stop the service by specifying the service name, process name, port name, and protocol name. Proper shutdown is necessary in this case.

We have faced this kind of issue, but we were able to see the open socket name, so we disconnect the device socket, then re-start the program soled the issue.

Another doubt I have is that it happens every 4 months!. Also there will be a limit in the number of instances of the service to run in the system, I doubt this because you mentioned the program does not go through the loop all the time.

Check dcl show system --- when you don't find any opensocket with that port. And check options from $ucx sho devic/full --- to see REUSEADR for any other similar process.

Archie

[Randy W. Suhrbier]

Hi,

Normally the device looks like:

Device_socket: bg621 Type: STREAM
LOCAL REMOTE
Port: 37395 0
Host: * *
Service:
RECEIVE SEND
Queued I/O 0 0
Q0LEN 0 Socket buffer bytes 0 0
QLEN 0 Socket buffer quota 900000 900000
QLIMIT 4 Total buffer alloc 0 0
TIMEO 0 Total buffer limit 7200000 7200000
ERROR 0 Buffer or I/O waits 1 0
OOBMARK 0 Buffer or I/O drops 0 0
I/O completed 0 0
Bytes transferred 0 0

Options: ACCEPT REUSEADR KEEP LOOP FDPX_CLOSE
State: None
RCV Buff: WAIT
SND Buff: None

The LOOP option gives me pause since the documentation says it is reserved for VMS usage.

None of the vendor's apps are registered as TCPIP services.

Some simple testing with the QIO sample server and client programs in sys$examples: shows that restarting an app too quickly will result in SS$_DUPNAM if the REUSEADR is not specified. UCX SHOW DEVICE again does not show a matching device. Is there a way to look deeper into UCX?

Randy S.