HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Socket problems after TCP/IP Eco 6 patch

 
Bernhard von Gunten
Occasional Advisor

Socket problems after TCP/IP Eco 6 patch

Hello

We've got an interesting network problem between Open VMS and Tru64 Unix.

On VMS runs an daemon accepting Soap calls from a software running on Tru64.

The daemon handles about 300 connections per minute from this unix machine. Every connection is opened and closed (no keepalive). For each call, locally (127.0.0.1) another socket connection will be opened (and closed) to a worker process. Calls are normally finished within 1 second.

Anyway, things have been fine for months, but after installing Eco 6 patch, problems arised from that day on.

From time to time (each 15 - 30 minutes) we encounter "stalls" between our two machines. At the moment it looks like VMS does not accept sockets from Tru64 for 1 - 5 seconds and is queueing these requests. After these "breaks" these connections will be handled correctly.

Configuration:
- Open VMS 7.3-2, TCP/IP 5.4 Eco 6
- The VMS TCP/IP settings are default and not tuned in any way.

Our Questions are:

- Is there a known issue in this area since Eco 6 ?
- Are there possible TCP/IP settings in VMS that might affect our software since Eco 6 ?
- Any other ideas or pointers ?

Thank you for any help ;-)

Greetings from Switzerland
18 REPLIES
roose
Regular Advisor

Re: Socket problems after TCP/IP Eco 6 patch

Hi Bernhard,

I managed to experience a problem myself after installating TCP/IP v5.4 ECO6 on our test server, but my problem was with Telnet. The solution I got from the other threads here was to downgrade to ECO5.

This thread might be helpful to you:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1057983

Regards,
Roose
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

One thing i forgot to mention:

The customer upgraded to eco 6 directly from eco 2.

labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

Hello

What gives
$ ucx sh dev bgxxx/full on a bg device for your application ?

Q0LEN ?
QLEN ?
QLIMIT ?
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

Wow, you guys are fast ;-)

Find the output in the attachement.
labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

The goods news: There is not
State: CANTSENDMORE CANTRECVMORE

You have some
Buffer or I/O waits

I check in the docs.
EdgarZamora
Trusted Contributor

Re: Socket problems after TCP/IP Eco 6 patch


I encountered a somewhat similar problem in the past. Not sure if it was in ECO 6. I will have to check this weekend. However, the problem was caused by a change in tcpip behavior due to an engineering change. It had something to do with an application option (called "linger") involved in closing socket connections. I'll try to find more details this weekend. Did you recently enable scalable kernel?
labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

One more thing, do you have the result of the following commands before the problem and now

$ @sys$startup:tcpip$define_commands
$ sysconfig -q inet
$ sysconfig -q socket

I am afraid you only have it now.
EdgarZamora
Trusted Contributor

Re: Socket problems after TCP/IP Eco 6 patch


I was just about to reply that the problem I had mentioned in my previous post was not caused by ECO 6 and therefore is probably not related but then I saw the post saying the upgrade was from ECO2 to ECO6. I'm almost positive my problem was due to a change contained in ECO5 (or maybe 4) but I can't check until this weekend. So it still might be a possibility that it's the linger option that's also biting you.
labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

Can you post the result of

pipe wr sys$output "tcpip sh inetcb/stat" |
ana/sys | sea sys$pipe statistical/win=(0,12)
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

Labadie: Find the output in the attachement.

(Sadly, only from today.)
Ian Miller.
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

IIRC the change to tcp_keepidle = 14400
has caused some people problems. The previous value was smaller.
____________________
Purely Personal Opinion
labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

The previous value was 150 or 3600 I think.

I have a system with Eco 5, the sysconfig settings are exactly the same, so the change was in a more old ECO I suspect.

And you said you came from ECO 2.

By the way, nothing suspect in your tcpip sh inetcb/stat.
labadie_1
Honored Contributor

Re: Socket problems after TCP/IP Eco 6 patch

Oh, the unit is the half second, so 7200 means one hour.
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

EdgarZamora:

Linger definitely has been an issue between the workers and the daemon lately. Still facing problems between the daemon and the outside world. But you're right, any Eco Patch between 2 and 6 could have "hit" us there.

How did you deal with the issue?

Re: Socket problems after TCP/IP Eco 6 patch

Just guessing here, but this could be a problem with retransmissions. You may have to take some network traces of the traffic between the two systems and see if there is a lot of retransmit traffic. If there is then we have a patch for you. If you have the "perf" symbol tables loaded, you can check the retransmit stats with:

SDA> FORMAT/TY=TCPSTAT TCPSTAT

Steve
EdgarZamora
Trusted Contributor

Re: Socket problems after TCP/IP Eco 6 patch


Bernhard,

In my previous job, the linger problem was due to TCPIP engineering making a change to adhere more strictly to whatever RFC it was that was related to the function. I don't have my notes handy until I get home tonite.
The user eventually had to change their application code to take out the linger option. That was their solution.

I think this change was made earlier than ECO 5. Maybe 4 or 3 but I can't verify right now.
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

EdgarZamora:

Our linger to the outside world is set to 1 second today. But the day the problems started (and the Eco patch has been installed) it was set to 0. No difference though.

Anybody:

Today we're doing some load tests on the pre-production systems.

This is the output of $ ucx netstat -p tcp


tcp:
63998289 packets sent
59713016 data packets (3480843508 bytes)
463060 data packets (675952321 bytes) retransmitted
2230472 ack-only packets (182444 delayed)
0 URG only packets
38 window probe packets
743606 window update packets
848097 control packets
36486319 packets received
23771333 acks (for 3482439193 bytes)
5922635 duplicate acks
0 acks for unsent data
2499866 packets (1952310313 bytes) received in-sequence
745 completely duplicate packets (35209 bytes)
1 packet with some duplicate data (88 bytes duped)
731786 out-of-order packets (70761044 bytes)
0 packets (0 bytes) of data after window
0 window probes
3757913 window update packets
16 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet was too short
361202 connection requests
620870 connection accepts
982071 connections established (including accepts)
981706 connections closed (including 31 drops)
1 embryonic connection dropped
7354000 segments updated rtt (of 7571306 attempts)
164 retransmit timeouts
0 connections dropped by rexmit timeout
40 persist timeouts
71 keepalive timeouts
28 keepalive probes sent
1 connection dropped by keepalive

Any comments ?
Bernhard von Gunten
Occasional Advisor

Re: Socket problems after TCP/IP Eco 6 patch

Hello all,

thank you for your support. It was impressive to watch how fast people tried to help here and how important communities are for all of us.

I close the thread, as further investigations have to be done on Tru64 side.

If you're interested in it, i've just opened a thread in the (evil hehe) Unix forum.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1085736

Thanks & Greetings from Switzerland,

Bernhard