TCPIP proxy oddity

[Jan van den Ende]

HELP!!!

Last night (and of course while I was on call) an application stopped using SOME of its communications.

The app uses incoming and outgoing communication, both socket-to-socket and remote batch activations via SCP messages.

I drilled it down to the TCPIP proxies, those of TYPE CD working fine, those of type C failing.

The funny part: about a week ago I had to add some more (other) proxies for the app, and checked them all, they were just fine. The app is used rather frequently, and only last night the errors started coming in.

Just a TYPE C proxy is not uncommon: it appears on all other nodes of the cluster if
$ UCX ADD PROXY ....
is entered on one node.
(or TCPIP, but just TCP is ambiguous, so I usually use UCX)

But the node on which the command is issued, normally gets TYPE CD.
On the other nodes issuing UCX SET TCP/SIGNAL sets all TYPEs to the latest setting.

BUT
WHY oh WHY have I lost a bunch of CD records, and can I _NOT_ set them anymore?

I found one that was not used anymore (the app for that area moved to a new machine).
I REMOVEd it, and ADDed it again, and THAT also is now a TYPE C ! So I am not going to experiment with any TYPE CDs that are still in use.

$ UCX SHOW VERS
HP TCP/IP Services for OpenVMS Alpha Version 5.4 - ECO 5 on an AlphaServer ES45 Model 2B running OpenVMS V7.3-2

I still have recollection of (ISTR) UCX 5.1 (?) ECO ? on VMS 7.2(-1?) crashing the system if more than a certain number of proxies were set up; can this be a milder version of the same issue? If so, anybody know WHAT that limit is, and if and how it can be raised?

If I am just plain stupid and did anything wrong, please say so, and I will just be SO glad that it can be repaired easily!


Proost.

Have one on me.

jpe

[Jan van den Ende]

Help, anybody?

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

Jan,

On UCX 5.3 eco 2 (yes ucx too).

If I add a proxy with a non-existing user, I get a C. In all other cases I get a CD.

May be something is wrong with the user (intrusion, disuser, ... ).

The D in CD indicates that it has been loaded (with success ?) in the cache.

Ucx show commun shows you the maximum number of proxies. Didn't test it but may be you get a C if that number is exceeded. Or may be there is a bug active only when exceeding this maimum.

Wim

[Jan van den Ende]

Wim,

(beste wensen nog)

The proxies that this is all about HAVE BEEN working correctly for over month.

The proxy _IS_ to an existing user, and many more proxies to that user exist, and are functioning correctly.

So, nothing wrong with the account.

Obviously, the problem _IS_ to load them into the volatile DB.

I did not know about that SHOW (Thanks!!).
Our Maximum setting is 2000,
Current & Peak are blank.
Currently defined proxies: about 180.

Any more ideas, anybody?

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

Yes. Download eco 6, extract the release notes and check them. When I search the patches of 7.3-2 for proxy the eco 6 is given but you have to download it before you can read the release notes. Bad and sad because downloading is damn difficult over here.

Wim

[Ian Miller.]

nothing in ECO6 release notes about proxies.

What else changed apart from proxies stopping working?

[Wim Van den Wyngaert]

Did some testing on my 5.3 eco 2.

I tried adding a lot of proxies. After about 100 proxies it got "out of dynamic mem" errors. I tried with another remote user name : it added 15 proxies and again "out of dyn mem".

ucx show prox : no CD for none of the proxies.

ucx set tcp/sign : didn't solve the problem.

Tried to use the new proxy : didn't work.

Rebooted : the proxies got their CD and they work now. My max in show comm is at 107 after the reboot while it was at 20 before. So, ucx increased the maximum itself (as documented in help set comm/prox).

I rebooted again after removing the new proxies. The 107 was back to 20.

Fwiw

Wim

[Wim Van den Wyngaert]

BTW : I think every remote host counts as 1 proxy (in max context).

Wim

[Wim Van den Wyngaert]

May sound stupid but did you see the output of ucx set tcp/sign ? No messages ?

Because during my test I got out of dyn mem.

Wim

[Jan van den Ende]

Ian,

thanx, saves me the effort.

Wim:
I counted every IP address (most comms partners have redundant networks) and also every DNS name, as separate entities. (and with 6 nodes in one of the remote clusters, that grows rather faster than ONE outgoing DECnet name, eh?).

Re: Set tcp/sig:
I expect the node on which the ADD PROXY is done to not need that. Nevertheless. I DID try, in vain.

----

UCX HELP says that remote user, and remote host, can be wildcarded. Does anybody have experience (or the facilities to experiment) with partial names, like

USRXY*
or
NODEX*
or
node*.dom.ain
or
10.20.30.*
?


Partial answers also welcome!

Proost.

Have one on me.

[Jan van den Ende]

Oh,

Ian,

>>>What else changed apart from proxies stopping working?
<<<

Nothing that I could identify (yet).

It _IS_ at about the time when Backup processing does several applic tricks (quiesce a DB, stop & restart an app, dismount a member of each shadow set),
but AFAIK nothing with network.


For a bit more circumstantial info: this is NOT our long-standing cluster, but the config that everything is supposed to be moving towards. And I am by far not as well aware of everything that happens there, or should happen, or should NOT happen. I was just lucky enough to be on-call when "something" started mis-behaving.

Thnx for thinking along with me so far.

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

Jan,

can you confirm that you did ucx set tcp/sign
and that no abnormal output was given ?
Not even xxx skipped where xxx > 0 ?

Wim

[Jan van den Ende]

Wim,

>>>
can you confirm that you did ucx set tcp/sign
and that no abnormal output was given ?
>>>

Definitely. Retried it to make double sure.

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

I found notes of J. Huber saying that you need to do
$ @tcpip$examples:TCPIP$PROXY_RELOAD
$ tcpip set tcp/signal
to reload proxies.

The @ is doing "ucx load prox" but also stopping NFS. So, may be do the ucx load prox by hand ? I tried it on my station and it tries to re-add the permanent database to volatile memory and gives "duplicate name" for all operations (so, harmless ?). In my case 0 proxies were added.

It would be better if all this was documented.

Wim

[Jeffrey Goodwin]

Do any of the new proxies you created conflict with proxies already on the system? This is especially true if wildcards are used.

For example:

$ TCPIP show proxy VMSUSER

VMS User_name Type User_ID Group_ID Host_name

VMSUSER CD * NODE1
VMSUSER C remuser NODE1, NODE2

The proxy with the specific user will not load because it is already accounted for in the proxy with the wildcard.

-Jeff

[Jan van den Ende]

Wim,

I will try this first thing tomorrow morning (and I postpone the points, depending on the result).

Jeffrey,
No. And, those proxies WERE successfully in use 'till "something" (someone, at shortly after midnight??) broke it, and apparently beyond repair until now.

Proost.

Have one on me.

jpe

[Jan van den Ende]

So,

it now became clear that all of this was just a very unlucky coincidence of circumstances.

- the app. is clusterwide. Meaning that some servicejobs need to run on all nodes.
- those service jobs apparently have some limitation (but what??). If they run too long, the just stop functioning.
Therefore, they are normally stopped & restarted each night. Somehow that had was not, or not fully, implemented (the app moved to this cluster about 6 weeks ago).
- from within the cluster, connection is make to the IP cluster impersonator; from (at least some) other systems it is to a round-robin over the nodes.

... and the TCPIP proxy type C or CD now looks like just a cosmetical problem, the type C working just as correctly as the type CD.

All in all: just an application issue, which in itself is easily circumvented, but has bitten us by coincidence.

Thanks to all that have tried to help, and sorry to have lead you chasing red herrings.

Proost.

Have one on me.

jpe