topic Re: DECNET PHASE IV - FLOW CONTROL in Operating System - OpenVMS

DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Thu, 03 Dec 2009 12:19:33 GMT

We run the following version of DECNET.
Identification = HP DECnet for OpenVMS Alpha
Management version = V4.0.0

Under OpenVMS V7.3-2 (Alpha ES45).

Question :
We have six pathworks clients that send data via DECNET to the Alpha Host.

Five function ok , but the sixth sends seven times more data than the other five.
Thus the sixth pathworks client , gets flow control messages.
Such as flow stop / flow start messages.
Thus this causes latency issues.

I presume that some sort of buffer for this DECNET link to the sixth pathworks client is getting overloaded.
I also presume that the buffer in question is not shared, otherwise the other clients would also suffer flow control latency.

What is the buffer in question and can it be increased with relative ease and saftey ?

Thanks

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Thu, 03 Dec 2009 12:22:47 GMT

More data on system below ....

Identification = HP DECnet for OpenVMS Alpha
Management version = V4.0.0
Counter timer = 30
Incoming timer = 45
Outgoing timer = 60
Incoming Proxy = Enabled
Outgoing Proxy = Enabled
NSP version = V4.1.0
Maximum links = 512
Delay factor = 16
Delay weight = 5
Inactivity timer = 60
Retransmit factor = 10
Routing version = V2.0.0
Type = routing IV
Routing timer = 600
Broadcast routing timer = 180
Maximum address = 1023
Maximum circuits = 16
Maximum cost = 19
Maximum hops = 4
Maximum visits = 10
Maximum area = 63
Max broadcast nonrouters = 128
Max broadcast routers = 128
Maximum path splits = 1
Area maximum cost = 1022
Area maximum hops = 30
Maximum buffers = 100
Buffer size = 576
Default access = incoming and outgoing
Pipeline quota = 20000
Alias incoming = Enabled
Alias maximum links = 128
Alias node = 11.100 (XXXX)
Path split policy = Normal
Maximum Declared Objects = 31

NCP>

$ PRODUCT SHOW HIST
----------------------------------- ----------- ----------- --------------------
PRODUCT KIT TYPE OPERATION DATE AND TIME
----------------------------------- ----------- ----------- --------------------
HP VMS T4 V4.0 Full LP Install 16-NOV-2006 12:20:43
DEC AXPVMS VMS732_UPDATE V8.0 Patch Install 11-OCT-2006 09:50:51
DEC AXPVMS VMS732_MP V1.0 Patch Install 15-SEP-2006 14:18:25
DEC AXPVMS VMS732_SYS V10.0 Patch Install 15-SEP-2006 14:18:00
DEC AXPVMS VMS732_UPDATE V7.0 Patch Install 15-SEP-2006 14:16:01
DEC AXPVMS TCPIP_ECO V5.4-155 Patch Install 15-SEP-2006 12:54:56
DEC AXPVMS VMS732_FIBRE_SCSI V9.0 Patch Install 15-SEP-2006 12:53:36
DEC AXPVMS VMS732_UPDATE V6.0 Patch Install 15-SEP-2006 12:42:12
DEC AXPVMS VMS732_PCSI V3.0 Patch Install 15-SEP-2006 11:19:03
CPQ AXPVMS CDSA V2.0-109 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS DECNET_PHASE_IV V7.3-2 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS OPENVMS V7.3-2 Platform Install 14-SEP-2006 14:47:39
DEC AXPVMS TCPIP V5.4-15 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS VMS V7.3-2 Oper System Install 14-SEP-2006 14:47:39
HP AXPVMS KERBEROS V2.0-6 Full LP Install 14-SEP-2006 14:47:39
----------------------------------- ----------- ----------- --------------------

15 items found
$

Re: DECNET PHASE IV - FLOW CONTROL

Hoff — Thu, 03 Dec 2009 15:14:40 GMT

PATHWORKS 32 is dead.

DECnet on Windows is dead.

Even DECnet on Linux is dead.

It is time to move off of DECnet here and off of the (retired) PATHWORKS 32 package, and to move onto IP-based protocols and communications.

Given that my advice here will likely remain unheeded...

You'll want to indicate exactly what those flow control messages are. DECnet flow control is transparent to applications (until you wedge), which implies that something else is going on here. Or that you're looking at the lower level traffic within DECnet.

The flow control messages are usually triggered due to a network-level problem (slow link, flaky link, flaky NIC), or due to the target host or the target application(s) on that host simply not being fast enough to maintain the network load.

DECnet flow control is mildly nasty, too. In a moderate to complex environment, DECnet can back-pressure itself into a completely non-functional configuration. In more than a few cases, I've had to deliberately introduce message loss (message dropping) to better isolate the error, rather than causing multiple hosts within a configuration to all wedge.

If you have a slow link or a slow application, increasing the buffers won't help; it's much like a memory leak, in that regard. (Increasing the buffers can help with very bursty traffic...)

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Thu, 03 Dec 2009 15:24:49 GMT

We are moving away from DECNET , Pathworks , OpenVMS in the near future.
So your advise is being followed :-)
But it's a big project and will take a while.

The application runs on a ES45 and is written in ADA.
Pathworks clients (6 of them) , pass messages to the application. In this instance one of the clients has 7 times the load that the other 5 clients have.
Is just the loaded client that gets the flow control messages.

The solution, I suppose is to rebalance the load between the 6 pathworks clients.

Question ?
Will each network link to each of the 6 clients have it's own buffer ?

We have had someone look at the network traffic and we are seeing flow control messages only to this one client.
Also pathworks client does not handle them well :-(
We see NSP Link control message (stop)
and NSP Link control message (go) ....

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Thu, 03 Dec 2009 15:29:30 GMT

More info on flow control ....

No. Time Source Destination Protocol Info

6626 2009-10-27 14:03:40.947149000 11.151 11.208 DEC DNA NSP link control message(stop)

Frame 6626 (68 bytes on wire, 68 bytes captured)

Ethernet II, Src: DigitalE_00:97:2c (aa:00:04:00:97:2c), Dst: DigitalE_00:d0:2c (aa:00:04:00:d0:2c)

802.1Q Virtual LAN

DEC DNA Routing Protocol

DEC DNA NSP message: Link service message (0x10)

Destination node: 0x90ad

Source node: 0x21cd

Last interrupt/link service msg positively acknowledged: 3565

.... 1000 1100 1100 = Message number: 2252

...0 .... .... .... = Delayed ACK allowed: No

.... ..01 = Flow control: do not send data (0x01)

No. Time Source Destination Protocol Info

6685 2009-10-27 14:03:40.951708000 11.151 11.208 DEC DNA NSP link control message(go)

Frame 6685 (68 bytes on wire, 68 bytes captured)

Ethernet II, Src: DigitalE_00:97:2c (aa:00:04:00:97:2c), Dst: DigitalE_00:d0:2c (aa:00:04:00:d0:2c)

802.1Q Virtual LAN

DEC DNA Routing Protocol

DEC DNA NSP message: Link service message (0x10)

Destination node: 0x90ad

Source node: 0x21cd

Last interrupt/link service msg positively acknowledged: 3565

.... 1000 1100 1101 = Message number: 2253

...0 .... .... .... = Delayed ACK allowed: No

.... ..10 = Flow control: send data (0x02)

Re: DECNET PHASE IV - FLOW CONTROL

Hoff — Thu, 03 Dec 2009 16:28:32 GMT

>The solution, I suppose is to rebalance the load between the 6 pathworks clients.

That depends.

If the server application is common and if the server is getting overloaded, then rebalancing probably won't help.

I'd expect that various modern x86 processors can be faster than Alpha boxes, and there's a good chance that six x86 will be faster than a typical Alpha configuration.

>Will each network link to each of the 6 clients have it's own buffer ?

There are all sorts of buffers within the DECnet stack, and there are link buffers and common resources. If just one link is showing this, then chances are excellent it's a link buffer that is getting slammed. (But if the buffer is getting slammed because the application is too slow, increasing the buffer simply defers the inevitable.)

>We have had someone look at the network traffic and we are seeing flow control messages only to this one client.
Also pathworks client does not handle them well :-(
We see NSP Link control message (stop)
and NSP Link control message (go) ....

I'd suggest swapping in IP as one of your early-stage migration changes, either as a TCP connection or with Paxos or some middleware implemented. You can use that in situ, and then after you migrate.

Here's some (possibly dated) reading on this topic:

http://h71000.www7.hp.com/wizard/wiz_0265.html

Also note that (and not discussed in the ATW reply above) that any network errors or low-level configuration errors (eg: duplex) can really slam DECnet performance.

Check the counters in LANCP, as a start.

Re: DECNET PHASE IV - FLOW CONTROL

Andy Bustamante — Thu, 03 Dec 2009 17:28:56 GMT

Contact HP and make sure you have the latest patches for Pathworks32. There are known issues which have been corrected but never shipped in a media kit. To the best of my knowledge, access to the fixes is via HP's support center, there is is no download site.

As previously pointed out, consider migrating from DECnet to TCP/IP as the protocol presenting your network shares.

Involve your network group to verify there are no port errors on the Windows side or the OpenVMS side. Additionally, check the bandwidth (T4 can be your friend). I preferred to keep user access (http, telnet, ssh) and DECnet Pathworks access on different NICs. We had an application which loaded images with Pathworks32.

Andy Bustamante

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Thu, 03 Dec 2009 18:35:01 GMT

Thanks for the responses so far.

HOFF ;
The new solution will be IP only and running on Solaris servers.
Pathworks/DECNET/OpenVMS does not factor in the new solution , anywhere.

I now have come to the conclusion , that increasing buffer sizes is not a solution.

Thanks

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Fri, 04 Dec 2009 10:01:48 GMT

With reference to the reply so far and the fact that DECNET uses many buffers.
How do i find out the size of these buffers ?

I guess two in question are ....

$MC NCP SHOW EXEC CHAR
.......
Buffer size = 576
and
Pipeline quota = 20000

Others are ?

Thanks
Kevin

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Fri, 04 Dec 2009 11:59:11 GMT

I have another question.
Our application response times are measured in milli seconds.
Does the Mailbox or Network Driver use an AST that fires on the millisecond boundary ?

Re: DECNET PHASE IV - FLOW CONTROL

Art Wiens — Fri, 04 Dec 2009 14:02:58 GMT

NCP> help set line receive

SET LINE Subtopic? receive

SET

LINE

RECEIVE BUFFERS number

Specifies the length of the line's receive queue. Use a
number in the range of 1 to 32. A value in the range of 2
to 4 is adequate for line speeds of less than 56K bits.
Megabit line speeds may require 8 or more buffers depending
on the observed error rate.

Cheers,
Art

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Fri, 04 Dec 2009 15:10:29 GMT

Thanks for the reply so far.
I have found the various buffers used by DECNET ...! phew ....
Logical Device
Physical device
Line
Exec
Logical link

The main question now is ...how often does OpenVMS tick , to sense that DECNET has data in a buffer to raise a AST to service the data.
Is it every millisecond or less ?
Our application logs , show data being pumped into the OpenVMS server once every millisecond.

We were seeing transaction rates at 680 trans per second, when flow stop/go messages were sent to the pathworks DECNET client.

Re: DECNET PHASE IV - FLOW CONTROL

Hoff — Fri, 04 Dec 2009 16:02:50 GMT

>Our application response times are measured in milli seconds.

As you're already well aware, you've some very tight timing requirements here. Alpha ticks over at 1000 tps or faster, but the clocks still chunk forward more slowly. The clocks on various OpenVMS boxes only get updated at roughly ten millisecond intervals.

>Does the Mailbox or Network Driver use an AST that fires on the millisecond boundary ?

As a rule, ASTs do not fire on boundaries by intention, they fire on an event. Depending on the priority involved, you can get your activity chunked behind somebody else. (And I don't know off-hand if DECnet honors priority here; I'd not assume it.)

I doubt that any of the timing boundaries here would be documented (and AFAIK, none are) and this whole area could thus be subject to change, which means you can go measure it and then choose to depend on the observed behavior at your own discretion. Changes probably would not be intentional, but I could easily see OpenVMS going off into Deep Thought over (say) a memory error, or DECnet getting distracted by routing topology changes.

DECnet would not be my choice for millisecond response time requirements, given how the protocol tends to respond to congestion and to transient errors, and particularly around how it back-pressures traffic.

IP and networking in general can inject considerations here, too. I'd not automatically expect a switch's spanning tree to run in a millisecond, for instance.

In practice, I'd probably look to get a controller out closer to or onto the target (whether an Arduino or more expensive) and work to push the timing-critical activities off of OpenVMS, Solaris or other multi-user multi-processing operating system. I'm sure you've thought of this, though, so I'd assume there are constraints that skew against local intelligent control of the task.

But we're getting deep into real-time system design here, and that's not something I design for free.

Re: DECNET PHASE IV - FLOW CONTROL

Kevin Raven (UK) — Fri, 04 Dec 2009 17:58:28 GMT

Hoff - Thanks for the answers on this.
I can't go into to much detail on a public forum, on how and what our current system is used for.

I would love to go into detail, but can't :-(

I could always e-mail you off-line if your interested.

Thanks
Kevin

Re: DECNET PHASE IV - FLOW CONTROL

Hoff — Fri, 04 Dec 2009 18:55:25 GMT

Well, if you want to share, send it along.

http://labs.hoffmanlabs.com/contact