DECNET PHASE IV - FLOW CONTROL

[Kevin Raven (UK)]

We run the following version of DECNET.
Identification = HP DECnet for OpenVMS Alpha
Management version = V4.0.0

Under OpenVMS V7.3-2 (Alpha ES45).

Question :
We have six pathworks clients that send data via DECNET to the Alpha Host.

Five function ok , but the sixth sends seven times more data than the other five.
Thus the sixth pathworks client , gets flow control messages.
Such as flow stop / flow start messages.
Thus this causes latency issues.

I presume that some sort of buffer for this DECNET link to the sixth pathworks client is getting overloaded.
I also presume that the buffer in question is not shared, otherwise the other clients would also suffer flow control latency.

What is the buffer in question and can it be increased with relative ease and saftey ?

Thanks

[Kevin Raven (UK)]

More data on system below ....

Identification = HP DECnet for OpenVMS Alpha
Management version = V4.0.0
Counter timer = 30
Incoming timer = 45
Outgoing timer = 60
Incoming Proxy = Enabled
Outgoing Proxy = Enabled
NSP version = V4.1.0
Maximum links = 512
Delay factor = 16
Delay weight = 5
Inactivity timer = 60
Retransmit factor = 10
Routing version = V2.0.0
Type = routing IV
Routing timer = 600
Broadcast routing timer = 180
Maximum address = 1023
Maximum circuits = 16
Maximum cost = 19
Maximum hops = 4
Maximum visits = 10
Maximum area = 63
Max broadcast nonrouters = 128
Max broadcast routers = 128
Maximum path splits = 1
Area maximum cost = 1022
Area maximum hops = 30
Maximum buffers = 100
Buffer size = 576
Default access = incoming and outgoing
Pipeline quota = 20000
Alias incoming = Enabled
Alias maximum links = 128
Alias node = 11.100 (XXXX)
Path split policy = Normal
Maximum Declared Objects = 31


NCP>


$ PRODUCT SHOW HIST
----------------------------------- ----------- ----------- --------------------
PRODUCT KIT TYPE OPERATION DATE AND TIME
----------------------------------- ----------- ----------- --------------------
HP VMS T4 V4.0 Full LP Install 16-NOV-2006 12:20:43
DEC AXPVMS VMS732_UPDATE V8.0 Patch Install 11-OCT-2006 09:50:51
DEC AXPVMS VMS732_MP V1.0 Patch Install 15-SEP-2006 14:18:25
DEC AXPVMS VMS732_SYS V10.0 Patch Install 15-SEP-2006 14:18:00
DEC AXPVMS VMS732_UPDATE V7.0 Patch Install 15-SEP-2006 14:16:01
DEC AXPVMS TCPIP_ECO V5.4-155 Patch Install 15-SEP-2006 12:54:56
DEC AXPVMS VMS732_FIBRE_SCSI V9.0 Patch Install 15-SEP-2006 12:53:36
DEC AXPVMS VMS732_UPDATE V6.0 Patch Install 15-SEP-2006 12:42:12
DEC AXPVMS VMS732_PCSI V3.0 Patch Install 15-SEP-2006 11:19:03
CPQ AXPVMS CDSA V2.0-109 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS DECNET_PHASE_IV V7.3-2 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS OPENVMS V7.3-2 Platform Install 14-SEP-2006 14:47:39
DEC AXPVMS TCPIP V5.4-15 Full LP Install 14-SEP-2006 14:47:39
DEC AXPVMS VMS V7.3-2 Oper System Install 14-SEP-2006 14:47:39
HP AXPVMS KERBEROS V2.0-6 Full LP Install 14-SEP-2006 14:47:39
----------------------------------- ----------- ----------- --------------------

15 items found
$

[Hoff · Accepted solution]

PATHWORKS 32 is dead.

DECnet on Windows is dead.

Even DECnet on Linux is dead.

It is time to move off of DECnet here and off of the (retired) PATHWORKS 32 package, and to move onto IP-based protocols and communications.

Given that my advice here will likely remain unheeded...

You'll want to indicate exactly what those flow control messages are. DECnet flow control is transparent to applications (until you wedge), which implies that something else is going on here. Or that you're looking at the lower level traffic within DECnet.

The flow control messages are usually triggered due to a network-level problem (slow link, flaky link, flaky NIC), or due to the target host or the target application(s) on that host simply not being fast enough to maintain the network load.

DECnet flow control is mildly nasty, too. In a moderate to complex environment, DECnet can back-pressure itself into a completely non-functional configuration. In more than a few cases, I've had to deliberately introduce message loss (message dropping) to better isolate the error, rather than causing multiple hosts within a configuration to all wedge.

If you have a slow link or a slow application, increasing the buffers won't help; it's much like a memory leak, in that regard. (Increasing the buffers can help with very bursty traffic...)

[Kevin Raven (UK)]

We are moving away from DECNET , Pathworks , OpenVMS in the near future.
So your advise is being followed :-)
But it's a big project and will take a while.

The application runs on a ES45 and is written in ADA.
Pathworks clients (6 of them) , pass messages to the application. In this instance one of the clients has 7 times the load that the other 5 clients have.
Is just the loaded client that gets the flow control messages.

The solution, I suppose is to rebalance the load between the 6 pathworks clients.

Question ?
Will each network link to each of the 6 clients have it's own buffer ?


We have had someone look at the network traffic and we are seeing flow control messages only to this one client.
Also pathworks client does not handle them well :-(
We see NSP Link control message (stop)
and NSP Link control message (go) ....

[Kevin Raven (UK)]

More info on flow control ....

No. Time Source Destination Protocol Info

6626 2009-10-27 14:03:40.947149000 11.151 11.208 DEC DNA NSP link control message(stop)



Frame 6626 (68 bytes on wire, 68 bytes captured)

Ethernet II, Src: DigitalE_00:97:2c (aa:00:04:00:97:2c), Dst: DigitalE_00:d0:2c (aa:00:04:00:d0:2c)

802.1Q Virtual LAN

DEC DNA Routing Protocol

DEC DNA NSP message: Link service message (0x10)

Destination node: 0x90ad

Source node: 0x21cd

Last interrupt/link service msg positively acknowledged: 3565

.... 1000 1100 1100 = Message number: 2252

...0 .... .... .... = Delayed ACK allowed: No

.... ..01 = Flow control: do not send data (0x01)



No. Time Source Destination Protocol Info

6685 2009-10-27 14:03:40.951708000 11.151 11.208 DEC DNA NSP link control message(go)



Frame 6685 (68 bytes on wire, 68 bytes captured)

Ethernet II, Src: DigitalE_00:97:2c (aa:00:04:00:97:2c), Dst: DigitalE_00:d0:2c (aa:00:04:00:d0:2c)

802.1Q Virtual LAN

DEC DNA Routing Protocol

DEC DNA NSP message: Link service message (0x10)

Destination node: 0x90ad

Source node: 0x21cd

Last interrupt/link service msg positively acknowledged: 3565

.... 1000 1100 1101 = Message number: 2253

...0 .... .... .... = Delayed ACK allowed: No

.... ..10 = Flow control: send data (0x02)

[Hoff]

>The solution, I suppose is to rebalance the load between the 6 pathworks clients.

That depends.

If the server application is common and if the server is getting overloaded, then rebalancing probably won't help.

I'd expect that various modern x86 processors can be faster than Alpha boxes, and there's a good chance that six x86 will be faster than a typical Alpha configuration.

>Will each network link to each of the 6 clients have it's own buffer ?

There are all sorts of buffers within the DECnet stack, and there are link buffers and common resources. If just one link is showing this, then chances are excellent it's a link buffer that is getting slammed. (But if the buffer is getting slammed because the application is too slow, increasing the buffer simply defers the inevitable.)

>We have had someone look at the network traffic and we are seeing flow control messages only to this one client.
Also pathworks client does not handle them well :-(
We see NSP Link control message (stop)
and NSP Link control message (go) ....

I'd suggest swapping in IP as one of your early-stage migration changes, either as a TCP connection or with Paxos or some middleware implemented. You can use that in situ, and then after you migrate.

Here's some (possibly dated) reading on this topic:

http://h71000.www7.hp.com/wizard/wiz_0265.html

Also note that (and not discussed in the ATW reply above) that any network errors or low-level configuration errors (eg: duplex) can really slam DECnet performance.

Check the counters in LANCP, as a start.

[Andy Bustamante]

Contact HP and make sure you have the latest patches for Pathworks32. There are known issues which have been corrected but never shipped in a media kit. To the best of my knowledge, access to the fixes is via HP's support center, there is is no download site.

As previously pointed out, consider migrating from DECnet to TCP/IP as the protocol presenting your network shares.

Involve your network group to verify there are no port errors on the Windows side or the OpenVMS side. Additionally, check the bandwidth (T4 can be your friend). I preferred to keep user access (http, telnet, ssh) and DECnet Pathworks access on different NICs. We had an application which loaded images with Pathworks32.

Andy Bustamante

[Kevin Raven (UK)]

Thanks for the responses so far.

HOFF ;
The new solution will be IP only and running on Solaris servers.
Pathworks/DECNET/OpenVMS does not factor in the new solution , anywhere.

I now have come to the conclusion , that increasing buffer sizes is not a solution.

Thanks

[Kevin Raven (UK)]

With reference to the reply so far and the fact that DECNET uses many buffers.
How do i find out the size of these buffers ?

I guess two in question are ....

$MC NCP SHOW EXEC CHAR
.......
Buffer size = 576
and
Pipeline quota = 20000

Others are ?

Thanks
Kevin

[Kevin Raven (UK)]

I have another question.
Our application response times are measured in milli seconds.
Does the Mailbox or Network Driver use an AST that fires on the millisecond boundary ?