Slow cluster connection

[Dirk Bogaerts]

Hi folks,

Current config: 2x DS20E cluster connected to dual HSZ50 storage cabinet / OpenVMS V7.2-1 / TCP/IP V5.1 / each node has 1x 100Mb NIC plugged into the same 100Mb switch.

Problem: when MONI/CLUST is started on the slowest (500 MHz) node, the normal message MONITOR-I-ESTABCON appears and within 3 to 4 sec. the Cluster statistics screen comes up. Which is perfectly normal. When doing this on the fastest node (dual 883MHz - 4GB mem.), the ESTABCON appears normally, but is takes 40+ secs. before the statistics screen comes up. And the refresh interval of 6 secs. works fine on the slower node; on the faster one, it takes sometimes 12 to 18 secs to refresh.
Telnet session to both nodes have normal speed, no errors reported on the NIC's by Decnet nor by TCPIP.

What can be done to further investigate/solve this problem ???
Furthermore, can such a slow cluster connection endanger the cluster-state. Can the slow cluster interconnect cause the node to leave the cluster ??

Thanks,

[John Gillings]

Sanity check - make sure the console settings of the NIC speed on BOTH nodes is set to the same as the switch.

Do not trust autonegotiation.

On an unmanged auto-only switch, it may be safer to set the host to 100 half duplex.

Be sure to check both nodes and all switch ports.

$ MCR LANCP SHOW DEVICE/CHARACTERISTICS

[Dirk Bogaerts]

Indeed John, 'not to trust autonegotiate' is a lesson we learned the hard way some years ago on Wintel platforms. The NIC's on the OpenVMS systems have been always set fixed to the appropriate speed.

However, on the 'slow' cluster connect system, it's half-dup, the 'normal' cluster connect system, it's full-dup. BUT, just to rule out NIC/Switch speed problems, I ran some tests: a 300K-blocks file was FTP-ed from each cluster node to another node (over the LAN), as well as between the cluster-nodes (going no further than the switch they share). Speeds were consistent at about 1000blocks/sec over the 10Mb LAN and 1800blocks/sec for the intra-cluster FTP's (to the switch and back). The 'slow' cluster connect system, however, was always slightly (+/- 10%) faster.
This confirms the 'feeling' we had, that network speeds are similar when working on both nodes.

But at the same time the MONI CLUST connection remained about 10x slower on the 'slow' system...

My biggest worry remains ,can the slowness on one node impact the more important cluster-management communications. Can it for example cause the node to leave the cluster, if certain timeouts are reached...

[Sylvain Haex]

Hi,

If a set host between the nodes gives you the same effect than:
Check you DECnet configuration Phase IV and V towers.

[Dirk Bogaerts]

Indeed Sylvain, if Set Host would show similar slowness, then it could be a good startingpoint for further investigation. But on both nodes, the Set Host reacts with the speed of lighting (too fast for me to measure it with a stopwatch...).

Writing this note, I just got a hunch that maybe I should look a bit more in the Decnet direction as you indicated Sylvain. So I did lastnight's filetransfer-tests again, but this time not with FTP but with dcl COPY and .... from the 'slow' MONI CLUS to the other cluster member, good speeds (better than FTP !) were achieved. BUT from the 'normal' MONI CLUS node towards the 'slow' node, it was unbelievably slow (I stopped the copy after 80', as the other way around it took only 3' !!).

With this info in mind, I went back to SET HOST as originally suggested and I did some more extensive tests (dir/full on a very big directory). And although the data scrolls by at high speed, there are a couple of very distinctive 'hangs' of up to 5" - 7". And these hangs only appear on the slow node AND only using SET HOST (Decnet) and not when using TELNET !!
So maybe the long waiting time on the MONITOR-I-ESTABCON message on one node, was not for outgoing communication but for incoming, which would confirm the extremely long time to copy towards the 'slow' node.

So Julian, your Decnet hint may prove to be a very good one... and as I gave your message 3 points before starting my reply, I now realize I owe you at least 3 points ;-)

[Dirk Bogaerts]

*******************
* PROBLEM SOLVED *
*******************

The problem was solved by changing the half-duplex of one node to the full-dup which it should have had all along, because the switch is set fixed to full-dup.

I really can't explain why tests with FTP didn't reveal any speed problems. Is IP less prone to Ethernet mis-configuration compared to Decnet ? I firmly believed that such a basic Ethernet configuration mismatch, would impact any protocol used ! The normal speeds obtained by FTP mislead me to focussing onto the Decnet config, instead of checking all the basic stuff first! Lesson learned!

So John & Sylvain (sorry for getting your name mixed-up in a previous reply), I owe you both a couple of points !

[Wim Van den Wyngaert · Accepted solution]

Dag Dirk,

TCPIP will retransmit lost packages after 1 second (sysconfig -q inet tcp_rexmit_interval, unit 0.5 seconds).

DECNET has however a "back-off" mechanisme that will increase the retransmit interval between 2 retransmits (ncl show nsp all but I don't know the ncp command, check the delay fields). Thus subsequent transmit errors will result in long wait times (I have seen 30 seconds using vms 6.2).

MOP handles retransmissions very badly (when decserver was connected to full duplex switch MOP loads simply failed).

I guess the cluster protocol does the retransmission after about 5 seconds (check http://h71000.www7.hp.com/doc/72final/4477/4477pro_032.html but I didn't read it completely) which explains why monitor behaved badly. May be one of the goeroes knows the details.

Groetjes

Wim

[Dirk Bogaerts]

Hallo Wim, long time no see....

indeed, that could explain a lot !!

Groeten vanuit Antwerpen,
Dirk

[Mike Naime]

Have you thought about using a Crossover cable instead of the HUB?

We use "purple" crossover cables for all of our 2-node clusters. It takes the network electronics out of the picture. I know that I will not loose cluster communications if the network weenies loose a switch. :-)

[Dirk Bogaerts]

Indeed Mike,that's a good protection for the cluster against network mishaps. Never thought about such a solution (but don't know how to config it either...).
By the end of the year, I'll move to a double NIC per node solution with one of the NIC-failover methods provided by 7.3-2. And each NIC will be hooked up to a different switch, so a single NIC/switch/cable failure, won't bother me anymore....

[Martin P.J. Zinser]

Hi,

a cross-over cable is nice if the systems are in close proximity to each other. If you do need to do any kind of disastertolerance this is not really an option.

Greetings, Martin