Telnet/SSH intermittent hanging on new Itanium servers

[roose]

Hi Guys,

Just wanted to check the forum whether somebody else have already experienced our current issue with our new Itanium servers, and if they were able to resolve it, what they did.

We are currently in the midst of migrating our Alpha ES80 servers (OVMS 7.3-2) to rx7640 servers (OVMS 8.3-1H1, TCPIP v5.6ECO5). Our rx7640 servers are in a 4-node cluster configuration, whereas currently, our 4 ES80 servers are in a 2-node cluster configuration. Also, our Alpha servers have 100mbps NICs whereas our rx7640 server have 1gbps NICs. We have also implemented LAN failover on our rx7640 with 2 NICs on our LLA0 device, whereas we did not have such in our Alphas.

Problem now is that whenever we do direct telnet or ssh to our new Itanium servers, we are experiencing intermittent 1-2 seconds session hangs, then after that, our sessions will continue. Sessions are never disconnected so far. However, if we do telnet first to one of our Alpha servers, then do a set host from there to one of our Itanium servers, we are not seeing this problem.

We have already set our network switch ports to full-duplex 1gbps, and also on our Itanium servers to be the same, but still we are seeing this problem.

I have already opened a case with HP since last month, but we are still doing problem isolation as of today.

Has anyone encountered a similar problem before with a similar setup like ours? If so, where you able to resolve it? What did you do to resolve it?

[roose]

Here is one of our Alpha server's LANCP info.

[Volker Halle]

roose,

from your description, this sounds like a problem affecting TELNET and SSH, but not SET HOST. For further isolation, you would need to disclose, over which network interfaces you're running which protocols, if you believe this problem to be related to NICs.

Do you - by chance - have any NICs configured with TCPIP, but without connecting them to the network ?

Can you reproduce the problem using $ TCPIP PING/NUMBER_PACKETS=0 (continuous stream of ping packets) ?

If you use TELNET (instead of SET HOST) from your Alphas, does the problem also occur ?

Volker.

[Oswald Knoppers_1]

Can you also show the output of 'mc lancp sho dev/counter'?

Oswald

[roose]

Volker - Yes, we do have some additional NICs that are not connected to the network. I tried TCPIP PING command you gave, I was not able to reproduce the problem. Telnet from Alpha, we are also encountering this problem.

Oswald - I attached the result of the mc lancp show dev/counter for one of the node.

[J Asson]

Is this upon login only or during normal telnet sessions.

Not specific to your setup but I have seen issues with the config of name servers where delays were introduced to telnet based logins only.

In some cases the lack of an operational name server (meaning hosts had to be locally defined) presented quicker telnet login and when using a name server there were delays.

[roose]

J - this is during normal telnet session already, not during login. What do you mean by name server? Is it the DNS server? We did verify already with our DNS server admin and checked that our DNS entries are correct.

[J Asson]

Yes - Name Server = DNS.

The only time I have ever seen such delays with in-progress telnet sessions was when we had a faulty switch. What was odd in my case was that remote access was faster than those at the site (despite the last meter of cabling being common) and the issue was then isolated to the switch.

Since you see the issue across all new systems could you isolate the new kits and confirm the same results with direct network connections to each. I mean no other connections to the systems (one by one) other than your laptop or PC with no fail-over connections live either. As you can see I am all for breaking it down into simple chunks.

I assume the usual network port checks on your servers have been performed with no issues identified.

[Oswald Knoppers_1]

The counters look clean to me.

But the 1-2 second delay does indicate packet loss. And this only happens when you use the direct path from your wokstation to the new system (not via the Alpha).

Somewere in the new path this packet loss happens. So let your network people check the path from your workstation to the new system for errors.

Oswald

[Volker Halle]

Oswald,

the TELNET session hangs ALSO happen between the Alpha and the Itanium - according to the response from roose ! And they do NOT happen, when using SET HOST from Alpha to Itanium.

This indicadtes, that there must be some TCPIP or TELNET related problem in the path to the iatnium system !

rooose,

please report the output of TCPIP SHOW INT from the Itanium system. I'd like to see, on how many/which interfaces, TCPIP is configured. If there is more than one, is any of them NOT connected to the LAN ?

Volker.