Virtual Circuit Timed out

[Alanjones_1]

Dear Friends,

we have four nodes in MC Cluster. Error messages showed in all terminals

"PEA0% Virtual Circuit timed out - Remote port on Node XXXXX "

what is the cause for this error.

Regards
Balamurali

[Volker Halle · Accepted solution]

Balamurali,

in a cluster, the OpenVMS nodes use SCS to communicate. Each node establishes a Virtual Circuit (VC) to each other node using the available channels (e.g. LAN adapter to LAN adapter), which provide bi-directional communication.

PEDRIVER manages the SCS communication via LAN/FDDI. It will check communication on all channels every couple of seconds. The SCS layer will also check for timeouts on the VC.

Please always provide the exact message ! This message cannot show up on 'all terminals', it can only be printed on the CONSOLE terminals of the nodes. The exact text of this message is supposed to be:

%PEA0 - Virtual Circuit Timeout - REMOTE PORT xxx

This error happens, if the remote node xxx does not responds on it's VC. This error typically indicates, that the remote node is hung at IPL 8 (or higher). Force a crash on the remote node and find someone to analyze that crash.

Do all nodes report the same remote node ? Any problems with the network ? Any other console messages at that time ?

Volker.

[Alanjones_1]

Thanks Volker,

The Error message:

%PEA0 - Virtual Circuit Timeout - REMOTE PORT xxx

There is no other message appeared in all terminals. all error messages are appeared in console terminals only.

What is IPL? where i have to check the IPL values?

i have observed that the error message was displayed in all nodes in cluster.

i found that only one port in a switch displayed amber color, so i detach that port cable now it's working with out this error messages in console terminals.

What are the counters require to check in system devices?


Regards
Balamurali

[Volker Halle]

Balamurali,

did all cluster nodes report the SAME node (PORT) name in the VC timeout message ? Did those messages show up regularily ? And they did stop, once you disconnected the cable to the switch with the 'amber' port light ?

Find out which LAN interface on which node this switch port was connected to. Look at the LAN errors on that port:

$ MC LANCP SHO DEV/COUNT lan-device-name

IPL = Interrupt Priority Level. This is the CPU hardware priority for executing certain code inside the operating system, e.g. drivers run at IPL 8 doing certain IO work. If a driver is running at IPL 8 for too long, it is blocking other drivers and may be causing this type of VC timeout problem. If disonnecting the cable makes these messages go away, it's very likely the LAN driver to that specific LAN interface, which was connected to that 'amber' switch port, which was trying to send data on a failing port and maybe getting lots of errors.

Volker.

[Alanjones_1]

Thanks volker,

it shows all the nodes in a cluster. I have removed the faulty cable from the switch port now there is no errors in console terminals.

when i check the Lancp device counters it shows the like this:

Device Counters EIA0:
Value Counter
----- -------
498037 Seconds since last zeroed
26594807339 Bytes received
24766422574 Bytes sent
142478102 Packets received
77188158 Packets sent
1358591042 Multicast bytes received
38885426 Multicast bytes sent
15857015 Multicast packets received
333606 Multicast packets sent
45 Unrecognized unicast destination packets
2547024 Unrecognized multicast destination packets
0 Unavailable station buffers (XX-XXX-2006 17:44:01.27)
0 Unavailable user buffers
0 Alignment errors
0 Frame check errors
0 Frame size errors
0 Frame status errors
0 Frame length errors
0 Frame too long errors
13135 Data overruns
0 Send data length errors
0 Receive data length errors
0 Transmit underrun errors
0 Transmit failures
0 Carrier check failures
0 Station failures
0 Initially deferred packets sent
0 Single collision packets sent
0 Multiple collision packets sent
0 Excessive collisions
0 Late collisions
0 Collision detect check failures
8 Link up transitions (XX-XXX-2006 04:22:35.18)
7 Link down transitions (XX-XXX-2006 04:21:42.08)
XX-XXX-2006 04:22:34 Time of last generic transmit error
None Time of last generic receive error

How to clear the Data overrun errors in the particular EIA0 card?

Regards
Balamurali

[Volker Halle]

Balamurali,

you cannot zero the device counters shown by LANCP - except through a reboot.

I'm still not completely clear, which nodes have had VC timeout problems to which other nodes. A VC is a connection between the local node and a remote node. Each node in this 4 node cluster has 3 VCs - to the other 3 nodes (best seen with MC SCACP SHOW VC).

If there was a problem with a switch port or LAN interface (EIA0) on ONE node, this could have effected THAT node and it's VCs and also the VCs TO that node on all other nodes in the cluster. In that case, I would expect the following messages on the consoles:

other nodes: VC timeout to : VC timeout to other 3 nodes

Could you clarify this ?

BTW, I'm pretty sure, you're looking for the wrong LAN interface. If that EIA0 is the one, for which you've disconnected the cable, it should show lots of Carrier Check Failures ! Also note that the last transaction for that link was a Link UP - so there is still a cable and a switch port connected to that EIA0 interface.

Volker.