Lan getting failed in the cluster

[SUDHAKAR_18]

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 17:39:09.925945
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 17:48:42.969381
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 20:11:25.236822
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 20:26:38.507654
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 20:27:04.577416
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 20:27:15.996185
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 20:27:44.076472
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Thu Jul 02 IST 2009 21:12:46.853906
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Fri Jul 03 IST 2009 00:54:41.409399
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Mon Jul 06 IST 2009 04:32:15.519897
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Mon Jul 06 IST 2009 04:32:36.019125
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Mon Jul 06 IST 2009 04:33:38.240252
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Mon Jul 06 IST 2009 04:34:21.023888
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

-------------------100BT/Gigabit Ethernet LAN/9000 Networking---------------@#%
Timestamp : Tue Jul 07 IST 2009 13:12:33.257609
Process ID : [ICS] Subsystem : IETHER
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 2 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable.

[likid0]

You can also check what's your network polling config in the cluster.ascii.

NETWORK_POLLING_INTERVAL

you could give it, a higher value(not to high), and take a look if your lan failures continue.

[Jitesh purohit_1]

Hi Sudhakar,

Please check with network team, The nettlog log clearly says it could be a loose network cable connection issue...

2000> 1000Base-T in path 0/2/1/0/6/0
Detected a faulty or disconnected cable

Thanks
Jitesh

[Mel Burslan]

If your network team says their equipment is not rebooting or doing anything funky at the time these errors get logged to the console, there is only one way to explain this phenomena: A NIC or Network Switch port that is intermittently failing. Since the problem is not periodic or following any pattern as far as the timing is concerned, my best suggestion is to start with using two different, known good, switch ports and connect these cables coming from two servers, into the new switch ports. If the problem does not go away, then I hope you have a spare NIC on each of these cluster members. You will need to recompile your cluster binaries with a different set of NICs. replacing ones you are using.

This problem sounds anything but trivial. Good luck.

[melvyn burnard]

take the cable out of lan2, and plug it into lan0, then plug teh cable that was in lan0 into lan2.
This should give a better indication of where the issue may lie.
If it follows the cable, then it is a cable/switch issue. If it stay with lan2, you have a faulty NIC

[SUDHAKAR_18]

After changing lan cables (active & standby) problem is resolved.