ProCurve 5412 zl: Port off-line / on-line

[Torsten Roloff]

We expirience a lot of port off-line/on-line toggling through random ports of our ProCurve 5412zl (J8698A).

Updating software to the latest version did not solve the problem. Searching the forum and knowledgebase did not reveal any solutions.

Attached an excerpt from syslog. Do you have any suggestions how we can stop this behaviour?

Thanks a lot
Torsten


ProCurve RKP# show version
Image stamp: /sw/code/build/btm(K_15_03)
Feb 3 2011 21:05:19
K.15.03.0007
68
Boot Image: Primary
ProCurve RKP# show flash
Image Size (bytes) Date Version
----------------- ------------ -------- --------------------
Primary Image : 13706253 02/03/11 K.15.03.0007
Secondary Image : 7566208 07/30/09 K.13.68

Boot ROM Version : K.15.11
Default Boot : Primary
ProCurve RKP# show spanning-tree

Multiple Spanning Tree (MST) Information
STP Enabled : No
ProCurve RKP# show lacp
LACP
no LACP ports found.

[cenk sasmaztin]

this not a problem

when client shutdown operating system or disconnect port go to off line.... when a client open operating system or connect network cable link go to online.

[Torsten Roloff]

Tanks for the quick reply. Anyhow, please have a look at the attached csv file, since I don't consider this normal behaviour.

- toggling happens even in the non-working hours, we have logs full of these messages
- ports go off-line/on-line within seconds; sometimes more than one cycle (e.g. see port B12 from 07:15:08 to 07:15:36 in the attached csv)
- users report short network outages, which may correspond to the toggling
- besides this, we have the normal "off-line" messages (somebody turned off his computer) without following "on-line" message

[Gerhard Roets]

Hi Roloff

These messages are not only triggered by switching machines on and off. These are also triggered when the machines just power states.

For instance a machine in production run at 1GB then it goes to low power it drops speed and restarts auto-negotiation. This would show as a message.
Then when some one wakes it it might go up again. If you have a patch installed or being pushed bringing the machine out of low power mode this might happen again.

If you can post us a "show log -r" and "show tech stat", "show modules detail" and if possible a "show LLDP i r" <-- you can sanitize the LLDP information I just want to check if you have uplink ports "toggling".

I would suggest you try to identify a few machines with this issue where it happens cronically, and force them into an always on full power state. Check if this changes the behavior you are seeing

Thanks in advance
Gerhard

[Torsten Roloff]

Hi Gerhard,

here's the information you requested:
ProCurve RKP# show tech stat

statistics

Real Port Transitions: #transitions(Port Name)
Slot 1:
0(A1 ) 6(A2 ) 1(A3 ) 71(A4 )
8(A5 ) 17(A6 ) 3(A7 ) 5(A8 )
3(A9 ) 3(A10) 1(A11) 9(A12)
10(A13) 7(A14) 15(A15) 0(A16)
5(A17) 13(A18) 5(A19) 5(A20)
31(A21) 16(A22) 13(A23) 0(A24)

Slot 2:
5(B1 ) 1(B2 ) 22(B3 ) 11(B4 )
9(B5 ) 13(B6 ) 9(B7 ) 5(B8 )
13(B9 ) 13(B10) 13(B11) 13(B12)
5(B13) 13(B14) 2(B15) 19(B16)
0(B17) 1(B18) 21(B19) 1(B20)
1(B21) 1(B22) 2(B23) 1(B24)

Slot 3:
9(C1 ) 9(C2 ) 13(C3 ) 7(C4 )
7(C5 ) 1(C6 ) 1(C7 ) 13(C8 )
1(C9 ) 1(C10) 1(C11) 9(C12)
17(C13) 2(C14) 9(C15) 11(C16)
5(C17) 9(C18) 2(C19) 9(C20)
17(C21) 0(C22) 7(C23) 9(C24)

Slot 4:
17(D1 ) 9(D2 ) 9(D3 ) 1(D4 )
13(D5 ) 0(D6 ) 9(D7 ) 13(D8 )
3(D9 ) 9(D10) 7(D11) 5(D12)
0(D13) 13(D14) 0(D15) 5(D16)
0(D17) 0(D18) 0(D19) 1(D20)
1(D21) 0(D22) 1(D23) 1(D24)

Slot 5:
9(E1 ) 9(E2 ) 0(E3 ) 9(E4 )
1(E5 ) 9(E6 ) 1(E7 ) 0(E8 )
1(E9 ) 0(E10) 4(E11) 9(E12)
0(E13) 7(E14) 29(E15) 1(E16)
1(E17) 0(E18) 5(E19) 1(E20)
0(E21) 0(E22) 0(E23) 0(E24)

Slot 7:
1(G1 ) 0(G2 ) 1(G3 ) 1(G4 )
1(G5 ) 1(G6 ) 1(G7 ) 1(G8 )
1(G9 ) 1(G10) 1(G11) 1(G12)
1(G13) 1(G14) 1(G15) 1(G16)
1(G17) 1(G18) 1(G19) 1(G20)
1(G21) 1(G22) 1(G23) 1(G24)

Slot 8:
1(H1 ) 1(H2 ) 1(H3 ) 1(H4 )
1(H5 ) 1(H6 ) 13(H7 ) 1(H8 )
1(H9 ) 1(H10) 1(H11) 1(H12)
1(H13) 1(H14) 1(H15) 1(H16)
1(H17) 1(H18) 1(H19) 1(H20)
1(H21) 1(H22) 1(H23) 10(H24)

Slot 9:
1(I1 ) 1(I2 ) 1(I3 ) 1(I4 )
1(I5 ) 1(I6 ) 0(I7 ) 1(I8 )
1(I9 ) 1(I10) 1(I11) 1(I12)
1(I13) 1(I14) 1(I15) 0(I16)
1(I17) 1(I18) 1(I19) 0(I20)
0(I21) 0(I22) 0(I23) 0(I24)

Slot 10:
1(J1 ) 1(J2 ) 1(J3 ) 1(J4 )
1(J5 ) 1(J6 ) 1(J7 ) 1(J8 )
1(J9 ) 1(J10) 1(J11) 1(J12)
1(J13) 1(J14) 1(J15) 1(J16)
1(J17) 1(J18) 1(J19) 0(J20)
0(J21) 0(J22) 3(J23) 3(J24)


Linked Ports on box = 173

ProCurve RKP# show lldp i r

LLDP Remote Devices Information

LocalPort | ChassisId PortId PortDescr SysName
--------- + ------------------------- ------ --------- ----------------------


ProCurve RKP# show modules details

Status and Counters - Module Information

Chassis: 5412zl J8698A! Serial Number: SG906SV3LL
Allow V1 Modules: Yes

Management Module: J8726A Serial Number: ID906AS05Y Core Dump: YES


Programmable devices on MM

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
A ProCurve J8702A 24p Gig-T zl Module SG903AT2CP Up NO 1

Programmable devices on A

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
B ProCurve J8702A 24p Gig-T zl Module SG903AT27Q Up NO 1

Programmable devices on B

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
C ProCurve J8702A 24p Gig-T zl Module SG903AT1X9 Up NO 1

Programmable devices on C

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
D ProCurve J8702A 24p Gig-T zl Module SG903AT2CC Up NO 1

Programmable devices on D

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
E ProCurve J8705A Gig-T/SFP zl Module SG909AU03K Up NO 1

Programmable devices on E

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
G ProCurve J8702A 24p Gig-T zl Module SG820AT2PG Up NO 1

Programmable devices on G

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
H ProCurve J8702A 24p Gig-T zl Module SG841AT0T5 Up NO 1

Programmable devices on H

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
I ProCurve J8702A 24p Gig-T zl Module SG841AT0SL Up NO 1

Programmable devices on I

Device Description Version
------ ---------------------------------------- ----------------------


Core Mod
Slot Module Description Serial Number Status Dump Ver
---- -------------------------------------- -------------- -------- ----- ---
J ProCurve J8702A 24p Gig-T zl Module SG821AT0GS Up NO 1

Programmable devices on J

Device Description Version
------ ---------------------------------------- ----------------------

Regards,
Torsten

[Torsten Roloff]

and the log file

[Gerhard Roets]

Hi Torsten(Let me use your first name :) )

Gave it a quick eye-ball. It looks relatively sane except for a few ports.

Let me show one example
Keep in mind that most modern systems have Wake On Lan enabled by default.
I 04/07/11 11:54:16 00076 ports: port B14 is now on-line
I 04/07/11 11:54:14 00077 ports: port B14 is now off-line
--- ^^^ Operating system driver loaded NIC renegotiation

I 04/07/11 11:47:53 00076 ports: port B14 is now on-line
I 04/07/11 11:47:50 00077 ports: port B14 is now off-line
--- ^^^ NIC tries to boot from LAN
I 04/07/11 11:47:33 00076 ports: port B14 is now on-line
I 04/07/11 11:47:30 00077 ports: port B14 is now off-line
--- ^^^ Machine powered on Ethernet NIC speed change forced by power saving state change.

Now the above sequence would increment the "Real Port Transition" from "show tech stat counter by 6 as online-to-offline is 1, and offline-to-online is 1.

Having said that. I think your worry is a lot about port A4
I 04/07/11 07:11:56 00076 ports: port B12 is now on-line
I 04/07/11 07:11:51 00077 ports: port B12 is now off-line
I 04/07/11 07:07:07 00076 ports: port B11 is now on-line
I 04/07/11 07:07:05 00077 ports: port B11 is now off-line
--- ^^^ People getting in
I 04/07/11 05:05:24 00076 ports: port A4 is now on-line
I 04/07/11 05:05:23 00077 ports: port A4 is now off-line
I 04/07/11 05:05:22 00076 ports: port A4 is now on-line
I 04/07/11 05:05:19 00077 ports: port A4 is now off-line
I 04/07/11 05:05:18 00076 ports: port A4 is now on-line
I 04/07/11 05:05:15 00077 ports: port A4 is now off-line
I 04/07/11 05:05:14 00076 ports: port A4 is now on-line
I 04/07/11 05:05:11 00077 ports: port A4 is now off-line
I 04/07/11 05:05:10 00076 ports: port A4 is now on-line
I 04/07/11 05:05:07 00077 ports: port A4 is now off-line
I 04/07/11 05:05:06 00076 ports: port A4 is now on-line
I 04/07/11 05:05:03 00077 ports: port A4 is now off-line
I 04/06/11 21:37:51 00076 ports: port A4 is now on-line
I 04/06/11 21:37:49 00077 ports: port A4 is now off-line
I 04/06/11 21:37:48 00076 ports: port A4 is now on-line
I 04/06/11 21:37:45 00077 ports: port A4 is now off-line
I 04/06/11 21:37:44 00076 ports: port A4 is now on-line
I 04/06/11 21:37:41 00077 ports: port A4 is now off-line
I 04/06/11 21:37:40 00076 ports: port A4 is now on-line
I 04/06/11 21:37:37 00077 ports: port A4 is now off-line
I 04/06/11 21:37:36 00076 ports: port A4 is now on-line
I 04/06/11 21:37:33 00077 ports: port A4 is now off-line
I 04/06/11 21:37:32 00076 ports: port A4 is now on-line
I 04/06/11 21:37:29 00077 ports: port A4 is now off-line
--- ^^^ Oddness.
---- vvv People going home and shutting down
I 04/06/11 20:41:08 00077 ports: port D1 is now off-line
I 04/06/11 20:38:19 00077 ports: port D3 is now off-line
I 04/06/11 20:35:54 00077 ports: port D2 is now off-line
I 04/06/11 20:11:10 00077 ports: port C3 is now off-line

The odd thing is port A4 only seems to toggle at night. In the day it seems to be stable.
1. I would suggest you document the kind of system connected to Port A4, network card(with chipset), network drivers and network config.
2. Try to move it to a different port on a different module. See if the behavior changes.
3. Move it to a port A12 or higher on module B and see if the behavior changes

Do you have a backup running over that time of night. I have seen some NICS who might have an issue show this kind of symptom under load.

Port A21 and A15 showed a bit of an issue
I 04/06/11 10:50:23 00076 ports: port A15 is now on-line
I 04/06/11 10:50:21 00077 ports: port A15 is now off-line
I 04/06/11 10:50:20 00076 ports: port A15 is now on-line
I 04/06/11 10:50:17 00077 ports: port A15 is now off-line
I 04/06/11 10:50:17 00076 ports: port A15 is now on-line
I 04/06/11 10:50:12 00077 ports: port A15 is now off-line
I 04/06/11 10:50:12 00076 ports: port A15 is now on-line
I 04/06/11 10:50:09 00077 ports: port A15 is now off-line
I 04/06/11 10:50:08 00076 ports: port A15 is now on-line
I 04/06/11 10:50:05 00077 ports: port A15 is now off-line
I 04/06/11 10:50:04 00076 ports: port A15 is now on-line
I 04/06/11 10:50:01 00077 ports: port A15 is now off-line
I 04/06/11 10:39:33 00076 ports: port A21 is now on-line
I 04/06/11 10:39:31 00077 ports: port A21 is now off-line
I 04/06/11 10:39:11 00076 ports: port A21 is now on-line
I 04/06/11 10:39:08 00077 ports: port A21 is now off-line
I 04/06/11 10:39:07 00076 ports: port A21 is now on-line
I 04/06/11 10:39:04 00077 ports: port A21 is now off-line
I 04/06/11 10:39:03 00076 ports: port A21 is now on-line
I 04/06/11 10:38:57 00077 ports: port A21 is now off-line
I 04/06/11 10:38:37 00076 ports: port A21 is now on-line
I 04/06/11 10:38:35 00077 ports: port A21 is now off-line

Those feel a bit more like normal use I would have a quick look at the machines themselves.

Does this sound in line of what you saw or did I miss something ?

HTH
Gerhard

[scannon]

I am seeing the same issues with my 5406's and 5412's. 

show logging -r 

I 08/26/15 06:24:40 00076 ports: port B5 is now on-line
I 08/26/15 06:23:58 00077 ports: port B5 is now off-line
I 08/26/15 06:22:14 00076 ports: port L3 is now on-line
I 08/26/15 06:22:10 00077 ports: port L3 is now off-line
I 08/26/15 06:06:41 00076 ports: port L22 is now on-line
I 08/26/15 06:06:40 00077 ports: port L22 is now off-line
I 08/26/15 06:06:23 00076 ports: port L22 is now on-line
I 08/26/15 06:06:21 00077 ports: port L22 is now off-line
I 08/26/15 05:47:32 00077 ports: port I22 is now off-line
I 08/26/15 05:41:51 00076 ports: port B5 is now on-line
I 08/26/15 05:41:30 00077 ports: port B5 is now off-line
I 08/26/15 05:41:29 00076 ports: port B5 is now on-line
I 08/26/15 05:41:07 00077 ports: port B5 is now off-line
I 08/26/15 05:40:28 00076 ports: port B5 is now on-line
I 08/26/15 05:40:25 00077 ports: port B5 is now off-line
W 08/26/15 05:39:37 00564 ports: port I22 PD Invalid Signature indication.
I 08/26/15 05:39:36 00076 ports: port I22 is now on-line
I 08/26/15 05:38:53 00076 ports: port L9 is now on-line
I 08/26/15 05:38:51 00077 ports: port L9 is now off-line
I 08/26/15 05:38:34 00076 ports: port L9 is now on-line
I 08/26/15 05:38:32 00077 ports: port L9 is now off-line
I 08/26/15 05:32:52 00076 ports: port B8 is now on-line
I 08/26/15 05:32:49 00077 ports: port B8 is now off-line
I 08/26/15 05:32:35 00076 ports: port L23 is now on-line
I 08/26/15 05:32:34 00077 ports: port L23 is now off-line
I 08/26/15 05:32:24 00076 ports: port L23 is now on-line
I 08/26/15 05:32:22 00077 ports: port L23 is now off-line
I 08/26/15 05:29:36 00076 ports: port B5 is now on-line
I 08/26/15 05:29:26 00076 ports: port L22 is now on-line
I 08/26/15 05:29:22 00077 ports: port L22 is now off-line
I 08/26/15 05:29:15 00077 ports: port B5 is now off-line