10GbE Intel card will not connect to procurve 6120XG

[Andrew Bialock]

I have an intel 82598 based 10000Base-SR (10G) fiber card running linux and intel driver version 3.1.5. It will not link with the 6120XG procurve blade switch running ver Z.14.26. If I plug the card into a Cisco Nexus 5000 or Nexus 7000 it links just fine and runs. No such luck with the HP

If I issue a reboot to the 6120XG switch ("reload" from the command line), the card will link up for about 2.9 seconds as evidenced by this log entry:

[6553915.964818] ixgbe: eth2: ixgbe_watchdog_link_is_up: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[6553915.965006] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[6553918.801094] ixgbe: eth2: ixgbe_watchdog_link_is_down: NIC Link is Down

autonegotiate does not seem possible with the driver and ethtool will also not set the advertising or autonegotiate properly.

Can anyone shed some light on what might be wrong?

[rick jones]

Some of the usual, pedantic questions:

1) you sure it is a 10GbE SR transceiver in the 6120?

2) you sure it is a supported 10GbE SR transceiver in the 6120?

3) have you tried more than one port in the 6120?

4) have you tried more than one SR transceiver in the 6120?

5) does any other 10GbE SR card connect successfully to the 6120?

6) can you successfully connect (being careful not to create loops) the 10GbE SR port on the 6120 to either your 5000 or 7000?

[Andrew Bialock]

Some of the usual, pedantic questions:

1) you sure it is a 10GbE SR transceiver in the 6120?

Yes. Positive. J9150A

2) you sure it is a supported 10GbE SR transceiver in the 6120?

Yes. From HP. Part number on quickspecs for 6120XG. It is a J9150A.

3) have you tried more than one port in the 6120?

At least ports 17 thru 24 on 10 different 6120XG switches with various code versions up to and including Z.14.26. None work.

4) have you tried more than one SR transceiver in the 6120?

Yes, 4 different SFP+. All of them are J9150A.

5) does any other 10GbE SR card connect successfully to the 6120?

Don't have another card to try but I tried 2 different SFPs in the Intel card. One from Finisar and one from Avago.

6) can you successfully connect (being careful not to create loops) the 10GbE SR port on the 6120 to either your 5000 or 7000?

Yes. The 6120XG will connect to the Cisco Nexus 5000 or 7000 with any combo of SFP+ SR and aqua 10G cable or twinax cables from both HP and Cisco.

My issue is that client "A" with the Nexus hardware has the configuration: Intel card <--> Nexus <--> HP 6120XG and works flawlessly. Whereas, client "B" has no Nexus equipment so it is configured as: Intel card <--> 6120XG and Linux will not get link with the HP switch. Although the HP switch claims the port is "UP" and link is active, linux thinks it is down.

If I go back to client "A" and directly connect the 6120XG to the Intel card (effectively bypassing the Nexus equip), the connection will not come up and the linux machine thinks the link is down with the same symptoms as client "B".

I've tried about 4 different Intel ixgbe driver versions and 4 different linux kernels versions. All work when connected to a Nexus 5000 or 7000 with any of the SFP+'s and none work with the 6120XG.

The only link I have ever seen on linux is part way thru the 6120XG boot process - about 20 or 30 seconds before the telnet service is enabled on the switch - when the card connects for almost 3 seconds before being disconnected.

I read a note from Intel that 82598/82598EB cards will fail to link in a K/KX configuration when one side does auto-negotiate and one side is set to fixed speed and duplex. I see no way to turn ON auto-negotiate on the Linux side and no way to turn OFF auto-negotiate on the HP 6120XG side. The question becomes, "If during the boot process the HP 6120XG gets link because it assumes 10000/full, why does it then decide to switch to auto-negotiate 2.9 seconds later?"

Thanks in Advance for any insight,
-Andy

[rick jones]

Thanks for answering the pedantic questions :)

I'm not familiar with the guts of the 6120, but at one level at least, switches can be somewhat like hosts, and in a host, at first the "card" is driven by firmware, and then at some point in the boot process the "OS" driver takes-over and does its thing. In that sense a later auto negotiation does not seem all that surprising.

I'd always thought that auto negotiation was required per the 10GbE specs.

[rick jones]

In my guise as "Mr. Netperf" (and only that, no official HP promises of any sort here...) I took the liberty of contacting someone I know at Intel. That person expressed a desire to see the output of

a) ethtool -e ethX

and

b) ethregs

where ethregs is available via sourceforge. He also speculated about the card perhaps not driving the tx laser on and asks if you might shine each end of the cable onto a piece of paper - once when one end is connected to the card and once when one end is connected to the switch and make sure that both sides have tx laser on.

He also asked about success or failure when using DAC instead, assuming the distances involved are amenable to it.