gigabit lan not working

[Burvil Chang]

I have a system that's seeing a lot of network latency, so I'm trying to see
if I can configure the gigabit port. I've outlined the steps below that I
took. For some reason, even though it works when set to 100 full, it doesn't
work for gigabit even when set to auto.

1. Stop the network on the RP8400:
bash-3.1# /sbin/init.d/net stop

2. Configured the switch (I know this isn't a Cisco board, but I include it
here for completeness). I know gigabit on this switch works, since we have gigabit configured for some other hosts on the same switch (not hp-ux).
SOME-SW3#config t
Enter configuration commands, one per line. End with CNTL/Z.
SOME-SW3(config)#int gi 9/20
SOME-SW3(config-if)#speed 1000
SOME-SW3(config-if)#exit
SOME-SW3(config)#exit
SOME-SW3#write
Building configuration... [OK]

So the result looks like this on the switch. Note that it says 1000 Full duplex.

SOME-SW3#sho int gi 9/20
GigabitEthernet9/20 is down, line protocol is down (notconnect)
Hardware is C6k 1000Mb 802.3, address is 0018.7354.8973 (bia 0018.7354.8973)
Description: some-hostname lan0 U1-4
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA, loopback not set
Full-duplex, 1000Mb/s


3. Changed the /etc/rc.config.d/hpgelanconf file, so the speed is AUTO_ON. These are my settings.

HP_GELAN_INTERFACE_NAME[0]=lan0
HP_GELAN_STATION_ADDRESS[0]=
HP_GELAN_SPEED[0]=AUTO_ON
HP_GELAN_MTU[0]=1500
HP_GELAN_FLOW_CONTROL[0]=1
HP_GELAN_AUTONEG[0]=1
HP_GELAN_SEND_COAL_TICKS[0]=1000
HP_GELAN_RECV_COAL_TICKS[0]=0
HP_GELAN_SEND_MAX_BUFS[0]=16
HP_GELAN_RECV_MAX_BUFS[0]=1


4. Restart the network, after which I get a bunch of errors:

bash-3.1# /sbin/init.d/net start
ERROR: Failed to add route entry because its interface is not
yet initialized. May need to add this route entry with
a route commad after the interface is up :
add net default: gateway x.x.x.x: Network is unreachable
ERROR: Failed to add route entry because its interface is not
yet initialized. May need to add this route entry with
a route commad after the interface is up :
add net y.y.y.0: gateway x.x.x.x: Network is unreachable
ERROR: Failed to add route entry because its interface is not
yet initialized. May need to add this route entry with
a route commad after the interface is up :
add net z.z.0.0: gateway x.x.x.x: Network is unreachable


5. I've since re-set it back to 100fd on the switch and
/etc/rc.config.d/hpgelanconf. It's working properly, but I would like to get
this working. This box is currently in production, although I should have a
similar one set up soon I can test with. In the meantime, is there something
I'm missing here? Anything I can a) try without disrupting network
connectivity, b) try on a test system or on this current when I have a
downtime for it? (The system is currently down for maintenance, but won't be much longer - see below).

FYI, here's the current settings. Note that it's currently down, and will be until some services this machine depends on are up. I'm not sure when that will be, but I don't think I'd be able to think of anything or get a response on this post before I have to bring it up again.
bash-3.1# lanadmin -s 0
Speed = 1000000000
bash-3.1# lanadmin -x 0
The link is down. The speed and other link state information
are undefined. The configuration specified for this card is:
Speed = 100 Full-Duplex.
Autonegotiation = Off.

bash-3.1# ioscan -fnC lan
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
lan 0 0/0/0/1/0 gelan CLAIMED INTERFACE HP A6096-69001 PCI 1000Base-T Built-in I/O

[Mark Nieuwboer]

Hi,

even if the switch is gigabit. still the port on the switch can be configured for 100 full duplex.
Ask your network guys to look for that.

grtz. Mark

[Burvil Chang]

even if the switch is gigabit. still the port on the switch can be configured for 100 full duplex.
==================
Yes, I have admin access on the switches, and I configured the port for 100 full. That's how it's working now. I want it -

a) preferably gigabit, since it *is* a gigabit card, or, failing that,
b) any other way to check if the network's misconfigured.

FYI, netstat -i shows no errors, inbound or outbound.

[rick jones]

When running Gigabit, do not hardcode the duplex settings on _either_ end of the cable. Heck, unless you know the Cisco switch is completely and irrevocably fubar wrt autonegotiation, and cannot be replaced with a working switch, I'd leave everything (100BT too) at auto.

Stepping back a bit - when you say you have latency issues, what sort of issues and of what magnitude? Just how loaded is your NIC when running in 100BT mode? What is the typical packet size through the interface? lanadmin stats might be a place to start - take them over an interval and run them through beforeafter:

ftp://ftp.cup.hp.com/dist/networking/tools/

Also, take some snapshots and see what is reported as the outbound queue length.

Finally, what is the _per CPU_ utilization on the system - especially for the CPU taking interrupts from the NIC? (intctl will tell you which)

At the risk of preaching to the choir, attached is some boilerplate on autoneg.

[Burvil Chang]

When running Gigabit, do not hardcode the duplex settings on _either_ end of the cable. Heck, unless you know the Cisco switch is completely and irrevocably fubar wrt autonegotiation, and cannot be replaced with a working switch, I'd leave everything (100BT too) at auto.
==============================
Yeah, that's what I'd figured, too. But my coworkers tell me that they've noticed from time to time the switch will get confused if it's set at auto. I'll bring this up again, and try it both sides with auto.


Stepping back a bit - when you say you have latency issues, what sort of issues and of what magnitude?
=========================
It just takes a long time to transfer files... not sure how to quantify it, but I'll try in this posting.


Just how loaded is your NIC when running in 100BT mode?
==========================
I transferred a ~500MB file via ftp. The result was 509985396 bytes sent in 47 seconds, or 82 Mbps. I guess that's reasonable for a 100 MBps connection ... perhaps the slowness may be restricted to ssh, which is primarily where I've noticed it. The version is somewhat old (openssh 4.2p1), but as I figure not fixing what's not broken, is there a way to check the viability of an installed package?


What is the typical packet size through the interface?
===========================
Not sure how to measure this.


lanadmin stats might be a place to start - take them over an interval and run them through beforeafter:

ftp://ftp.cup.hp.com/dist/networking/tools/
===================
OK, I've downloaded beforeafter, but how do I gather statistics through lanadmin? I go to the menu, then display, and get some statistics. Are you saying do that repeatedly a couple times and have beforeafter process the text?



Also, take some snapshots and see what is reported as the outbound queue length.
===================
I've done that, again, during the time that the 500 MB file was transferred. The outbound queue length was always 0.


Finally, what is the _per CPU_ utilization on the system - especially for the CPU taking interrupts from the NIC? (intctl will tell you which)
=================
What's the pathname for that command? I can't find it on my box, and man intctl returns nothing.

[rick jones]

82 Mbit/s over 100BT is in the reasonable range.

netstat -s -p tcp before and after a transfer and see if there are many retransmissions.

lanadmin -g mibstats will allow you to avoid going through the menues:

lanadmin... > before;sleep N; lanadmin... > after

beforeafter before after > delta

(ditto with netstat -s -p tcp)

if there is no intctl on your system, check the patches. ISTR it was available as a patch to earlier 11iv1

[Ben Dehner]

At risk of starting a flamewar -- or maybe in hopes of one -- I'd recommend that you should fix both sides of the connection to 1G/Full. We've had problems in the past with Cisco switches and auto-negotiate, and find that -- with Cisco -- it's better off to use a fixed port speed. Some 3rd party network providers, when installing Cisco switches on premises, require that all links to their switches have fixed speed settings.

It's very also important that both the switch and the lan interface be the same. If one side is set to auto-negotiate, and the other is not, you are going to have lots of problems.

[rick jones]

No worries about a flameware. I'll just say that in my personal opinion, there should be no reason any contemporary gigabit ethernet equipment shouldn't be able to autonegotiate properly.