1000Base-SX slow

[Jeff Smee]

I have an N4000-55 running HP-UX 11.00.
I have an A4926A card installed and I am only getting 12MBytes/sec out of it.
It is running at 1000 Full Duplex.

Any ideas?

Jeff

[Patrick Wallek]

What is your switch set to? Is it also 1000FD? Are you manually setting the card and switch or using auto negotiate?

[Jeff Smee]

Sorry for the triple entry.

Jeff

[Jeff Smee]

lanadmin -x says...
Speed = 1000 Full-Duplex
Autonegotiation = On

Not sure about the switch.

[Steven E. Protter]

I don't have one of these cards.

I do believe that I saw docs saying that contrary to slower HP NICs, both switch and card must be configured to auto negotiate.

lanadmin -x #

replace # with the lan number of the card.

SEP

[Steven E. Protter]

You and I were posting at the same time.

I believe that its important that the switch be set to Autonegotiate. If its not, it could result in performance problems. Contact your network folks. Its a simple change on a Cisco switch, but its the default.

SEP

[Michael Tully]

Your best bet is to set the switch to the same as your lan card, DON'T USE AUTONEGOTIATE. Hard code as 1000 Full Duplex autonegotiate =off both ends, that way it is set in concrete, if that is what the switch can handle. If not, then set it accordingly.

# lanadmin -X 1000FD 0 (will set it to 1000FD)
# lanadmin -X 100FD 0

Make sure that update the appropriate config file in /etc/rc.config.d or you'll lose your settings on the next reboot.

[doug mielke]

I thought I'd read that the 1000 base cards can not be set to auto off.

Thus forcing us to go through the pain of having the network folks confirm their auto settings on their gear

[Jeff Smee]

I also read that this card has to have autonegotiate turned on. i.e. you can't set it to 1000FD. However it is running at that speed.

[Jeff Smee]

http://www.hp.com/products1/unixserverconnectivity/adapters/ethernet/specifications/gigabit_ethernet_specs_print.html

[rick jones]

12 MByte/sec using what sort of transfer method? What sort of system(s) are receiving the data? Are you sure there isn't a 100BT link somewhere between the two?

What do your netstat -p tcp statistics look like (ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt)

What do the lanadmin statistics look like?

[Jeff Smee]

I've tried cp, tar and ftp. 12MB/sec is the best I can do. I also have 2 RP7410's that get the same result. And I am sure there is no 100Mb in the loop.

[Geoff Wild]

We had this same issue - you have to get your network support to set "Auto Negotiation On" on the switch. The strange thing is, there is only one setting for a 1000SX card 1000 Full Duplex. That said, it seems the HP wants to see the aut-negotiate flag on the switch....

We had an HP tech onsite - and that was the only solution - set auto-negotiate on the switch.

Rgds...Geoff

[rick jones]

i'd suggest getting filesystems and discs completely out of the equation - http://www.netperf.org/ run the TCP_STREAM test, and use:

$ netstat -p tcp > /tmp/before_56K
$ netperf -H -l 30 -- -s 56K -S 56K -m 32K
$ netstat -p tcp > /tmp/after_56K
$ beforeafter /tmp/before_56K /tmp/after_56K > delta_56K

and
$ netstat -p tcp > /tmp/before_128K
$ netperf -H -l 30 -- -s 128K -S 128K -m 32K
$ netstat -p tcp > /tmp/after_128K
$ beforeafter /tmp/before_128K /tmp/after_128K > /tmp/delta_128K

Where remote is the remote system where you have a netserver running, and beforeafter is the utility you can retrieve from ftp://ftp.cup.hp.com/dist/networking/tools/

Then post the output of each netperf run, and the delta file from before after and we'll see what things look like. Also include a description of the system actingas the target of the netperf test.

[Steven E. Protter]

We seemed to have read the same doc.

Two choices right now.

If you have a software contract, contact HP.

Or seriously check into those port settings with the switch admin.

Thought of another idea.

From a remote unix box. ping the nic. Watch the ping times. If they continually rise, the card might just be bad.

SEP

[Jeff Smee]

I got the beforeafter program, but I can't find netperf on that site.
Does someone have a link they know works.

Thanks
Jeff

[Jeff Smee]

I'm patching one of my systems now. But I have the same problem on all my HP's with Gigabit cards. I do have software support, but I know what HP will say. They'll say it's the switch. I have opened a case with Cisco to see what they say about our switches. Ping times seem fine (all 0ms).

Jeff

[Patrick Wallek]

I had an issue today with some of my 100Mb cards. NFS performance between a couple of servers was pitiful. One card was hard set at 100FD the other Auto Neg'ed to 100FD. The problem was that the switch was set to Auto Neg, but it just negotiated 100HD. Fixed the switch and all was well.

So I guess my advice is that the first thing you should do is talk to the network guys, have them actually log into the switch and see what the settings are. Make absolutely sure it auto negotiated the correct settings.

[rick jones]

netperf would be under ftp://ftp.cup.hp.com/dist/networking/benchmarks/ - that is where http://www.netperf.org/ should be pointing - if that is amis, please let me know and I'll see what might be wrong.

[Jeff Smee]

netperf revealed quite a bit. Now I think the problem might be disk. I have Ultra160 LVD SCSI, but if I copy a file from disk to disk I get 10Mbytes/sec. I also have a fibre attached EMC SAN. If I copy to it, I get 17Mbytes/sec.

Any one have any ideas now?

Jeff

[rick jones]

i'm much better at networking than mass storage, but from what I do know, LVD SCSI can also fall-back to lower speeds, depending on the discs involved.

instead of going disc to disc, i'd probably go disc to /dev/null and /dev/zero to disc with dd - if nothing else that will more closely approximate what is going-on with FTP.


as far as EMC's and the like, my understanding is that with most of those large SAN things if you can stay in their caches they do quite well, but once you have to start going to back-end discs things slow-down cosiderably. there may also be some RAID level issues particularly with write performance.

as for the other autoneg failure mentioned by someone else, if both sides were set to auto and once did FD and the other HD the bug could be on either end. however, when one side is hardcoded (to either FD or HD), the spec requires that the side set for auto go to HD. hardcoding means that side will not respond to any autoneg stuff (iirc), and failure to get autoneg stuff means the autoneg side must presume autoneg isn't supported, and since FD came along with autoneg, it then presumes that the other side is only capable of HD. so the spec says, autoneg failure -> go to HD

[doug mielke]

An easy test of the EMC speed is a sar -d
My N class internal scsi's and my EMC / Clarian times run 20 to 30 ms.

My Compaq EVA SAN is always in the single digit.

Not a test of throughput, only response time ( of the cache on those arrays)