Bottleneck on 1000Base-T Adapter

[Stafford Hyacinth]

Glance alerts claims there is a network bottleneck. The speed is 1000 Full-Duplex. I looked at from the network side and the port utilization goes to 100MB but glance is crying of bottleneck when that happens. The switches are hard set to 1000 Full. Is this just a glance bug?

[Pete Randall · Accepted solution]

The standard recommendation with Gigabit NICs is to set both ends to auto-negotiate. In fact that's the only way, supposedly, that they'll work properly.


Pete

[Mel Burslan]

as Pete said, if you set the switch speed to 1000FD fixed, your Gig-E card will be stuck at 100HD as it will see the other side non-negotiable and thinking 100 HD is the best it can do.

Let your network admins relase the 1000 FD setting on the switch first. Then you check glance to see if the bottleneck s still there.

[Florian Heigl (new acc)]

Of course glance will consider the NIC a bottleneck if You hit 100MB/s, as this is very close to the line speed maximum.

You should be very happy to actually see such high performance, if You need more, You'd have to add a second NIC and use PortAggregation, or switch to something generally faster like Infiniband.

(I don't recall if there's already a 10GE adapter supported by HP-UX and the switch port cost would be ugly ;))

But unless Your applications really demand more than 100MB/s (imaging, supercomputing), just be happy.

[Jay Kidambi]

While it is true that the standard recommendation for Gigabit Ethernet operation is to set both ends to Auto-Negotiate -- the setting where the link partner is set to 1000FD mode is also fully supported by HP-UX Gigabit Ethernet NICs. Mel Burslan's statement that using 1000FD on the switch causes the HP-UX side to operate at 100 Mbps is *not correct*. The fact that you are getting 100 MB (~800 Mbps) shows that you are operating at close to gigabit wire speed. Not sure how glance determines that the link is a bottleneck -- it may just be complaining about sustained high utilization.

[Jay Kidambi]

The following clarification regarding my previous posting is in order:

Though the '1000FD fixed' option on link partners is supported by HP-UX, HP-UX Gigabit Ethernet products don't have an explicit 1000FD option. When the switch port is set to 1000FD, you must use the 'auto_on' setting on the HP-UX side.

Hope that helps.

Jay

[Bill Hassell]

Glance "bottlenecks" are a big nusiance because they imply that something is wrong when it is just an ordinary measurement passed through an advisor script. When you buy a computer, you want to take advantage of all the capability, so 100% CPU, 100% RAM, 100% disk and 100% network is not necessarily a bad thing. There's nothing wrong with this shell script:

while :
do
:
done

Yet it will consume 100% of a CPU. Is that bad? Not really. Like all multi-tasking opsystems, multiple CPU-bound programs are timeshared among the available CPUs. RAM and disk usage have been discussed several times and they are more involved, but LAN usage is pretty simple. You paid for 1000Mbits and you can expect data to flow at that speed. If you transfer a large file (say 100 megs) then you can reasonably expect 100% LAN usage although actual throughput depends on CPU speed, some overhead in the LAN packets and how busy the OS is in handling other tasks.

So take the advisory for the simple measurement that it is. A long FTP transfer will have no measureable effect on telnet connections, but if you start 5 or 10 FTP transfers at the same time, the LAN will be about as busy as it can get and each FTP transfer will be sharing time on the LAN, trhus lowering the apparent throughput for individual jobs. A simple way to measure throughput is to simply transfer a large file, mweasure the elapsed time and calculate the throughput. Anything around 5-7 Mbytes/sec is good (1000 Mbits ~= 100 Mbytes) And note: if your disks are busy (on either side), they will slow down a file transfer test over the LAN because the disks can't handle data fast enough.

Now if you see a very large packet count on the LAN and you know there aren't any jobs performing large transfer, then it's time to look at the traffic to see why the LAN is busy. You'll need netstat and a freeware tool: lsof to track down the owner of network sockets.

[Ted Buis]

There is a 10Gigabit card for newer servers, but don't expect 10X throughput improvement. You also need a 10Gigabit switch.

[Stafford Hyacinth]

Thanks all. I am going to set it to auto negotiate on both ends and check performance. I plan on adding APA later but mostly for redundancy.