LAN connection speed

[Kevin (Gonzo) Bushman]

OK. I hate to start another thread on this, I've done searches here and tried all the things I can think of to resolve this myself. The problem is I'm getting conflicting info from my system and slow FTP's to/from this box - even to another 11.11 box on the same segment!

This is an 11.11 box with igelan, 1GB cards. The card in question is set to 1GB, auto (verified manually via sam). I verified the switch is set the same. However...

FTP's out of the box seem to not be able to exceed about 9-9.5MB/sec, which is why I started looking at this config.

/etc/rc.config/hpigelanconf looks like this (the end of it anyway):

HP_IGELAN_INTERFACE_NAME[0]=lan1
HP_IGELAN_STATION_ADDRESS[0]=
HP_IGELAN_SPEED[0]=auto_on
HP_IGELAN_MTU[0]=1500
HP_IGELAN_FLOW_CONTROL[0]=1
HP_IGELAN_AUTONEG[0]=
HP_IGELAN_SEND_COAL_TICKS[0]=150
HP_IGELAN_RECV_COAL_TICKS[0]=0
HP_IGELAN_SEND_MAX_BUFS[0]=10
HP_IGELAN_RECV_MAX_BUFS[0]=1

Which supports that it's set to auto.

Now here's where I get conflicting info. I've verified that it is PPA 0, so, lanadmin gives me this:

#lanadmin -g 0 | grep -i speed
Speed = 10000000

or 10MB, while lanadmin -x 0 tells me this:

#lanadmin -x 0
Current Config = NO LINK AUTONEG

I've verified that it is lan1 that I'm referring to here:

#ifconfig lan1
lan1: flags=1843
inet ZZ.XX.90.3 netmask ffffff00 broadcast ZZ.XX.90.255

and lan1 is PPA 0 b/c if I go into lanadmin interactively and ask what ppa number I'm working with it tells me it's ppa 0:

#lanadmin lan1

LOCAL AREA NETWORK ONLINE ADMINISTRATION, Version 1.0
Mon, Nov 22,2010 16:00:56
.
.
.
Test Selection mode.

lan = LAN Interface Administration
menu = Display this menu
quit = Terminate the Administration
terse = Do not display command menu
verbose = Display command menu

Enter command: lan

LAN Interface test mode. LAN Interface PPA Number = 0

If it helps, lanadmin -g also tells me:

Administration Status (value) = up(1)
Operation Status (value) = down(2)

Which I also find interesting. Why does the Operation Status report down when this is a working NIC as verified with ifconfig above?

So, why on one hand do I get 10M, on another the current config says no_link all while this is the NIC we're using (i.e., it's running and the IP address is correct)?

From this, I understand why my FTP's are slow - we've got 1GB cards running at only 10MB. So, what do I have to do to get this to work at 1GB as it should? What am I missing here? We have some BIG FTP's coming up and 12-18 hours per file for 8-10 files is not going to work very well...

TIA - points will be given.

-Gonzo

[Patrick Wallek]

What does 'lanscan' show?

>>and lan1 is PPA 0 b/c if I go into lanadmin interactively and ask what ppa number I'm working with it tells me it's ppa 0:

That doesn't necessarily mean anything. I think lanadmin defaults to '0'. What happens if you change the PPA to 1?

What does 'lanadmin -x 1' show?

[Shibin_2]

Can you provide the output of netstat -in and lanscan?


>> LAN Interface test mode. LAN Interface PPA Number = 0

This is the default value you are getting. It is not correct one.

To get / check correct value,

lanadmin

lan

ppa ( default will be 0 )

enter 0 and then enter display.

After that, enter 1 and display.

Now, you will see the difference.

[Kevin (Gonzo) Bushman]

Then what's the point in providing the NIC instance on the command line (lanadmin lan1) if it's going to go to PPA 0 anyway?

Output requested below:

lanscan
Hardware Station Crd Hdw Net-Interface NM MAC HP-DLPI DLPI
Path Address In# State NamePPA ID Type Support Mjr#
0/0/0/1/0 0x00306E4AA466 0 UP lan0 snap0 1 ETHER Yes 119
0/0/6/0/0 0x00306EE90F20 1 UP lan1 snap1 2 ETHER Yes 119
0/0/12/0/0 0x00306EE91F96 2 UP lan2 snap2 3 ETHER Yes 119
michigan: / #lanadmin -x 1
Speed = 1000 Full-Duplex.
Autonegotiation = On.

michigan: / #netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lan1 1500 XX.ZZ.90.0 XX.ZZ.90.2 665421112 0 378822864 0 0
lan0* 1500 192.168.0.0 192.168.0.1 0 0 0 0 0
lo0 4136 127.0.0.0 127.0.0.1 2328307 0 2328308 0 0

And you're both right. Once I told it to look at PPA 0, it shows it's at 1GB.

Which leaves my question, why are my FTP's so slown then? I even tried (to test) an FTP over the weekend when the system was all but idle. (CPU at about 10%, I/O at 15-20%, etc.).

Thoughts?

[Kevin (Gonzo) Bushman]

Forgot to add, the tests with the system all but idle showed no increase in speed. So what's misconfigured then?

As for the output, in a different format directly from lanadmin for PPA 1:

Enter command: display

LAN INTERFACE STATUS DISPLAY
Mon, Nov 22,2010 17:10:58

PPA Number = 1
Description = lan1 HP PCI 1000Base-T Release B.11.11.27
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 1000000000
Station Address = 0x306ee90f20
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 1767133580
Inbound Octets = 3600354776
Inbound Unicast Packets = 1690392766
Inbound Non-Unicast Packets = 27167709
Inbound Discards = 0
Inbound Errors = 0
Inbound Unknown Protocols = 367
Outbound Octets = 893144508
Outbound Unicast Packets = 1411935800
Outbound Non-Unicast Packets = 2444898
Outbound Discards = 12106
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367

Ethernet-like Statistics Group

Index = 2
Alignment Errors = 0
FCS Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

[Shibin_2]

>> Then what's the point in providing the NIC instance on the command line (lanadmin lan1) if it's going to go to PPA 0 anyway?

lanadmin without options won't accept PPA. It simply execute lanamdin.

If you execute lanadmin -x , it accepts the whole part. Where as if you simply give lanadmin lan1, it avoid the last part.

>> why are my FTP's so slown then?

Now you know your LAN card speed is correct and there is no problem.

Is this FTP from other server or PC ? Have you checked the routing and access points to see any packet drops?

[TwoProc]

Simple answer: avoid "auto".

Set both the network port on your switch to full duplex, and do the same on your server. This will, in all probability clear up your problem right away.

Granted, it *could* be something else - but I've seen this problem lots of times on my servers (probably half a dozen or so), and setting everything to full duplex on both sides always clears it up.

In fact, for the last several years, its a required step in setup, and we've not seen the problem recur since.

[Patrick Wallek]

>>Simple answer: avoid "auto".

You can't with Gigabit NICs. HP-UX will not allow you to set the speed at 1000 FD on a Gb NIC. In order to achieve Gb speed, it must be set to AUTO.

[rick jones]

Start minimizing variables. For example, take disc and filesystem completely out of the picture by running a netperf TCP_STREAM test between the two systems.

Also, while it may not use HP-UX syntax, the attached is some of the things I look at when looking to diagnose performance issues.

[Keith Jahn]

Two software things to try. You can get chronic perfomance from ftp with large block sizes.

1) Try ftp with a small block size
ftp -B 1

2) Lower the block size on your ftp daemon in inetd.conf .

ftp stream tcp nowait root /usr/lbin/ftpd ftpd -B 32 -l



A couple of network hardware item to check.

1) Auto-negotiation or not any re-plugging of LAN cables with the power up can cause slow paths across a switch. A cold start of network then servers can clear this issue.

2) If any network component in the data patch has "Jumbo Packets" enabled this can be a performace disaster for ftp. It is a particular problem in mixed manufacturer networks (e.g. HP and Cisco).

[Bill Hassell]

> Keith:
> 1) Try ftp with a small block size ftp -B 1
> 2) Lower the block size on your ftp daemon in inetd.conf .

This doesn't match my experience and is counter-intuitive to the concept that small block sizes will require a much greater percentage of system overhead to handle the extra I/O requests. Using ftp -B 10000, a 500 Mbyte file transfer was significantly faster. Conversely, ftp -B 1 was not significantly different than using the default. This is probably due to the network driver coalescing small packets to reduce I/O counts.

> 2) If any network component in the data patch has "Jumbo Packets" enabled this can be a performace disaster for ftp. It is a particular problem in mixed manufacturer networks (e.g. HP and Cisco).

Again, larger packets means lower overhead, but this assumes that all the components along the network path understand jumbo frames.

[Keith Jahn]

@Bill Hassell
These tips are based on real world problems where FTP runs slow when it should run fast.

Lowering the ftp block size and buffer size can cure this issue where the target is a Microsoft platform or even a DEC unix server.

There are real problems with FTP negotiation on modern network hardware and a connection which might work perfectly on a 100 Mbit/s hub can run like a dog on cascaded 1000 Mbit/s switches.

[Kevin (Gonzo) Bushman]

All - I appreciate your replies and I've continued to research my issue taking some of your replies into account. Here's what I've found.

- Patrick is right - you have to set it to auto to get GB speeds. As shown, mine are set correctly. As I also stated, once I found to look at the (ahem) RIGHT NIC, it was connected at 1GB.

- Block size tests. I've run some sample FTP's between the various systems (all HP's, some at 11.11 - the source for example, some at 11.31 - the destinations mostly). The FASTEST I've been able to transfer is 26.5MB at the default 32768b block size and most transfers were only at 10-15MB/sec. I then ran additional transfers at both 10,000b and 50,000b block sized to run at both small and large block sizes (sftp -b 10000 and sftp -b 50000). These times were slower which resulted in slower transfer speeds. They also tended to stall more often.

- Load on the systems. The source system is fairly heavily loaded. That is, the CPU's are generally pretty busy (50-60% of capacity on average), so that MAY be a factor. I don't think memory is an issue as this system generally runs at about 60-65% of physical in use, and with "only" 16GB total (sorry, it's an old machine), that still leaves 6GB or so free. The destination systems on the other hand are not even in use yet. And they are 11.31 systems with much more memory. Also, I ran a speed test early Sunday morning when the source system wasn't busy (10% CPU utilization generally) and got the same speeds.

So, what else are we missing? Why can I not get anything better than 10-15% of the card's capacity (10-15MB/sec = 100-150Mb/sec, roughly or 10-15% of 1Gb)? What is preventing this from working at the rated speed? Could it possibly be as simple as the fact that I'm running sftp, not ftp?

Any ideas guys/gals?

-Gonzo

[Patrick Wallek]

SFTP could definitely have an impact due to the overhead required for encryption.

Would it be possible for you to try a test with straight FTP (not SFTP) and see what kind of speeds you get?

[Bill Hassell]

sftp is not ftp -- it is a completely different product and is actually slower than scp. ftp actually has a number of design features such as sending multiple blocks and not waiting on each block for an acknowledgment.

If CPU were a problem, there would be 100% usage by the sftp process. And memory would not be an issue as sftp (and ftp) don't grab big chunks of RAM for execution.

I would try ftp -B 1000 host_name just to see what the results are. I have also seen that multiple ftp sessions (3-5 total) going in both directions can exceed a single session throughput. You might use lanadmin to clear the stats and then count the results (lanadamin -g 1) when running several copies at the same time.

[rick jones]

What is netstat -s -p tcp telling you when bracketing an FTP (or better still netperf TCP_STREAM) test?

netstat -s -p tcp > before

netstat -s -p tcp > after
beforeafter before after > delta

where beforeafter can be had (for the time being, the server will be going away eventually) from ftp://ftp.cup.hp.com/dist/networking/tools/

[rick jones]

Keith - what exactly do you mean by "ftp negoitiation?"

[Kevin (Gonzo) Bushman]

So I did as suggested and tried some plain ftp's. Here's the results:

56 buffers (default) -> 15.65MB/sec
10 buffers -> 15.05MB/sec
64 buffers -> 14.94MB/sec

Even considering overhead, this is still only in the neighborhood of 150Mb/sec, far short of the 1Gb card's capacity. The destination system was all but idle. The source system was at about 50% CPU (on 8 proc's) and memory at about 63%.

-G

[Shibin_2]

I guess, enough testing at system side. How many devices connected to the switch, where this server connected? Have you tried to replace the cable with new one or with anyone that delivering better speed to others?

[rick jones]

If his cable was bad that should have shown-up as errors of some form or another in the lanadmin statistics.

[Shibin_2]

I know Rick. We had similar kind of issues at storage. We can access disks etc, but data transfer is very slow. Later found it was problem with cable. But there were no errors anywhere.

[rick jones]

That is just.... wierd. Unless there were bugs in the statistics, a bad cable should lead to error stats incrementing. Or, in your case, did the cable simply not "train" to the higher speed? Even then, the statistics should have shown the lower speed.

[Shibin_2]

It was all in a sudden slow response. First FC link failed found the issue at Gbic. Changed this and started slow response.

[Kevin (Gonzo) Bushman]

Sorry, all, but not much has helped here. Gonig from sftp to ftp only gave us a small increase in speed, 10MB to 15-16MB. Still not anything near where it should be. My next step is to talk to our network guy to see what he can tell me from his end.

Points given.

-Gonzo

[Bill Hassell]

Another test is to bypass networking completely and just use a single LAN cable between two systems with 1Gb cards (no switch). Ideally the other system can keep up with the 1Gb NIC since network speeds are dependent on both ends. This is the fastest way to prove that there is an HP-UX or NIC problem, or a network (switch, routers, etc). Also make sure the network path doesn't go through some QoS or other packet prioritizing appliances. Your throughput will be the limited by the slowest component in the chain.