Gig Nic (igelan driver) speed problems

[AEL]

Hi Everyone -

I've followed many discussions based around this issue, but have yet to figure out the solution to our issue at hand. We have 2 systems, each with a 1000Base-T Nic card set to AUTO connected to a switch set to AUTO. Each have Cat 6 cable run from the servers to the switches. These systems are both 11.11 and are new rp7420s.

The problem we're having is that we aren't getting even close to typical speeds for a 1000Mb connection. An SCP of a 4 GB file between systems shows average speed of 7,200 Kbytes/s. This is typical of some of our other servers with 100FD Nic setups.

Running the test below between these same systems shows the proper speeds.

ftp> binary
200 Type set to I.
ftp> put "|dd if=/dev/zero bs=1024k count=10240" /dev/null
200 PORT command successful.
150 Opening BINARY mode data connection for /dev/null.
10240+0 records in
10240+0 records out
226 Transfer complete.
10737418240 bytes sent in 123.64 seconds (84810.45 Kbytes/s)
ftp>

Anyone have any suggestions to get the most out of these cards, or how we can get close to what the above test shows for transfer rates? We are entering a upgrade phase and have to backup roughly 600 GB of data and need the fastest backup possible.

Thanks

[Steven E. Protter · Accepted solution]

Shalom,

Well you've already looked at the obvious factors, like switch settings and system configuration.

Factors not covered yet:
1) Bad LAN card.
2) Bad cable.
3) Bad switch port or other issues on the network.
4) OS patches for these cards or network i/o issues.
5) Network configuration issues, like lack of /etc/hosts reference to the two systems or a slow dns server.

Its not easy for fun, but these issues need to be looked at now.

In your situation, I'd at the least put into production the latest bi-annual patch set and perhaps even look for specific patches for the hardware for your OS (not stated btw)

I'd replace cables and investigate these less likely issues and see which one improves the situation.

Note, I agree your speeds are slow, but not outrageously slow. These cards never perform at 100% of theoretical speed.

SEP

[Bill Hassell]

Change you cabling to a crossover between the systems and measure again. That will eliminate a poorly performing switch. For the fastest performance, consider APA (auto port aggregation). Note that if these systems are on different subnets, your router(s) will be a serious bottleneck. You can also configure jumbo frames to lower the overhead with small TCP/IP packets. You'll need to use production quality backup program (NOT legacy tools like cpio, tar, etc, or NFS).

[skt_skt]

I hope you have stated NIC card (spped)set to AUTO; what about the duplex ;

could you send the output of
laadmin -x ppa and netstat -in from both the systems.

Is that lan cards configured for a fail over through SPOF .

[Steve Lewis]

The problem is scp. It is a very slow protocol. If you simply ftp'd the file you would get far better throughput - like your dd test (nice test btw I never knew you could do that!).

If your file is compressable you may get better throughput if you enable compression.

Another option if you really need the username/password encrypted is to try sftp. I don't think that will make much difference. But sftp doesn't have the compression option that scp has.

I am not sure why scp is so slow, it may have something to do with a lack of random data or something (???? comments ).

If you just want the data encrypted and aren't worried about the username/password then download gnupg (= gnu pgp), gzip then encrypt/gpg the file locally and use ftp for the fastest transfer.

[AEL]

I'm going to try and answer these in order - BTW thank you for your assistance.

SEP: I don't think this is an issue of bad NIC, because we have 6 servers (3 rp7420 + 3 rp3440) all with the same issue. As far as cabling goes, we have had all 6 of these servers run with new Cat 6 cable and we've had our operations team trace the cables from the systems to the switch several times to make sure they are correct. As far as other network issues - anything is possible, but our network team doesn't seem to think the network is the bottleneck. Do you know what patches are relevant or have a link I can follow for guidance? I seem to see alot based around TACH (fiber) patches, which won't apply for our case.

Bill: If we connect crossover - won't the best speeds we see be limited to those of a 100FD? I'm not aware of crossovers being able to achieve higher. I'll follow up with networking to see if we can do this. I'll look into APA as well. We haven't configured any of the TCP options yet, and our networking team tells us we can't change our MTU because we'd have to change it across the whole network (each point of contact aka switch, router, backup server) and they're afraid it would effect our other servers. We're currently using IBM's TSM for our backups. It is connected to the network via 1000Base-t SX fiber.

Santhosh:
# lanadmin -x 1
Speed = 1000 Full-Duplex.
Autonegotiation = On.
# netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lan1 1500 10.100.16.0 10.100.19.57 588607139 0 1818048676 0 0
lan0* 1500 10.100.24.0 10.100.27.30 481654 0 1732551 0 0
lo0 4136 127.0.0.0 127.0.0.1 753232 0 753232 0 0


Steve: I used SCP just to see the transfer speeds as the file was copied from server to server. I wanted to know if there were spikes in the transfer or if it stayed steady. FTP moves the data file in the same amount of time but doesn't show the on-going transfer rate.

ALL: What TCP settings are optimal for Gig copper NIC's?

[rick jones]

The point about scp being a slow protocol is probably on the right track, given you seem to get decent performance out of an FTP transfer (although I myself would use netperf to check the basic networking, not ftp :)

With scp, it _could_ be the overhead of the encrytion/authentication (although for a 4GB file I would guess the encryption). It could also be that the intra-scp windowing (IIRC there is some if my recollections of openssh email are correct) isn't large enough for a high bandwidth delay link.

So, while you might consider increasing stuff like tcp_xmit_hiwater_def (IIRC that is the name for the default TCP SO_SNDBUF size as it were) you also need to look into the version of scp you are running. What HP-UX 11.11 probably ships is likely rather old and may pre-date the high bandwidth delay changes.

Soooo, two things to look at:

1) the CPU util of _each_ CPU on _both_ sides while running the scp

and

2) a tcpdump trace of the scp transfer, with an eye towards looking at how much data is ever outstanding on the connection at any one time.

You should know that HP-UX FTP defaults to asking for a 56KB (historical reasons) socket buffer/TCP window. The default TCP window in 11.11 is 32768 bytes, and 32768 bytes isn't quite enough for a gigabit link. You can confirm that with netperf TCP_STREAM, or by passing some -B options to your ftp client and daemon.

[AEL]

Hi Rick -

Thanks for the input. It's a little bit off track though and maybe that's my fault. I'm not worried about using SCP or any encryption for that matter. I used SCP because it shows transfer rates as files are moved from one server to another. Running an FTP wasn't any faster than this method. The original post I made showing the FTP protocol in use was transferring 10GB of between my two systems from RAM to RAM, basically taking the disks/filesystems out of the picture. Doing this showed 90,000Kbytes/s. The SCP/FTP of a DATA FILE from any filesystem is averaging around 9,000Kbytes/s. I'll try testing using the netperf to get more accurate results.

Our main concern is utilizing the GIG NIC such that running a backup of 600GB of data to our Tivoli Storage Manager (backup system) transfers at a rate faster than 9,000Kbytes/s. Our 100FD NIC's produce this same speed so it doesn't make sense for our GIG NICs to be performing at the same level. I understand we won't get speeds even close to 90,000Kbytes/s but it would be great to even double the transfer rate that we say from our 100Base-T NICs.

[rick jones]

So, if I'm parsing correctly, things without the filesystem appear to be OK perfwise over your GbE, things with the filesystem are not.

I presume you've already run tests to show that the filesystem runs >> 10 MB/s for what you do with/to it?

While I don't think that scp uses sendfile(), I know that FTP will try to use it. Still, when/if you try netperf, you may want to compare a TCP_STREAM test with a TCP_SENDFILE test.