nic performance

[Ron Irving]

Hi!!

I have an rp7410 with 2 partitions. 1st partition running 11.23, and the 2nd running 11.31. We have had an ongoing problem with the NIC perfomance on the 11.23 side. lanadmin -x 0 shows 1000, autoneg on. We can't seem to get more than 10MB/s out of it. The cable and the switch are all ok, as we have tested both on the 11.31 partition.

This has been an ongoing problem. We just commissioned this server after running into the same problem on an N4000 with a GB NIC installed. Couldn't get more than 10MB/s on that one either. I was thinking maybe something in the setup or config is off.

If you need any config files, or more info, please let me know.

thx

[Hein van den Heuvel]

What did you use to verify the speed and path?

- traceroute
- MTU size / Jumbo frames : netstat -in

ttcp - ftp.arl.mil/ftp/pubttcp
netperf - www.netperf.org

FTP? get a large incoming file to /dev/nul?
I assume that you used the same test on both partitions and thus ensured the source and sink are up to the task?

NFS? has its own set of parameters.. are they ok? And NFS got better with 11.31.

Find a recent version of David Olkers NFS tuning documents ? (recent HPTF proceedings 2002 flavor in
http://docs.hp.com/en/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf )

Good luck!
Hein.

[Ron Irving]

Hi and thanks!!

We are using netbackup to restore a DB to an attached SAN. The backup itself will take about 4 1/2 hours on average. The restore takes 20+ hours. Like I said, through the 11.31 partition, just using scp as a test, we're getting 40MB/sec, but using the same file to transfer via scp on the 11.23 side, 10MB/sec tops.

[Ron Irving]

MTU size is 1500

[Robert-Jan Goossens_1]

Hi Ron,

Could you post the iocsan -fnClan?

I don't know if you have a spare nic port on both partitions, you could try to eliminate the switch. Just use a cross-over cable to connect both nics and setup a private network between the nics.


On other thought is the GigEther-01 (igelan) software.

http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=GigEther-01

Defects fixed in B.11.23.0809 release:

* QXCR1000591671: Support for 1000FD speed on IGELAN

Regards,
Robert-Jan

[Ron Irving]

Ok...here it is.

[Robert-Jan Goossens_1]

Are you using cat5E or cat6 cables?

[Ron Irving]

Cat5 and everything checks out.

[Hein van den Heuvel]

>> MTU size is 1500

On both systems right?
Please consider Jumbo-frame at 9000 bytes.

>> We are using netbackup to restore a DB to an attached SAN. The backup itself will take about 4 1/2 hours on average. The restore takes 20+ hours. Like I said, through the 11.31 partition, just using scp as a test, we're getting 40MB/sec, but using the same file to transfer via scp on the 11.23 side, 10MB/sec tops.

Sorry to be a pest, but you did not explicitly indicate that the performance difference is also visible during netbackup?
OR is the backup from 11.31 and the restore to 11.23? Still for performance backup != restore.
I'm sure it does, and that is probably what prompted you to investigate, but still...

Maybe try one more tool other than scp, or establish the core performance with a foundation tool like ttcp?

There are a lot of tuneable to check, depending on the protocols used. Settings like tcp_sack_enable, and whether the driver supports 'trains' of packets:
# ndd â get /dev/ip ip_ill_status
look for 'train'

A most excellent paper describing it all, but maybe overwhelming is: http://docs.hp.com/en/11890/perf-whitepaper-tcpip-v10.pdf
Dig in!

Regards,
Hein.

[Ron Irving]

I'll try to be a little clearer. We backup our production server, (rp8420,) every night, approximately 400GB. It takes between 4 and 6 hours to complete this. I am restoring to the dev system, (rp7410,) using that image created by netbackup. Again, it's taking 20+ hours to complete the restore. Our backup admin cannot find any problems on his side. Everything looks good on my end too.

Should I go ahead and change the MTU value to something else? Is this a dynamic change? I know it's a lanadmin setting...can you give me the command?

thx

[Duncan Edmonstone]

How do you *know* this is a network problem?

Cos all the tests I can see here will have taken disk access into account as well...

You need to isolate the problem so you can eliminate other variables. Start by testing the speed of the network only.

Get a copy of netperf installed on the 11.23 partition and wherever the Netbackup data is coming from and run a simple TCP_STREAM test between the 2. Netperf is easy to use and only takes 15-20 minutes to install and setup. Downloads and manual available here:

http://www.netperf.org

You will need a compiler to build it though...

HTH

Duncan

[Hein van den Heuvel]

Right Duncan. That's the gist of my replies, and that's why I mentioned using /dev/null,
or /dev/zero to sink or source lots of data for free.

The scp tests are a step, but just a step.

Of course in the end tests are just that, tests. And only the real mcCoy counts (backup/restore).

To that end I would suggest to Ron to reverse the application. Try a backup from dev, even though you might not 'need' that. See whether that takes more close to the 4 or to those 20 hours. And keep an eye on the system as to what it is doing during that time. idle? waiting for disk or network? what kind of disk IO rates are generated and do you believe those to be sustainable in the configuration.

And uh... any good reason NOT to go forwards to 11.31 on the other partition? In the plans? Accelerate those plans?

Hein.

[Ron Irving]

Ok...I was looking around, (very carefully...trying to do a restore now,) and here's what I found:


GE-DRV B.11.23.0512 HP PCI Gigabit Ethernet Driver
IGELAN-DRV B.11.23.0712 HP PCI Gigabit Ethernet Driver

Could these 2 drivers be conflicting? What would be the result of me removing the GE-DRV?

It's a start, right?

[Duncan Edmonstone]

This isn't Windows - all the drivers are from HP and all work fine together - as it happens the drivers are for different cards anyway... if you're interested you can see which driver for which cards in the matrix here:

http://docs.hp.com/en/supportmatrixEthernetver2.pdf/supportmatrixEthernetver2.pdf

The important thing in diagnosing these sorts of issues is to approach it in a systematic fashion. You think you have a network problem - you need to prove that by removing the other items from the equation and just testing the performance of the network. Try that with netperf or as Hein suggests by sneding data to /dev/null.

HTH

Duncan

[Ron Irving]

Hey Duncan, et al. Thanks for all of your responses. We did that yesterday. We dd'd a /dev/zero file, and created a 500MB 'file' to transfer. Like I said, we rarely got above 10MB/sec. That was outside of netbackup, which seems to move even slower.

I, again, was looking around, and noticed that the hpigelanconf file has none of the variables set.

Perplexed yet? I am

[Ron Irving]

nope...that's not it

[Duncan Edmonstone]

>> created a 500MB 'file' to transfer

so you created a file to transfer

so it wasn't just based on network performance - the file had to be read off disk.

Honestly spend the time to get netperf installed - it doesn't use files at all so you get "true" network performance only from tests with it...

HTH

Duncan

[Bill Hassell]

Are you doing any testing with direct cables? Network performance depends on a lot of components in the middle. I would start by testing with a crossover cable between the two partitions -- that gets rid of switch config errors and well as routers/gateways, etc. You should see 50-75 Mbytes/sec between the two Gbit cards. As mentioned, simple disks may be too slow to meet the top speed of the two cards, but a high performance fibre array should keep the data flowing at full speed. Use ftp for the test to minimize the overhead associated with scp.

Then connect the cables to the switch and perform the same test. It should be virtually identical to the crossover cable test. If not, check the switch's port settings. Then work down the chain to the other system. In a network, the maximum performance of the Gbit card will be the slowest link along the way. Also note that Jumbo frames (mtu=9000) must be supported through every hop to be effective.

[Viktor Balogh]

instead of creating a file, just pipe the output of dd trough the network like this:

dd if=/dev/zero | ssh frances "dd of=/dev/null"

This will eliminate the disk speeds in the process. But as it was suggested before, best to use netperf. ;)

[Viktor Balogh]

I should have mentioned, that testing the LAN with ssh/scp is not the best practice, ssh does a lot of encryption, it is a known problem of HP-UX machines. So again, just go ahead with netperf! ;)

[BUPA IS]

Hello Ron ,

Just to be sure the link is error free, can you post the output of
lanadmin -g mibstats_ext 0 (or which ever ppa is in use ) also can you check that the port at switch end is also set to 1000 auto negotiate.

Mike

[Ron Irving]

Ok...I'm moving forward on this. I downloaded netperf, and am trying to figure out how to use it. Here's the lanadmin output. If y'all see anything out of sorts, etc, etc...

thanks!!

[rick jones · Accepted solution]

There should be some installation instructions in http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html

On both systems:
*) download tar file
*) unpack tar file
*) cd to the directory created by tar
*) ./configure
*) make
*) on the system to receive data say src/netserver
*) on the system to send data say src/netperf -H -f M

the -f M is to get netperf reporting in MegaBytes (mega == 1024 * 1024) per second rather than megabits (1000 * 1000) per second.

we can go from there.

[Ron Irving]

We're going to install APA, and drill down from there