C7000, Flex-10, Centos 5.4 and network performance

[Schwager]

Hi all, (Hi Dave, hi Rick),

we are not able to setup a ethernet connection with more that about 120MByte/second transfer rate between 2 linux boxes using a 2500Mbps (or 10000 Mpbs) VC-Flex-10 link:

[root@kvm21 ~]$ time dd if=/dev/zero bs=1M count=1000 | nc kvm22-storage 6151
copied, 8.24583 seconds, 127 MB/s

[root@kvm22 ~]$ nc -l 6151 |dd of=/dev/null

Our setup:
- C7000 enclosure
- 2 x VC-Enet 1/10Gb Module in Bay1+2
- 1 x VC-Flex10 10Gb Module in Bay3
- 2 x VC-FC on Bay5+6
- 2 x BL460c G5/6 (with an additional HP NC532m in M1 for VC-Flex10 in bay3) running centos5.4

The centos (2.6.18-164.11.1.el5) using the current Broadcom NetXtreme II bnx2x (v1.52.12) loaded with the option disable_tpa=1.

Ethtool (on both machines) tells me a correct setup:
[root@kvm21 ~]$ ethtool eth4
Settings for eth4:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
2500baseX/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
2500baseX/Full
10000baseT/Full
Advertised auto-negotiation: Yes
Speed: 2500Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 16
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)
Link detected: yes

If i try a performance test on the loopback, i got more than 120MByte/sec - so, the kernel is not the limiting factor (:
[root@kvm21 ~]$ time dd if=/dev/zero bs=1M count=1000 | nc localhost 6151
1048576000 bytes (1.0 GB) copied, 3.58561 seconds, 292 MB/s

Could somebody tell me how to track down this problem ? I hipe, we will get more than 250MByte/sec over the 10000 Mpbs link ...

best regards
Danny

[Schwager]

Ups, using iperf shows other values:

- kvm21 -> kvm22, 9000 Mbs eth4
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 7.86 GBytes 6.75 Gbits/sec

ok, netcat should not be used for performance testings ...

I think, the problem is fixed.

Thx
Danny

[rick jones]

More generally, if you want to measure network performance, don't include filesystem :) Measuring networking performance is why we have netperf :)

[Schwager]

refer to the last posting..