HPE Community
BladeSystem - General
1823950 Members
3508 Online
109667 Solutions
New Discussion

Virtual connect and SLES bonding - huge packet loss

 
FilipB
New Member

‎05-14-2018 04:49 PM

‎05-14-2018 04:49 PM

Virtual connect and SLES bonding - huge packet loss

Hi,

 

HUUUGEEE packet loses on short path:

blade server     <- 10GB chassis link ->      VC FLex (L2)     <- X5 uplink 1GB ->     CISCO router (L3 gateway)

Here pinging results from SLES on blade serer to the CISCO router IP Gateway

rdscho201:~ # ping -c 100000 -i 0.001 -l 100000 -q 10.115.200.129
PING 10.115.200.129 (10.115.200.129) 56(84) bytes of data.

--- 10.115.200.129 ping statistics ---
100000 packets transmitted, 47625 received, 52% packet loss, time 36810ms
rtt min/avg/max/mdev = 0.214/463.081/3082.246/733.394 ms, pipe 3042
rdscho201:~ #

rdscho201:~ # ping -c 100000 -i 0.001 -l 100000 -q 10.115.200.129
PING 10.115.200.129 (10.115.200.129) 56(84) bytes of data.

--- 10.115.200.129 ping statistics ---
100000 packets transmitted, 67388 received, 32% packet loss, time 35539ms
rtt min/avg/max/mdev = 0.197/1302.706/4560.360/1533.084 ms, pipe 32763
rdscho201:~ #

 

What is the reason of such a huge packet loses, and also bond0 (eth0, eth1) interface is fluctuating a bit

Below you can find host, VCFlex and CISCO router speciffic configuration:

 

 


========================================== HOST configuration - BladeSystem c7000 Enclosure G3 - ProLiant BL460c Gen8 - SLES 11 SP3 x64 =====================================


eth0 and eth1 bonded into bond0 (bonding configuration: arp polling , active-standby with 1s polling interval)

 

rdscho201:~ # ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.29/25 brd 10.115.200.127 scope global bond0
5: vlan2032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.135/26 brd 10.115.200.191 scope global vlan2032
6: vlan2033@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.197/26 brd 10.115.200.255 scope global vlan2033


rdscho201:~ # cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 10.115.200.1

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 5
Permanent HW addr: 00:17:a4:77:40:20
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: 00:17:a4:77:40:22
Slave queue ID: 0
rdscho201:~ #

 

rdscho201:~ # cat /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='none'
STARTMODE='off'
_nm_name='eth0'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-eth1
BOOTPROTO='none'
STARTMODE='off'
_nm_name='eth1'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-bond0
BONDING_MASTER='yes'
#BONDING_MODULE_OPTS='mode=1 miimon=100'
BONDING_SLAVE0='eth0'
BONDING_SLAVE1='eth1'
IPADDR='10.115.200.29/25'
PREFIXLEN='25'
#STARTMODE='auto'
STARTMODE='onboot'
BOOTPROTO='static'
BONDING_MODULE_OPTS='arp_interval=1000 arp_ip_target=10.115.200.1 mode=1 arp_validate=all'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-vlan2032
ETHERDEVICE=bond0
IPADDR=10.115.200.135
NETMASK=255.255.255.192
STARTMODE='onboot'
VLAN_ID='2032'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-vlan2033
ETHERDEVICE=bond0
IPADDR=10.115.200.197
NETMASK=255.255.255.192
STARTMODE='onboot'
VLAN_ID='2033'
rdscho201:~ #

 

 


========================================== VCFlex configuration - BladeSystem c7000 Enclosure G3 - HP VC Flex-10/10D Module =========================================


- 2x VCXlexes with one uplink per VC switch over X5 towards CISCO router
- two ethernet networks defined CHO-enc0-NetA and CHO-enc0-NetB with VLan Tunneling (no Vlan definitions inside the VC)
- server profiles assigned to ethernet Networks

 

->show config

Generating configuration. Please wait...

#### IMPORT ENCLOSURE(S) FIRST

### Modify 'ExitOnFailure' property value to FALSE, when executing in script
### mode to avoid exiting (vcmcli returning FAILURE) on a command failure
### set cli ExitOnFailure=TRUE

set domain Name=CHO-oneNDS_domain


set stackinglink -quiet DomainStackingMode=Full


set snmp enet ReadCommunity=public EnableV1V2=true EnableV3=false
set snmp fc ReadCommunity=public EnableV1V2=true EnableV3=false


add snmp-trap snmpCHO Address=10.116.90.57 Port=162 Community=public Format=SNMPv2 Severity="Unknown, Normal, Info, Warning, Minor, Major, Critical" DomainCategories="Legacy, DomainStatus, NetworkStatus, FabricStatus, ProfileStatus, ServerStatus, EnetStatus, FcStatus" EnetCategories="PortStatus, PortThreshold, Other" FcCategories="PortStatus, Other" Inform=false

### Setting the ssl-strength causes the web server to restart. Any active
### GUI session must be re-established. This action does not impact CLI
### sessions.
set ssl -quiet Strength=Strong
set user-security StrongPasswords=Disabled MinPasswordLength=8
set ldap Enabled=false

### Below command can only be executed by remote user (ldap/tacacs/radius) having a 'domain' privilege
### Locally authenticated users are not allowed to disable or enable local user authentication
### set local-users Enabled=true PrimaryRemoteAuthenticationMethod=none

set tacacs Enabled=false Port=49 Timeout=10 SecondaryPort=49 SecondaryTimeout=10 LoggingEnabled=false

set radius Enabled=false Port=1812 Timeout=10 SecondaryPort=1812 SecondaryTimeout=10


set role domain Order=local,ldap,radius,tacacs Operations=FirmwareUpdate,RestoreConfig,SaveConfig,SupportFiles
set role network Order=tacacs,radius,local Operations=PortMonitoring
set role server Order=ldap,local Operations=PortMonitoring
set role storage Order=radius,local Operations=

set serverid Type=Factory-Default

### Password for user : 'emergency' set to 'changeme'
add user emergency Password=changeme Enabled=true Roles="domain,network,storage,server"
### Password for user : 'HPadmin' set to 'changeme'
add user HPadmin Password=changeme Enabled=true Roles="domain,network,storage,server"
### Password for user : 'netact' set to 'changeme'
add user netact Password=changeme Enabled=true Roles="domain,network,storage,server"

set session Timeout=15

set auto-deployment TftpMode=Auto


set qos CustomNoFCoE

set qos-map DOT1P Class=Best_Effort Values="0-7"

set qos-map DSCP Class=Best_Effort Values="AF11-CS7"

set qos-class Medium Enabled=true RealTime=false Share=25 EgressDOT1P=2 MaxShare=100
set qos-class Real_Time Enabled=true RealTime=true Share=10 EgressDOT1P=5 MaxShare=10
set qos-class Class1 Enabled=false RealTime=false MaxShare=100
set qos-class Class2 Enabled=false RealTime=false MaxShare=100
set qos-class Class3 Enabled=false RealTime=false MaxShare=100
set qos-class Class4 Enabled=false RealTime=false MaxShare=100
set qos-class Class5 Enabled=false RealTime=false MaxShare=100
set qos-class Best_Effort MaxShare=100

set qos-classifier Downlinks Classifiers=DOT1P,DSCP
set qos-classifier Uplinks Classifiers=DOT1P

set qos-map DOT1P Class=Best_Effort Values="0,1"
set qos-map DOT1P Class=Medium Values="2,3,4"
set qos-map DOT1P Class=Real_Time Values="5,6,7"
set qos-map DSCP Class=Best_Effort Values="AF11,AF12,AF13,CS0,CS1"
set qos-map DSCP Class=Medium Values="AF21,AF22,AF23,AF31,AF32,AF33,AF41,AF42,AF43,CS2,CS3,CS4"
set qos-map DSCP Class=Real_Time Values="CS5,CS6,CS7,EF"

set qos CustomWithFCoE

set qos-map DOT1P Class=Best_Effort Values="0-7"

set qos-map DSCP Class=Best_Effort Values="AF11-CS7"

set qos-class Medium Enabled=true RealTime=false Share=25 EgressDOT1P=2 MaxShare=100
set qos-class Real_Time Enabled=true RealTime=true Share=10 EgressDOT1P=5 MaxShare=10
set qos-class Class1 Enabled=false RealTime=false MaxShare=100
set qos-class Class2 Enabled=false RealTime=false MaxShare=100
set qos-class Class3 Enabled=false RealTime=false MaxShare=100
set qos-class Class4 Enabled=false RealTime=false MaxShare=100
set qos-class Best_Effort MaxShare=100

set qos-classifier Downlinks Classifiers=DOT1P,DSCP
set qos-classifier Uplinks Classifiers=DOT1P

set qos-map DOT1P Class=Best_Effort Values="0,1"
set qos-map DOT1P Class=Medium Values="2,3,4"
set qos-map DOT1P Class=Real_Time Values="5,6,7"
set qos-map DSCP Class=Best_Effort Values="AF11,AF12,AF13,CS0,CS1"
set qos-map DSCP Class=Medium Values="AF21,AF22,AF23,AF31,AF32,AF33,AF41,AF42,AF43,CS2,CS3,CS4"
set qos-map DSCP Class=Real_Time Values="CS5,CS6,CS7,EF"

set qos Passthrough


set domain MacType=Factory-Default

set mac-cache Enabled=true Refresh=5
set igmp Enabled=false
set enet-vlan -quiet VlanCapacity=Expanded
set statistics-throughput -quiet Enabled=true SampleRate=5m
set port-protect networkLoop=Enabled
set port-protect pauseFlood=Enabled
set lldp enhancedTlv=false
set lldp taggedLldp=false
set lacp-timer Default=Short

add network CHO-enc0-NetA -quiet NAGs=Default VLanTunnel=Enabled LacpTimer=Domain-Default
add uplinkport enc0:1:X5 Network=CHO-enc0-NetA Speed=1Gb
add network CHO-enc0-NetB -quiet NAGs=Default VLanTunnel=Enabled LacpTimer=Domain-Default
add uplinkport enc0:2:X5 Network=CHO-enc0-NetB Speed=1Gb
### PacketBufferOverallocationRatio is applicable only for
### HP 1/10Gb VC-Enet and HP 1/10Gb-F VC-Enet
### modules and deprecated for all other module types.

set advanced-networking -quiet PacketBufferOverallocationRatio=1 FlowControl=auto

set domain WwnType=Factory-Default


set link-dist-interval Interval=30


add profile ischo201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection ischo201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection ischo201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile ischo201 enc0:1

add profile admcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection admcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection admcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile admcho201 enc0:2

add profile pgwcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwcho201 enc0:3

add profile pgwcho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwcho202 enc0:4

add profile pgwdcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwdcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwdcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwdcho201 enc0:5

add profile rdscho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile rdscho201 enc0:6

add profile rdscho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile rdscho202 enc0:7

add profile bedscho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho201 enc0:8

add profile bedscho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho202 enc0:9

add profile bedscho203 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho203 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho203 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho203 enc0:10

add profile bedscho204 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho204 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho204 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho204 enc0:11

 

SUCCESS: Configuration script generated

->

 

========================================== CISCO router - one of the 2x routers ==================================================================================================0

vlan 2031
name NDS_OAM
!
vlan 2032
name NDS_Application
!
vlan 2033
name One_NDS
!


interface GigabitEthernet7/15
description NDS_Flexi_VC_A
switchport
switchport trunk native vlan 2031
switchport trunk allowed vlan 2031-2033
switchport mode trunk
switchport nonegotiate
logging event link-status
load-interval 30
storm-control broadcast level 0.10
storm-control multicast level 0.10
no cdp enable
spanning-tree portfast edge trunk
!


interface Vlan2031
description NDS_OAM
ip address 10.115.200.2 255.255.255.128
no ip redirects
no ip unreachables
standby 75 ip 10.115.200.1
standby 75 timers msec 300 1
standby 75 preempt
!
interface Vlan2032
description NDS_Application
ip address 10.115.200.130 255.255.255.192
no ip redirects
no ip unreachables
standby 76 ip 10.115.200.129
standby 76 timers msec 300 1
standby 76 preempt
!
interface Vlan2033
description One_NDS
ip address 10.115.200.194 255.255.255.192
no ip redirects
no ip unreachables
standby 77 ip 10.115.200.193
standby 77 timers msec 300 1
standby 77 preempt
!

 

==================================================================================

 

BR,

Filip

0 Kudos
Reply
The opinions expressed above are the personal opinions of the authors, not of Hewlett Packard Enterprise. By using this site, you accept the Terms of Use and Rules of Participation.