HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
BladeSystem Virtual Connect
cancel
Showing results for 
Search instead for 
Did you mean: 

Virtual connect and SLES bonding - huge packet loss

 
FilipB
Occasional Visitor

Virtual connect and SLES bonding - huge packet loss

Hi,

 

HUUUGEEE packet loses on short path:

blade server     <- 10GB chassis link ->      VC FLex (L2)     <- X5 uplink 1GB ->     CISCO router (L3 gateway)

Here pinging results from SLES on blade serer to the CISCO router IP Gateway

rdscho201:~ # ping -c 100000 -i 0.001 -l 100000 -q 10.115.200.129
PING 10.115.200.129 (10.115.200.129) 56(84) bytes of data.

--- 10.115.200.129 ping statistics ---
100000 packets transmitted, 47625 received, 52% packet loss, time 36810ms
rtt min/avg/max/mdev = 0.214/463.081/3082.246/733.394 ms, pipe 3042
rdscho201:~ #

rdscho201:~ # ping -c 100000 -i 0.001 -l 100000 -q 10.115.200.129
PING 10.115.200.129 (10.115.200.129) 56(84) bytes of data.

--- 10.115.200.129 ping statistics ---
100000 packets transmitted, 67388 received, 32% packet loss, time 35539ms
rtt min/avg/max/mdev = 0.197/1302.706/4560.360/1533.084 ms, pipe 32763
rdscho201:~ #

 

What is the reason of such a huge packet loses, and also bond0 (eth0, eth1) interface is fluctuating a bit

Below you can find host, VCFlex and CISCO router speciffic configuration:

 

 


========================================== HOST configuration - BladeSystem c7000 Enclosure G3 - ProLiant BL460c Gen8 - SLES 11 SP3 x64 =====================================


eth0 and eth1 bonded into bond0 (bonding configuration: arp polling , active-standby with 1s polling interval)

 

rdscho201:~ # ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.29/25 brd 10.115.200.127 scope global bond0
5: vlan2032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.135/26 brd 10.115.200.191 scope global vlan2032
6: vlan2033@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:17:a4:77:40:20 brd ff:ff:ff:ff:ff:ff
inet 10.115.200.197/26 brd 10.115.200.255 scope global vlan2033


rdscho201:~ # cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 10.115.200.1

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 5
Permanent HW addr: 00:17:a4:77:40:20
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: 00:17:a4:77:40:22
Slave queue ID: 0
rdscho201:~ #

 

rdscho201:~ # cat /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='none'
STARTMODE='off'
_nm_name='eth0'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-eth1
BOOTPROTO='none'
STARTMODE='off'
_nm_name='eth1'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-bond0
BONDING_MASTER='yes'
#BONDING_MODULE_OPTS='mode=1 miimon=100'
BONDING_SLAVE0='eth0'
BONDING_SLAVE1='eth1'
IPADDR='10.115.200.29/25'
PREFIXLEN='25'
#STARTMODE='auto'
STARTMODE='onboot'
BOOTPROTO='static'
BONDING_MODULE_OPTS='arp_interval=1000 arp_ip_target=10.115.200.1 mode=1 arp_validate=all'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-vlan2032
ETHERDEVICE=bond0
IPADDR=10.115.200.135
NETMASK=255.255.255.192
STARTMODE='onboot'
VLAN_ID='2032'


rdscho201:~ # cat /etc/sysconfig/network/ifcfg-vlan2033
ETHERDEVICE=bond0
IPADDR=10.115.200.197
NETMASK=255.255.255.192
STARTMODE='onboot'
VLAN_ID='2033'
rdscho201:~ #

 

 


========================================== VCFlex configuration - BladeSystem c7000 Enclosure G3 - HP VC Flex-10/10D Module =========================================


- 2x VCXlexes with one uplink per VC switch over X5 towards CISCO router
- two ethernet networks defined CHO-enc0-NetA and CHO-enc0-NetB with VLan Tunneling (no Vlan definitions inside the VC)
- server profiles assigned to ethernet Networks

 

->show config

Generating configuration. Please wait...

#### IMPORT ENCLOSURE(S) FIRST

### Modify 'ExitOnFailure' property value to FALSE, when executing in script
### mode to avoid exiting (vcmcli returning FAILURE) on a command failure
### set cli ExitOnFailure=TRUE

set domain Name=CHO-oneNDS_domain


set stackinglink -quiet DomainStackingMode=Full


set snmp enet ReadCommunity=public EnableV1V2=true EnableV3=false
set snmp fc ReadCommunity=public EnableV1V2=true EnableV3=false


add snmp-trap snmpCHO Address=10.116.90.57 Port=162 Community=public Format=SNMPv2 Severity="Unknown, Normal, Info, Warning, Minor, Major, Critical" DomainCategories="Legacy, DomainStatus, NetworkStatus, FabricStatus, ProfileStatus, ServerStatus, EnetStatus, FcStatus" EnetCategories="PortStatus, PortThreshold, Other" FcCategories="PortStatus, Other" Inform=false

### Setting the ssl-strength causes the web server to restart. Any active
### GUI session must be re-established. This action does not impact CLI
### sessions.
set ssl -quiet Strength=Strong
set user-security StrongPasswords=Disabled MinPasswordLength=8
set ldap Enabled=false

### Below command can only be executed by remote user (ldap/tacacs/radius) having a 'domain' privilege
### Locally authenticated users are not allowed to disable or enable local user authentication
### set local-users Enabled=true PrimaryRemoteAuthenticationMethod=none

set tacacs Enabled=false Port=49 Timeout=10 SecondaryPort=49 SecondaryTimeout=10 LoggingEnabled=false

set radius Enabled=false Port=1812 Timeout=10 SecondaryPort=1812 SecondaryTimeout=10


set role domain Order=local,ldap,radius,tacacs Operations=FirmwareUpdate,RestoreConfig,SaveConfig,SupportFiles
set role network Order=tacacs,radius,local Operations=PortMonitoring
set role server Order=ldap,local Operations=PortMonitoring
set role storage Order=radius,local Operations=

set serverid Type=Factory-Default

### Password for user : 'emergency' set to 'changeme'
add user emergency Password=changeme Enabled=true Roles="domain,network,storage,server"
### Password for user : 'HPadmin' set to 'changeme'
add user HPadmin Password=changeme Enabled=true Roles="domain,network,storage,server"
### Password for user : 'netact' set to 'changeme'
add user netact Password=changeme Enabled=true Roles="domain,network,storage,server"

set session Timeout=15

set auto-deployment TftpMode=Auto


set qos CustomNoFCoE

set qos-map DOT1P Class=Best_Effort Values="0-7"

set qos-map DSCP Class=Best_Effort Values="AF11-CS7"

set qos-class Medium Enabled=true RealTime=false Share=25 EgressDOT1P=2 MaxShare=100
set qos-class Real_Time Enabled=true RealTime=true Share=10 EgressDOT1P=5 MaxShare=10
set qos-class Class1 Enabled=false RealTime=false MaxShare=100
set qos-class Class2 Enabled=false RealTime=false MaxShare=100
set qos-class Class3 Enabled=false RealTime=false MaxShare=100
set qos-class Class4 Enabled=false RealTime=false MaxShare=100
set qos-class Class5 Enabled=false RealTime=false MaxShare=100
set qos-class Best_Effort MaxShare=100

set qos-classifier Downlinks Classifiers=DOT1P,DSCP
set qos-classifier Uplinks Classifiers=DOT1P

set qos-map DOT1P Class=Best_Effort Values="0,1"
set qos-map DOT1P Class=Medium Values="2,3,4"
set qos-map DOT1P Class=Real_Time Values="5,6,7"
set qos-map DSCP Class=Best_Effort Values="AF11,AF12,AF13,CS0,CS1"
set qos-map DSCP Class=Medium Values="AF21,AF22,AF23,AF31,AF32,AF33,AF41,AF42,AF43,CS2,CS3,CS4"
set qos-map DSCP Class=Real_Time Values="CS5,CS6,CS7,EF"

set qos CustomWithFCoE

set qos-map DOT1P Class=Best_Effort Values="0-7"

set qos-map DSCP Class=Best_Effort Values="AF11-CS7"

set qos-class Medium Enabled=true RealTime=false Share=25 EgressDOT1P=2 MaxShare=100
set qos-class Real_Time Enabled=true RealTime=true Share=10 EgressDOT1P=5 MaxShare=10
set qos-class Class1 Enabled=false RealTime=false MaxShare=100
set qos-class Class2 Enabled=false RealTime=false MaxShare=100
set qos-class Class3 Enabled=false RealTime=false MaxShare=100
set qos-class Class4 Enabled=false RealTime=false MaxShare=100
set qos-class Best_Effort MaxShare=100

set qos-classifier Downlinks Classifiers=DOT1P,DSCP
set qos-classifier Uplinks Classifiers=DOT1P

set qos-map DOT1P Class=Best_Effort Values="0,1"
set qos-map DOT1P Class=Medium Values="2,3,4"
set qos-map DOT1P Class=Real_Time Values="5,6,7"
set qos-map DSCP Class=Best_Effort Values="AF11,AF12,AF13,CS0,CS1"
set qos-map DSCP Class=Medium Values="AF21,AF22,AF23,AF31,AF32,AF33,AF41,AF42,AF43,CS2,CS3,CS4"
set qos-map DSCP Class=Real_Time Values="CS5,CS6,CS7,EF"

set qos Passthrough


set domain MacType=Factory-Default

set mac-cache Enabled=true Refresh=5
set igmp Enabled=false
set enet-vlan -quiet VlanCapacity=Expanded
set statistics-throughput -quiet Enabled=true SampleRate=5m
set port-protect networkLoop=Enabled
set port-protect pauseFlood=Enabled
set lldp enhancedTlv=false
set lldp taggedLldp=false
set lacp-timer Default=Short

add network CHO-enc0-NetA -quiet NAGs=Default VLanTunnel=Enabled LacpTimer=Domain-Default
add uplinkport enc0:1:X5 Network=CHO-enc0-NetA Speed=1Gb
add network CHO-enc0-NetB -quiet NAGs=Default VLanTunnel=Enabled LacpTimer=Domain-Default
add uplinkport enc0:2:X5 Network=CHO-enc0-NetB Speed=1Gb
### PacketBufferOverallocationRatio is applicable only for
### HP 1/10Gb VC-Enet and HP 1/10Gb-F VC-Enet
### modules and deprecated for all other module types.

set advanced-networking -quiet PacketBufferOverallocationRatio=1 FlowControl=auto

set domain WwnType=Factory-Default


set link-dist-interval Interval=30


add profile ischo201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection ischo201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection ischo201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile ischo201 enc0:1

add profile admcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection admcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection admcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile admcho201 enc0:2

add profile pgwcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwcho201 enc0:3

add profile pgwcho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwcho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwcho202 enc0:4

add profile pgwdcho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwdcho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection pgwdcho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile pgwdcho201 enc0:5

add profile rdscho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile rdscho201 enc0:6

add profile rdscho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection rdscho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile rdscho202 enc0:7

add profile bedscho201 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho201 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho201 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho201 enc0:8

add profile bedscho202 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho202 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho202 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho202 enc0:9

add profile bedscho203 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho203 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho203 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho203 enc0:10

add profile bedscho204 -NoDefaultEnetConn -NoDefaultFcConn -NoDefaultFcoeConn -quiet NAG=Default HideUnusedFlexNICs=true BootMode=Auto SriovMode=Advanced

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho204 Network=CHO-enc0-NetA PXE=Enabled SpeedType=Custom Speed=10000 SriovType=Disabled

# The MAC/WWN/Logical SN may be different than the original profile configuration depending on pool settings.
add enet-connection bedscho204 Network=CHO-enc0-NetB PXE=UseBIOS SpeedType=Custom Speed=10000 SriovType=Disabled

assign profile bedscho204 enc0:11

 

SUCCESS: Configuration script generated

->

 

========================================== CISCO router - one of the 2x routers ==================================================================================================0

vlan 2031
name NDS_OAM
!
vlan 2032
name NDS_Application
!
vlan 2033
name One_NDS
!


interface GigabitEthernet7/15
description NDS_Flexi_VC_A
switchport
switchport trunk native vlan 2031
switchport trunk allowed vlan 2031-2033
switchport mode trunk
switchport nonegotiate
logging event link-status
load-interval 30
storm-control broadcast level 0.10
storm-control multicast level 0.10
no cdp enable
spanning-tree portfast edge trunk
!


interface Vlan2031
description NDS_OAM
ip address 10.115.200.2 255.255.255.128
no ip redirects
no ip unreachables
standby 75 ip 10.115.200.1
standby 75 timers msec 300 1
standby 75 preempt
!
interface Vlan2032
description NDS_Application
ip address 10.115.200.130 255.255.255.192
no ip redirects
no ip unreachables
standby 76 ip 10.115.200.129
standby 76 timers msec 300 1
standby 76 preempt
!
interface Vlan2033
description One_NDS
ip address 10.115.200.194 255.255.255.192
no ip redirects
no ip unreachables
standby 77 ip 10.115.200.193
standby 77 timers msec 300 1
standby 77 preempt
!

 

==================================================================================

 

BR,

Filip