BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

Flex-10 nics and ESXi DCC compatibility

Antonio Milanese
Trusted Contributor

Flex-10 nics and ESXi DCC compatibility

Hello,
recently one of our client bought a new c7000 equipped with flex-10 vc modules to consolidate/virtualize some workloads, and so i've started researching for the best "cookbook" to implement flex-10 and ESXi 4 with DCC/smartlinks but now i'm really lost in traslation summaryzing the mixed infos and feedbacks i've found on HP/VMWARE websites, forums, blogs..

i've already read all flex-10 guides/cookbooks i'm aware of but there are too many other conflicting and mixed experinces out there:

http://communities.vmware.com/message/1556215
http://www.virtualtroll.com/?p=368
http://virtualnut.wordpress.com/2009/12/16/issues-with-esx-4-x-hp-virtual-connect-flex-10-firmware-v...
http://frankdenneman.nl/2009/04/flex-10-lessons-learned/

and mostly neither VMWARE nor HP have published a complete and clear (no HCL are messed) compatibility matrix on their websites, i mean something like SPOCK streams.

Essentially I'd like to get some advice or a blueprint on ESX build / firmware / driver compatibility to configure flex-10 vc in the following (and maybe general) scenarios:

C7000 pls BL-495G6 pls NC532i (arrg those pesky broadcoms!)
2 FLEX-10 VC + ACTIVE/ACTIVE + VLAN MAPPED VCDOMAIN
2 SUS Smartlink enabled
ESXi U1 or U2
PROCURVE 2900al/5300 and CISCO 3500 in the mix

According to what i've read i'm thinking about two possible solutions to have DCC and smartlinks working (vNic beacons not feasible too many vlans broadcasts):

assuming VC FW >= 2.32 + NC532i code 5.017 + >= OA 3.0

1) the semi-official DCC supported ESXi U1 + bnx2x version 1.48.107.v40.2 + VMWARE KB1017458 patchset
2) the official (but PSOD reported) ESXi U2 + bnx2x version 1.52.12.v40.3

Which is best/working?

Thanks in advance and my apologies for bad english.

Best regards,

Antonio
49 REPLIES
Dan Robinson_4
Regular Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Antonio Milanese
Trusted Contributor

Re: Flex-10 nics and ESXi DCC compatibility

Hello Dan,

>>Dont know what to tell you about the PSOD, but option 2 appears to be the official HP Answer.

thanks for your replay:

yes i've read that document but it's windows oriented and lacks a clear indication for ESX:

it states that vmware should provide the driver..well not so much useful at least for me =) since on the vmware HCL things are even more confusing:

if you try this query:

http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=server&productId=1&advancedORbasic=advanced&maxDisplayRows=1000000&key=&release[]=74&datePosted=-1&partnerId[]=41&formFactorId[]=1&cpuSeriesId[]=46&filterByEVC=0&filterByFT=0&min_sockets=&min_cores=&min_memory=&rorre=0

it'll return two results and the latter stating that you have to use the v1.48.107-1.0.4 of bnx2x driver but if you query BCM57711

http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=io&productId=1&advancedORbasic=basic&keyBasic=BCM57711&rorre=0

they tell you to use v1.52!

I think i'll follow the second path and use the 1.52.12.v40.3 driver hoping that when ESX 4.1 will be released there'll be a more clear compatibility documents!

Have a good day,

Antonio
Dominic Schoehl
Occasional Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Hello,

updated an BL460c G6 yesterday to ESXi 4.0 U2 and BCM driver 1.52.12.v40.3

No issues so far. SmartLink works perfect now. Virtual Connect Firmware is 3.01

If no problems occur within one week I'll update our other blades running ESXi (BL460 and BL490).

Regards
Dominic
DerekS_1
Frequent Advisor

Re: Flex-10 nics and ESXi DCC compatibility

I'm in the same boat as the previous poster. 4.0 U2, active/active dual SUS's, Flex-10 modules, BL490c G6, and v1.52 driver. Yesterday I perform tests to make sure smart link was working in all scenarios. All the tests passed. No PSOD, but this is just a small pilot environment with little to no network traffic.

VC 3.01, OA 3.10.
Antonio Milanese
Trusted Contributor

Re: Flex-10 nics and ESXi DCC compatibility

Hello,

first of all let's me thanks you for yours feedbacks!

@Derek , Dominic

I ended up installing ESXi U2 plus bnx2x v.1.52 with this firmware levels:

OA 3.10
VC FLEX 2.33
NICs 5.0.17

all seams to work as expected ( smartlinks,DCC,ecc) and no PSOD so far either under moderate load.

I hope to go out of testing lab and start to migrate some production VMs next week.

But now another question:

Can anybody please tell how ESXi 4.1 is not working out of the box with VC FLEX-10?!?
I've just tried to install on the same c7000 enclosure a fresh copy of ESXi 4.1 and......
surprise! DCC working no more..FLEX-10 nics stay up even when both SUS go offline!
Naturally esxpdate tell me that v.1.52 driver is not suitable!

I'm missing something? (probably!!)

/me rant/
On VMWare HCL or release notes issues no word at all about flex-10 platform..MIA
On HP no doc/advise..MIA
Is FLEX-10, or may be i must say broadcom FLEX-NICS (yup now replaced LOM with Emulex silicon!) really an exotic, marginal, old(!!) platform?
HP and VMWARE are you listening!?
/rant off/


Thanks in advance for your patient,

Antonio
Matt Kra
Occasional Visitor

Re: Flex-10 nics and ESXi DCC compatibility

We just upgraded to 4.1 and lost DCC/SmartLink. What gives! I've opened a support ticket with HP VMware support. Still waiting on an official response. The 4.1 driver is completly a new rev...

/var/log # esxupdate query --vib-view | grep -i bnx2x
deb_vmware-esx-drivers-net-bnx2x_400.1.54.1.v41.1-1vmw.0.0.260247 installed 2010-05-19T00:12:49+00:00
cross_vmware-esx-drivers-net-bnx2x_400.1.52.12.v40.3-1.0.4.00000 retired 2010-07-22T11:49:12.679307+00:00


dchmax
Frequent Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Matt, check the firmware on the network cards.

Antonio, I've been dealing with a similar issue but with ESX 3.5 U5. Basically to get it to work I downgraded the NIC boot code firmware using the firmare maint cd 8.60 on the BL490 G6 to 4.8.0. This firmware does not know what DCC is, yes at this point SmartLink does not work. Each of my vSwitchs has 2 network ports assigned, active/passive setup, but then had to enable beacon probing. I tested module failover and switch uplink failure and it worked like a charm. I believe this setup will work with 4.0 since it does not have a driver that support DDC.

VC firmware, NIC firmware, and drivers are competely dependent of each other more then ever now and my issue is the cryptic documentation from HP.
Jorge Fabregas
Regular Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Hello guys,

I'm having the same problem here. Our setup:

Blades: Bl460c G6 (running with latest Firmware DVD)
Flex-10 Module: 2.32 (I know it's not the last one)
ESXi: 4.1

I've been doing some tests with SmartLink (I will disconnect uplink) and see if ESXi notices it... So far, ESXi doesn't notice the disconnection. I know it must be something with the bnx2x driver within ESXi because I peformed the exact same test on another blade running Windows 2003 and Windows indeed detects the disconnection (NICS show "disconnected" right away).

I plan to update the Flex-10 firmware anyway but as I mentioned, Windows 2003 worked fine so I guess the issue is on the ESXi side.

If anyone gets any update on this please share.

Best regards,
Jorge
Matt Kra
Occasional Visitor

Re: Flex-10 nics and ESXi DCC compatibility

I have confirmation that HP is aware of the issue and is working on a fix. Having the latest firmware on the Flex-10 side along with the inbox bnx2x driver in 4.1 does not fix the issue.

I imagine we will see a new bnx2x driver released shortly. Don't roll back to the 1.52 driver as it causes the PSOD.
Jorge Fabregas
Regular Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Thanks Matt for confirming. It's a relief to know it is like that (I don't have to be troubleshooting anymore). I was worried I had something wrong somewhere...

Regards,
Jorge
Jorge Fabregas
Regular Advisor

Re: Flex-10 nics and ESXi DCC compatibility

BTW, does anyone knows can I query the ESXi (4.1) for the NIC driver version? That is, I want to find out the exact version of the stock bnx2x from ESXi 4.1. It's a disappoining that you can't find out this by using the vSphere Client.

I installed for the first time the vMA server and I tried some commands there against the ESXi but so far I'm unable to find the driver version number.

Regards,
Jorge
Matt Kra
Occasional Visitor

Re: Flex-10 nics and ESXi DCC compatibility

SSH into the ESXi host.

esxupdate query --vib-view | grep -i bnx2x

http://www.broadcom.com/support/ethernet_nic/faq_drivers.php

Jorge Fabregas
Regular Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Arrgh. Thanks again Matt! That worked. I forgot about enabling SSH in the ESXi! That was easier! Here's the output:

deb_vmware-esx-drivers-net-bnx2x_400.1.54.1.v41.1-1vmw.0.0.260247

So...I guess, as you mentioned, we'll have to wait for an updated driver.

All the best,
Jorge
Antonio Milanese
Trusted Contributor

Re: Flex-10 nics and ESXi DCC compatibility

Hello,

@Matt

Thanks for your report, i've recived the same response from HP..hope the new driver will be released soon =)

@dchmax

i know that with 3.5 and 4.0u1 downgrading the nic boot fw (and disabling DCC) works, but i want to keep failover times as low as possible and btw beaconing is an overkill if you have multiple VLANs :/

Anyhow with bnx2x v1.52.12.v40.3 DCC is working smoothly so i'll stick with this setup until the new 4.1 driver is out and tested =)

Thanks to all and regards,

Antonio
StefanW
Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Same problem over here; running a c7000 config with Flex-10 nics's and ESX 4.1. Anyone got the new driver we need ? I've also opened a case at VMware. They need to investigate the problem; waiting for an answer.

Stefan
bb4vm
Advisor

Re: Flex-10 nics and ESXi DCC compatibility

I have been having the same problem. Here is some information that HP Support provided me:


SUPPORT COMMUNICATION - CUSTOMER ADVISORY
Document ID: c02476622
Version: 1
Advisory: VMware ESX/ESXi 4.1 - Broadcom bnx2x VMware ESX Driver Version 1.54 Does Not Function With Virtual Connect Device Control Channel (DCC) and SmartLink Capability for 10 Gb Broadcom Adapters in VMware ESX/ESXi 4.1
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.
Release Date: 2010-08-13
Last Updated: 2010-08-13
________________________________________

DESCRIPTION
The Broadcom bnx2x VMware ESX Driver Version 1.54 does not function with HP Virtual Connect Device Control Channel (DCC) and SmartLink features on ProLiant and Integrity server blades configured with the NC532m or the NC532i adapter running firmware version 2.2.6. After installing or upgrading VMware ESX/ESXi 4.1 the following functionality is either not installed or is lost:
1. New installation - DCC and SmartLink functionality is unavailable in an HP Virtual Connect environment with the NC532m or NC532i Network Adapters after installing VMware ESX/ESXi 4.1.
2. Upgrade installation - If the bnx2x Asynchronous Driver Update CD version 1.52 was previously installed on a VMware ESX/ESXi 4.0 host, DCC/SmartLink capabilities will be lost after upgrading to VMware ESX/ESXi 4.1, which will overwrite the bnx2x driver version 1.52 with version 1.54 that is included with the base VMware ESX/ESXi 4.1operating system.
3. Network failover - ProLiant and Integrity server blades hosting VMware ESX/ESXi 4.1 may lose network failover capabilities that use the VMware ESX NIC teaming failover policy (vSwitch setting) "Link Status only."
SCOPE
Any ProLiant and Integrity server blade with Virtual Connect Version 2.30 (or later) and configured with the NC532m or NC532i adapter firmware version 2.2.6. after installing VMware ESX/ESXi 4.1 with the Broadcom bnx2x VMware ESX Driver Versions 1.54.

RESOLUTION
As a workaround, to allow network failover capabilities, use VMware Beacon Probing to determine proper VM NIC link status as follows:
Reconfigure the VMware ESX/ESXi 4.1 NIC teaming failover policies to "Beacon Probing." This modification will remove the dependency on SmartLink to toggle the VM NICs failing link status mapped to a FlexNIC.
There is no workaround that supports the Virtual Connect DCC and SmartLink capabilities with VMware ESX/ESXi 4.1 and the Broadcom bnx2x VMware ESX Driver Version 1.54.
This advisory will be updated when an updated driver is released to support DCC and SmartLink capabilities in Virtual Connect on ProLiant server blades.
Aaronb
Occasional Advisor

Re: Flex-10 nics and ESXi DCC compatibility

It is now Oct and there is still no driver!


This bug is precenting me from deploying vSphere 4.1 and it making me question our recent LARGE investment in new Blades.

Has anyone found a viable workaround other than switching to an Active/Standby uplink mode?

Thanks
Aaron B
Tate Eskew
Occasional Visitor

Re: Flex-10 nics and ESXi DCC compatibility

I agree Aaron. We spent a huge amount of money on this infrastructure and still no driver. It's quite ridiculous.
AaronK
Frequent Advisor

Re: Flex-10 nics and ESXi DCC compatibility

We have a bunch of bl460c g6s also and are having nothing but problems with the 1.52.12.v40.3 driver: PSODs and total loss of network connectivity under heavy network load.
We are running ESX 4.0 U2.
HP support is recommending we downgrade to version 1.48.107. Is that what other people have been doing also?
Aaronb
Occasional Advisor

Re: Flex-10 nics and ESXi DCC compatibility

I have contacted my reseller to see if it can be escalated to HP.

bb4vm
Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Now VMware has a KB, released 10/13.....

http://kb.vmware.com/kb/1029368
Aaronb
Occasional Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Hi,
Thanks for the heads-up!

But I have confirmed with HP that this is not a workaround for the DCC issue we have.

I wish it was....

Aaron
DerekS_1
Frequent Advisor

Re: Flex-10 nics and ESXi DCC compatibility

On a blog I saw a post from mid-September that v1.60 of the driver is supposed to be out "soon". Hopefully this fixes the DDC and PSOD problems. HP's in getting working drivers is really unacceptable.
AaronK
Frequent Advisor

Re: Flex-10 nics and ESXi DCC compatibility

Yep, We have a ticket open with HP on this issue (mainly the PSODs in the 1.52). We backdated to 1.48 and are waiting on the 1.60. They are giving me a 'late October/early November' date for release. I'm pressing them on letting us get our hands on it to validate it early...