Flex-10 nics and ESXi DCC compatibility

[Antonio Milanese]

Hello,
recently one of our client bought a new c7000 equipped with flex-10 vc modules to consolidate/virtualize some workloads, and so i've started researching for the best "cookbook" to implement flex-10 and ESXi 4 with DCC/smartlinks but now i'm really lost in traslation summaryzing the mixed infos and feedbacks i've found on HP/VMWARE websites, forums, blogs..

i've already read all flex-10 guides/cookbooks i'm aware of but there are too many other conflicting and mixed experinces out there:

http://communities.vmware.com/message/1556215
http://www.virtualtroll.com/?p=368
http://virtualnut.wordpress.com/2009/12/16/issues-with-esx-4-x-hp-virtual-connect-flex-10-firmware-v...
http://frankdenneman.nl/2009/04/flex-10-lessons-learned/

and mostly neither VMWARE nor HP have published a complete and clear (no HCL are messed) compatibility matrix on their websites, i mean something like SPOCK streams.

Essentially I'd like to get some advice or a blueprint on ESX build / firmware / driver compatibility to configure flex-10 vc in the following (and maybe general) scenarios:

C7000 pls BL-495G6 pls NC532i (arrg those pesky broadcoms!)
2 FLEX-10 VC + ACTIVE/ACTIVE + VLAN MAPPED VCDOMAIN
2 SUS Smartlink enabled
ESXi U1 or U2
PROCURVE 2900al/5300 and CISCO 3500 in the mix

According to what i've read i'm thinking about two possible solutions to have DCC and smartlinks working (vNic beacons not feasible too many vlans broadcasts):

assuming VC FW >= 2.32 + NC532i code 5.017 + >= OA 3.0

1) the semi-official DCC supported ESXi U1 + bnx2x version 1.48.107.v40.2 + VMWARE KB1017458 patchset
2) the official (but PSOD reported) ESXi U2 + bnx2x version 1.52.12.v40.3

Which is best/working?

Thanks in advance and my apologies for bad english.

Best regards,

Antonio

[Dan Robinson_4]

Dont know what to tell you about the PSOD, but option 2 appears to be the official HP Answer.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=3794423&prodTypeId=3709945&objectID=c02003458

[Antonio Milanese]

Hello Dan,

>>Dont know what to tell you about the PSOD, but option 2 appears to be the official HP Answer.

thanks for your replay:

yes i've read that document but it's windows oriented and lacks a clear indication for ESX:

it states that vmware should provide the driver..well not so much useful at least for me =) since on the vmware HCL things are even more confusing:

if you try this query:

http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=server&productId=1&advancedORbasic=advanced&maxDisplayRows=1000000&key=&release[]=74&datePosted=-1&partnerId[]=41&formFactorId[]=1&cpuSeriesId[]=46&filterByEVC=0&filterByFT=0&min_sockets=&min_cores=&min_memory=&rorre=0

it'll return two results and the latter stating that you have to use the v1.48.107-1.0.4 of bnx2x driver but if you query BCM57711

http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=io&productId=1&advancedORbasic=basic&keyBasic=BCM57711&rorre=0

they tell you to use v1.52!

I think i'll follow the second path and use the 1.52.12.v40.3 driver hoping that when ESX 4.1 will be released there'll be a more clear compatibility documents!

Have a good day,

Antonio

[Dominic Schoehl]

Hello,

updated an BL460c G6 yesterday to ESXi 4.0 U2 and BCM driver 1.52.12.v40.3

No issues so far. SmartLink works perfect now. Virtual Connect Firmware is 3.01

If no problems occur within one week I'll update our other blades running ESXi (BL460 and BL490).

Regards
Dominic

[DerekS_1]

I'm in the same boat as the previous poster. 4.0 U2, active/active dual SUS's, Flex-10 modules, BL490c G6, and v1.52 driver. Yesterday I perform tests to make sure smart link was working in all scenarios. All the tests passed. No PSOD, but this is just a small pilot environment with little to no network traffic.

VC 3.01, OA 3.10.

[Antonio Milanese]

Hello,

first of all let's me thanks you for yours feedbacks!

@Derek , Dominic

I ended up installing ESXi U2 plus bnx2x v.1.52 with this firmware levels:

OA 3.10
VC FLEX 2.33
NICs 5.0.17

all seams to work as expected ( smartlinks,DCC,ecc) and no PSOD so far either under moderate load.

I hope to go out of testing lab and start to migrate some production VMs next week.

But now another question:

Can anybody please tell how ESXi 4.1 is not working out of the box with VC FLEX-10?!?
I've just tried to install on the same c7000 enclosure a fresh copy of ESXi 4.1 and......
surprise! DCC working no more..FLEX-10 nics stay up even when both SUS go offline!
Naturally esxpdate tell me that v.1.52 driver is not suitable!

I'm missing something? (probably!!)

/me rant/
On VMWare HCL or release notes issues no word at all about flex-10 platform..MIA
On HP no doc/advise..MIA
Is FLEX-10, or may be i must say broadcom FLEX-NICS (yup now replaced LOM with Emulex silicon!) really an exotic, marginal, old(!!) platform?
HP and VMWARE are you listening!?
/rant off/


Thanks in advance for your patient,

Antonio

[Matt Kra]

We just upgraded to 4.1 and lost DCC/SmartLink. What gives! I've opened a support ticket with HP VMware support. Still waiting on an official response. The 4.1 driver is completly a new rev...

/var/log # esxupdate query --vib-view | grep -i bnx2x
deb_vmware-esx-drivers-net-bnx2x_400.1.54.1.v41.1-1vmw.0.0.260247 installed 2010-05-19T00:12:49+00:00
cross_vmware-esx-drivers-net-bnx2x_400.1.52.12.v40.3-1.0.4.00000 retired 2010-07-22T11:49:12.679307+00:00

[dchmax]

Matt, check the firmware on the network cards.

Antonio, I've been dealing with a similar issue but with ESX 3.5 U5. Basically to get it to work I downgraded the NIC boot code firmware using the firmare maint cd 8.60 on the BL490 G6 to 4.8.0. This firmware does not know what DCC is, yes at this point SmartLink does not work. Each of my vSwitchs has 2 network ports assigned, active/passive setup, but then had to enable beacon probing. I tested module failover and switch uplink failure and it worked like a charm. I believe this setup will work with 4.0 since it does not have a driver that support DDC.

VC firmware, NIC firmware, and drivers are competely dependent of each other more then ever now and my issue is the cryptic documentation from HP.

[Jorge Fabregas]

Hello guys,

I'm having the same problem here. Our setup:

Blades: Bl460c G6 (running with latest Firmware DVD)
Flex-10 Module: 2.32 (I know it's not the last one)
ESXi: 4.1

I've been doing some tests with SmartLink (I will disconnect uplink) and see if ESXi notices it... So far, ESXi doesn't notice the disconnection. I know it must be something with the bnx2x driver within ESXi because I peformed the exact same test on another blade running Windows 2003 and Windows indeed detects the disconnection (NICS show "disconnected" right away).

I plan to update the Flex-10 firmware anyway but as I mentioned, Windows 2003 worked fine so I guess the issue is on the ESXi side.

If anyone gets any update on this please share.

Best regards,
Jorge

[Matt Kra]

I have confirmation that HP is aware of the issue and is working on a fix. Having the latest firmware on the Flex-10 side along with the inbox bnx2x driver in 4.1 does not fix the issue.

I imagine we will see a new bnx2x driver released shortly. Don't roll back to the 1.52 driver as it causes the PSOD.