There have been a lot of questions surrounding the use of the latest Broadcom drivers with the Dynamic Control Channel(DCC) feature with VMware ESX4 operating system. Brian wanted to set things straight:
*************************************************************************************************************
Hi All.
I’m writing this email as I think there is a lot of confusion about this issue.
So I really hope that this email will help out internally. so we can help our customers fix/workaround this issue.
Description of issue:
When using virtual Connect Flex 10 with virtual connect firmware 2.3x together with newer Broadcom firmware on a FlexNic you can experience issues.
The common Issues can be “but not limited to”
1. When installing/or running ESX4 all nics are disconnected:
2. When installing ESX4U1 nics are not recognized and installation will not continue.
3. If Server is unplugged or reset server X is used via the OA CLI on a system running ESX4U1, the server is started in recovery shell and is displaying the following:
4. If A Virtual connect profile is change on a ESX4/ESX4U1 it might ask you to power down the Server !
A bit of information about DCC & a Q&A section
Why did we introduce DCC as it can cause all these issues !?
The idea behind DCC is actually good :)
The issues we had before DCC was for example that when changing a Virtual connect profile you would have to shutdown the host.
Another issue was that Smartlink did not work with a flexnic.
What will DCC give us of functions/why would I upgrade to enable DCC ?
Some current features are for ex that you can change all relevant settings in the VC profile without having to shutdown the Server (even bandwidth on the nics)
Also Smart Link is now working on a flexnic.
Which firmware versions support DCC ?
Virtual Connect firmware need to be 2.3x. currently 2.30,2.31,2.32
Broadcom nic firmware with boot code 5.0.11 or newer “5.2.7” (linux fw component 2.2.3,2.2.4,2.2.2.6) (windows fw component 2.1.5.7,2.1.5.5,2.1.5.1)
So what is the real problem with current DCC implementation and why are people seeing all these issues ?
The problem with DCC is that some of these functions are moved out into the Driver Layer (link state for ex).
So if the correct driver isn’t installed you can encounter several issues.
I have create a Q&A sections which will cover most common issues.
Q: Does the bnx2x driver version 1.45 or 1.48 support link state / DCC ?
A: NO. you will need to use 1.52 for supporting link state/DCC
http://downloads.vmware.com/d/details/esx_40_broadcom_bnx2xu1_dt/ZHcqYmRqcGhiZGVqdA
Q: If the links are down on my ESX4U1 host when using VC fw 2.3x and nic FW 2.2.4 Will updating the nic firmware to 2.2.6 fix the issue ?
A: No it will not fix the issue.
Q: If the links are down on my ESX4U1 host and I’m not able to upgrade the driver because it hangs or boot in recovery mode, what to do ?
A:
- Boot the FDT 1.70 and you will regain link on the nics / ESX4U1 will see the nics correctly again.
- Download the 1.52 Driver and update the host with that specific driver.
http://downloads.vmware.com/d/details/esx_40_broadcom_bnx2xu1_dt/ZHcqYmRqcGhiZGVqdA
Ps: if you don’t want to wait until the FDT load all the packages and reboot automatically you can just reset the server after “Activating available network interfaces”
Q: If I use vmware update manager and patch the host with all current patches will the driver be updated to 1.52.
A: At the time of writing vmware patch manager will not update the bnx2x driver to 1.52 !
Q: I’m using non DCC VC fw 2.1x and nic boot code firmware 5.0.11 but my nic are not enumerated correctly in ESX ?
A: Update boot code to 5.2.7 via the latest Broadcom firmware package as this firmware will resolve an issue where some NC532 devices may not enumerate correctly.
Q: Can FDT 1.70 cause my nics to be unlinked ?
A: yes I have seen this at multiple sites and in our lab. When the nic firmware is updated the server can enter a state where it needs to communicate with VC via a DCC
Capable driver to regain link. If you encounter such a situation just boot the FDT again and you will regain link.
Remember to update the driver afterwards.
Q: What if I have an unsupported “OS” which does not have a DCC capable driver ?
A: Well that’s really a big issue because if the server is unplugged from the enclosure or if the customer encounters a enclosure power loss
You will not get link on those nics again until a DCC capable driver is loaded
OS examples could be ESX 3.5 or XenServer 5.5.
Q: I have an DCC driver unsupported OS and my links are down. How do I regain link ?
A: I know of two ways.
- Just boot FDT 1.70 (as it will load a DCC capable driver). The server will remain linked until it is unplugged or a reset server x command is used.
- Downgrade the nic boot code to version 4.8.0 and you will not have the issue again. (please be aware that if you customer use the FW cd without removing
the Broadcom online nic fw you will have the same issue again
Q: Is there an easy way to reproduce this issue I my environment, so we can see what the customer is experiencing ?
A: The easiest way is just to install ESX4U1 and don’t upgrade to bnx2x driver to 1.52 (native is 1.48). After installation do a reset server X from the OA cli and you will
See the issue when the server is booting.
If you have any questions or comments please let us know. Join the conversation.
******************************************************************************************************************
