VIrtual Connect unavailable ethernet and FC ports

[Dan González]

We've got a new BladeSystem c7000 G2 with 16 Proliant bl460c G6 servers, two VC Flex-10 Enet modules and two VC 8Gb 24-Port FC modules.

The first problem that we've reached was that several servers doesn't have network connectivity on some ports. On Virtual Connect Manager, the domain status seems OK but on "Server connections" tab both on Ethernet Networks and San Fabric several ports from those servers change intermitently from OK to unavailable status. All other status (Ethernet networks, Shared Uplinks Sets, SAN Fabrics, External Connections) are always OK.

On the same server, with same ethernet networks attached to ports on both NIC ports, we get connectivity from one port and couldn't get it from the other one, so it seems that the cause isn't related with drivers, NIC firmware or so. The operating system neither since the OK/unavailable blinking occurs on uninstalled server that only tries to boot from PXE when powered on.

Other test was to move an ESXi 4 server with the problem from Bay 7 to Bay 15. When it gets reconfigured for Virtual Connect all networks attached worked. Bringing it back to Bay 7 spoiled it again.

Similarly moving a RHEL 5.4 server with no connectivity at all from Bay 6 to Bay 14 results on flawlessly working network. When moving it back to Bay 6 the networks continues to work. But, resetting Virtual Connect modules, causes the network to fail again.

Any ideas?

[Dan González]

I've just installed another Vmware ESXi 4 on blade server at Bay 16, just the way I did with the other (network failing) on Bay 7, and it works fine, with Shared Uplink Set, NIC teaming load balancing, and SAN Fabrics on an OK state. Weird.

[The Brit]

Hi Dan,
A couple of additional questions.

1. I assume that your Flex10's are in bays 1 & 2.
2. Which bays are your FC modules in?
3. Which Mezz slot are your HBA's installed in.
4. You DID use the same slot on all of the blades??

5. On the network switch, are your uplink ports configured for LCAP?

Dave

[Dan González]

1,2) Yes, Flex-10 modules are on bays 1 annd 2 while FC modules are on 3 and 4.

3,4) The HBAs are located on Mezzanine Slot 1. Not on all servers but nine of them. The rest doesn't have HBAs at all.

5) We've configured several uplinks:
- Three ethernet networks are defined at per VLAN basis, with an external connection from each VC module in an Active/Standby configuration. (This networks are defined as in our other c7000, and works OK. We've reviewed the switch configuration and it's right). Four servers (Bays 1,2,9 and 10 are using this networks and working fine, with two of them being Windows and two of them Linux machines.
- Two Shared Uplink Sets are defined for VMware machines. These are carrying three VLANs. "SUS1" connects from VC module at bay 1 and "SUS2" connects from VC module at bay 2. There is no port trunking on the switch side.
This is an step-by-step configuration from "HP Virtual Connect. Ethernet Networking Scenario Cookbook: Single Domain Scenarios. Fourth Edition April 2009", Scenario #18. As I mentioned on the second post, this configuration is working on an ESXi 4 server while not doing so on another. On the first one the ESXi virtual switch is configured with LOM1a and LOM2a (Using Mapped VLANs on Virtual Connect) on NIC Teaming with Load Balancing and it's working OK until now.

[Dan González]

Some more data. This could be a dumb thing, but maybe it's interesting, perhaps ...

Finally I've installed two more ESXi 4 servers for a total number of four, and the new ones have no problem with network or fibre channel. Thinking about what I've done differently about installation of the first no-networking server and the remaining I've concluded that I the only difference it's the value on BIOS for "System Options->Embedded NICs->NIC 1 Boot Options". On the first server installed I've disabled de Network Boot for NIC 1, while I've leave it enabled as default on the other three servers.

I really don't see the relationship but I have a try and enabled again Network Boot on NIC 1 on the first server, and IT WORKS!! In spite of this, it doesn' seem to be a two-way cause, because I've disabled de Network Boot on one of the working servers, and nothing changes; the network continues to working fine.


However, this is not true for the RHEL 5.4 server I've mentioned. Enabling Network Boot doesn't make a difference. I will check driver compatibility and so, but I've see it working fine when shuffled bays with it. Perhaps some negotiation problems, I don't know. I'm lost and I'm testing all the stupid things that doesn't make technical sense as far as my knowledge goes. Some of them seems working, but I'm not really confident about them as definitive solutions.