v3000 enclosure with FLEX10 unable to ping some IP's on vSwitch1

[Bas Rijniersce]

Hello,

I have a very, very strange problem (and spent many hrs with vmWare support on the phone).

We have a c3000 enclosure, BL465 blades. Enclosure bay 1 and 2 have 1G switches, bay 3 & 4 have Flex-10 switches. All blades have Mez1 as 1G, Mez2 as 10G. Placement of Flex10 interconnects in bay 3 & 4 (instead of default 1 & 2) is forced by size of Mez card.

We have just added the 10G stuff and some 10G iSCSI storage.

I have vSwitch0 in the management network, a vSwitch1 in the iSCSI network. I have two HP 4500 iSCSI targets at 10.x.x.11 and .12, gateway at .1 and the new targets at .14 & .15

From the blades I can reach the 11 and 12 targets without issues, when I tried adding the 14 & 15 I was unable to reach them. During the digging I also realized I can't ping the .1 (gateway, unneeded as such, but alas).

From the switch I can ping the vmkernel interface (using the same VLAN as the source address)

After lots of testing I found that if I make the management vmkernel port in the iSCSI network, all is well, everything pings as it should etc.

I can create a second vSwitch in my normal management lan. Add another VMkernel port and enable management traffic on it and all is still well. I can uncheck management traffic from vSwitch0 vmkernel port and I can manage on the normal management IP.

But the console GUI still shows vSwitch0 as the management IP, I'm sure that somebody at some point will think "that's wrong" let's fix it, and I'm in trouble.

So in short, with a vmkernel port on vSwitch1 I CAN ping 10.x.x.11 and 10.x.x.12
I CANNOT ping 10.x.x.1 and 10.x.x.14+15
I CAN ping the vmkernel IP from the switch (the .1)
I CAN see arp entry for .1 on ESXi (and it's correct)

If use the exact same nics and IP's on vSwitch0 it all works without issues.

Unfortunately troubleshooting on a low level is not so easy because I can't put a sniffer on the Flex10 interconnect (no hardware to do that available). ESXTOP suggests that traffic does go out via the correct interface.

Any ideas where this might be comming from?

Bas

[Steven Clementi]

How are your network profiles configured in your VC Manager? and your server profiles?


Which NIC's are your vSwitches using?

How is the new 10G stuff connected? to what is it connected to?


Do you have a network connection diagram you can share?


Steven