Troubleshooting Virtual Connect time out on 2 Chassis

[LaSh00]

Hi All,

Scenario is as follows:

2 x C7000 Enclosures

2 x Virtual Connect Flex-10 modules on each chassis

2 x Virtual Connect FC 8Gb modules on each chassis

16 x BL460c G6 Servers on each chassis

Servers running either ESX or Xen Citrix

1 VC-Domain for both chassis, 1 uplink on each chassis with multiple VLANs (active + standby) + Stacking cables between the 2 chassis.

Issue we encountered last week was erratic ping time outs on all the servers on both chassis.

Per every 100 pings, we had 5-6 timeouts average resulting in slow / loss of connectivity on some applications.

Both chassis were in production, what was done was create a new active uplink with the Standby uplink, and switch over each server from the previous to new uplink.

After all servers were moved to the new uplink, we were still getting erratic time outs, less, but still getting.  We had to disable the 1st uplink completely for the servers to start pinging continuously without any time outs.

We need to know 

1. Why this happened. Suspecting faulty Fibre Cable on first uplink, but need confirmation.

2. Is there any HP tool to trouble shoot these kind of errors? As per VC manager, everything was fine on the first uplink.

3. Any document explaining how data switching occurs between chassis when more than 1 for part of a VC domain and are stacked?

Thanks a lot for any answers :) 

[Hongjun Ma]

what's the OA and VC version?

Is this a new installation or it has been working for some time but started to have issues recently.

Also, are you saying that if you don't have active uplink, then server can ping each other without problem? The moment you have active uplink, the server will lose pings even when pinging each other in the same VC domain? Are you pinging from server to server  or server to something at the uplink side.

to isolate if it's something physical layer related, here are something you can do

1) At your uplink switch, do something like show interface to monitor counters

2) SSH into VC ip address and do "show statistics enc0:1:X1" <-enc0 means base enclosure, 1 means VC module 1 and X1 means uplinkport 1. Use "reset statistics enc0:1:X1 to reset the counter". you can also find the same counter info in GUI under H/W overview and click VC module and port.

3) There is a VC stacking guide.

VC Multi-Enclosure Stacking – http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02102153/c02102153.pdf

Basically servers at local enclosure with active uplink will use uplink to go out directly, servers at 2nd enclosure will use stacking link to reach the enclosure with the active uplink and then go out. For details, it depends on your topology and which specific server.

[neil mccullagh]

Did you get an answer for your original question as to why there were different response rates on your ping tests

We are also getting high discard rates but show no errors on the cisco side.

Interested if you found anything