Virtual Connect Domain failure

[Emarthi]

Hi Jeroen!

Your solution to the problem worked for me, very nice! Thanks!

After I set new ip's on both the VC's all start to work again.
I have pasted your soulution below, maybe this will work for other systems with the same problem.

Erik

"I see that your are using corporate DHCP. FUnny enough I don't see any kind of DNS/gateway listed at your EBIPA settings.
FOr now I would suggest the following:

1. do a VC CLI: SHow interconnect or a show status (this will show maybe the no comm status on your VC bay 1 & 2).

Reserve in DHCP your IP's used for Bay 1 & 2.
Then configure static EBIPA for interconnect on bay 1 & 2.
like this example in CLI:
#NOTE: SET EBIPA commands are only valid for OA v3.00 and later SET EBIPA INTERCONNECT VCM.IP.bay.1 255.255.248.0 1 SET EBIPA INTERCONNECT GATEWAY x.x.x.x 1 SET EBIPA INTERCONNECT DOMAIN "test.com" 1 ADD EBIPA INTERCONNECT DNS 0.0.0.0 ADD EBIPA INTERCONNECT DNS 0.0.0.0 SET EBIPA INTERCONNECT NTP PRIMARY x.x.x.x 1 SET EBIPA INTERCONNECT NTP SECONDARY x.x.x.x 1 ENABLE EBIPA INTERCONNECT 1 SET EBIPA INTERCONNECT VCM.IP.BAY.1 255.255.248.0 2 SET EBIPA INTERCONNECT GATEWAY x.x.x.x 2 SET EBIPA INTERCONNECT DOMAIN "test.com" 2 ADD EBIPA INTERCONNECT DNS 0.0.0.0 ADD EBIPA INTERCONNECT DNS 0.0.0.0 SET EBIPA INTERCONNECT NTP PRIMARY x.x.x.x SET EBIPA INTERCONNECT NTP SECONDARY 16 x.x.x.x ENABLE EBIPA INTERCONNECT 2

If you change your original VC bay 1 & 2 IP's to a different one then now in use then you need to {clear VCmode} on OA CLI.

After the new IP, subnetmask & GW are set on the interconnects (without DNS) then just poweroff and poweron the secondary VC module first:
OA CLI:
poweroff interconnect 2
poweron interconnect 2
then if needed do a VC CLI: reset vcm -failover (this might not work; then go rightaway to next step) poweroff interconnect 1 poweron interconnect 1
"

[Dennis Hinson]

I would like to thank everyone who contributed to the knowledge on this forum regarding this issue specifically.

6 hours of my life and 15 minutes after I read this blog we have 2 chassis problems fixed!

[Bob Firek]

Hello Everybody,

It my enclosure settings I have the following settings: Bay, Enabled, EBIPA Address, Subnet Mask, Gateway, Domain, DNS Servers, Autofill, and Current Address. All field are filled in. When they say remove the DNS information do they just the IP addresses in the DNS servers field? We also have our domain in the Domain field, should that be removed as well? We are currently on OA 3.11 and VC 3.10. Will be looking at upgrading soon. Need to ask a stupid question. Is it wise to upgrade when the Domain is in a failed state even if everything seems to be working?

Any and all tips will be appreciated.

Thanks,

Bob

[Mark Wibaux]

You only need to remove the IP addresses in the DNS server fields to work around the issue.

I would implement the work around and get your VC modules back online fully before performing the firmware update.

Below is some detail of what actually went wrong in the firmware (collected at the time from some ITRC posts)  and a link to the customer advisory about it.

HP Virtual Connect - Virtual Connect Manager May Be Unable to Communicate (NO_COMM) if DNS Is Enabled for Virtual Connect Ethernet Modules

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02720395

This is a bug, which translated 10. from ASCII to binary (ASCII 1 is 49, ASCII 0 is 48, ASCII . (period) is 46) giving a class C address in the 49.48.46.0 range. The bug was ignored, because that class C was not valid.

This Class C was allocated mid February. DNS queries to this (previously bogus) class C began to resolve, and returned errors. For instance, nslookup (for the 4 ASCII characters of 10.1) of 49.48.46.49 shows

mx-ll-49.48.46-49.dynamic.3bb.co.th

This was discovered (and reported on the SAN security mailing list) when outbound SSH connections to this Class C were observed by sites following best practice and monitoring outbound connection attempts

This caused a huge problem around the world. You could be affected if you were running any c class chassis with any virtual connect. The temporary fix was to erase the DNS server settings in the OnBoard admin under Enclosure Bay IP Addressing (for each tab). Sometimes that resulted in a complete reset of the V/C interconnect devices.

[Bob Firek]

Mark,

Thank you for the clarification. I wondering if you can help clarify something else for me. In the advisory to select the "Interconnect Bays" and remove DNS server IP addresses but in the comments included in your posting it states to remove the DNS server IP addresses from each tab. In your experience should I remove the DNS Server entries just from the Interconnect Bays tab or from each tab?

Thanks again for your help,

Bob

[Bob Firek]

It seems to have worked by just removing the DNS Servers entries from the Interconnect Bays tab. I removed the DNS Server entries just for the Enet modules but left the DNS entries for the fibre channel modules. Since the cards were out of sync they did resync which killed any VMs or servers running on the blades. Definitely do this during off hours or a maintenance window where you can bring down the servers running on the blades.