topic Re: Virtual Connect Domain failure in BladeSystem - General

Virtual Connect Domain failure

Steve MacKenzie — Sun, 16 Jan 2011 10:30:30 GMT

Hi,
I have the following issue, whenever I try and edit or delete a particular server profile the entire VC domain resets with the following error:
Enclosure state FAILED : All Enet modules not OK or DEGRADED
Critical] Domain state FAILED : 1+ enclosures not OK or DEGRADED
VCM initialized

All blades lose network connectivity until the modules re-initalize.

We are running:
OA version 3.21
VC version 3.15
All Blades on BIOS l24 09/02/2010

Any help appreciated.

Re: Virtual Connect Domain failure

dchmax — Mon, 17 Jan 2011 14:16:31 GMT

I've not seen this before. You could try failing the active vc domain module to the standby and make the same changes. Hopefully it can be narrowed down to a specific module issue. Support will more then likely walk you through reseating all the modules and the backplanes. What type of modules do you have in interconnect bay 1 and 2?

Re: Virtual Connect Domain failure

Steve MacKenzie — Mon, 17 Jan 2011 20:49:25 GMT

We have - HP VC Flex-10 Enet Modules.
We have logged a support call and they are still going through the logs.

Re: Virtual Connect Domain failure

dchmax — Mon, 17 Jan 2011 20:50:36 GMT

Post what you find out. I'm about to upgrade the firmware in our test chassis.

Re: Virtual Connect Domain failure

anthony.bucci — Thu, 20 Jan 2011 01:33:12 GMT

Have you figured out what is wrong here? I have the same issue right now.

OA version 3.21
VC version 3.15

VC-ENet gives a no communication error.
VC-FC seems to be working just fine.

Thanks!

Re: Virtual Connect Domain failure

Steve MacKenzie — Thu, 20 Jan 2011 01:56:15 GMT

This is the response for the support team:

"recommended that the only way to resolve this is to deleting and recreating the Domain.
This issue could be caused due to corruption in the Virtual Connect Domain Configuration. Hence to resolve the VCD crashing issue every time a profile is edited or deleted, is to delete the existing VCD configuration and create a new Domain."

Which isn't good when I'm running crtical production servers in the enclosure.

I have escalated the issue, but have yet to receive another response.

Re: Virtual Connect Domain failure

anthony.bucci — Thu, 20 Jan 2011 01:58:25 GMT

Thanks for the update...
I've done that and it does not seem to help either...

very bad answer on the part of HP.

Can't wait to hear what the next level has to say.

Thanks again.

Re: Virtual Connect Domain failure

ChrisTan_1 — Thu, 20 Jan 2011 11:38:51 GMT

Hi Steve,

are you using any of the new features in ver 3.15? I noticed you are still using Flex-10 and not FlexFabric modules so VC firmware ver 3.10 should be sufficient.

Force a VC firmware downgrade to ver 3.10?

Re: Virtual Connect Domain failure

Steve MacKenzie — Thu, 20 Jan 2011 23:18:18 GMT

The issue started while we were running 3.10
We were advised we needed to upgrade our firmware to the revisions mentioned in my 1st post.
After upgrading the issue remains.

Regards,
Steve

Re: Virtual Connect Domain failure

DerekS_1 — Fri, 21 Jan 2011 04:20:50 GMT

I had a similar problem in December when we did VC/OA firmware upgrades. Two linked chassis, and VC died on us such that all blades in both chassis lost network connectivity. Critical production servers were taken offline and I was not at all happy.

After several hours of being on the phone with HP, I gave up. I'm not sure what finally cured the problem, but I ended up de-assigning and assigning a profile to a blade then things got back in sync and the servers came back online.

I'm not at all a fan of VC, and will avoid future purchases of this technology as my faith in its reliability is nearly gone.

Re: Virtual Connect Domain failure

Cindy Mayer — Thu, 24 Feb 2011 19:42:45 GMT

This might help some people

The first thing I'd like to do is rule out an issue that can cause VCM to display no communication with Virtual Connect modules that occurs when DNS is enabled in the Onboard Administrtor under Enclosure Settings > Enclosure Bay IP Addressing > Interconnect Bays. If EBIPA is used to assign an IP address, make sure there is no DNS information entered. If there is, remove it and update by hitting the Apply button at the lower right. (This is if you use EBIPA to assign IP address. If you use an external DHCP server, then we would want to remove the DNS info there.)

The symptoms for the DNS issue is VCM reporting no communication for one or more VC modules and stacking links showing failed.

After i replied with this fixed it i got this in return.

Glad to hear that took care of it. There is a customer advisory on this issue, c02720395, but I don't believe it's availble on our website yet. There will be a future firmware fix, but for now the workaround is to remove the DNS information.

Re: Virtual Connect Domain failure

Jeroen_Kleen — Wed, 02 Mar 2011 08:26:11 GMT

Hello Folks,

This DNS issue as described in the document c02720395 is resolved in VC FW 3.17 that is our next targeted release of VC FW.

Cheers, Jeroen

Re: Virtual Connect Domain failure

RBC09 — Wed, 02 Mar 2011 16:11:22 GMT

Hey All,

We've what HP's advisory has asked and cleared the DNS records for our interconnect cards IPs. We run a DHCP not EBIPA. Also we temporarily set DNS not to update the IP scope. We are still experiencing the No communication error. Has anyone tried something else that resolved this issue? I am new to blade systems and am even having a hard time getting an HP engineer to resolve the issue.

Any advise or a "try this it worked for me" would be greatly appreciated.

Re: Virtual Connect Domain failure

HRT — Wed, 02 Mar 2011 23:10:34 GMT

I cleared the DNS settings, and once I reseated the secondary module, everything resynchronized correctly.

Re: Virtual Connect Domain failure

The Brit — Wed, 02 Mar 2011 23:18:26 GMT

For the older fw versions you need to remove the DNS info from the Device Bays page on the OA EBIPA.

Since you use DHCP, this might not help, but I would do it anyway.

Re: Virtual Connect Domain failure

JMG-IT — Thu, 03 Mar 2011 00:23:58 GMT

Hello - Jeroen or anyone else out there:

Is there a release date for VC FW 3.17?

Anyones input is greatly appreciated.

Thanks!

Re: Virtual Connect Domain failure

Chris House — Thu, 03 Mar 2011 11:57:49 GMT

I removed the DNS entries from my EBIPA settings for the Interconnect modules and within a few minutes it cleared up the loss of communication between my VC ethernet modules, without needing reseat/reboot anything. However, we were using a hostname in our LDAP server field so that had to be changed to the IP equivalent in order to use LDAP authentication again.

... looking forward to weird new bugs in 3.17..........

Re: Virtual Connect Domain failure

Emarthi — Fri, 04 Mar 2011 18:58:45 GMT

Hi,

I have the exact same problem, brand new C7000 with two VC Eth modules.
All latest FW.
Have also tried to remove DNS entry, without any luck!
We have not put the enclousure in production yet, so I also tried to delete the domain and recreate it, still not work.
Also tried to flush the FW on VC again.
I have open a case with HP, but the response arent wery quick, the only solution from HP so far are the workaround with DNS...

Erik

Re: Virtual Connect Domain failure

Jeroen_Kleen — Fri, 04 Mar 2011 19:05:02 GMT

Hello JMG-IT,

We don't have a formal release date yet although we are working on updated schedule to push it out within a month from now or sooner.

Erik, feel free to post your CaseID so I can lookup the status for your case.

A updated Customer Advisory with the same URL: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02720395 should be available very soon.

Cheers, Jeroen

Re: Virtual Connect Domain failure

Emarthi — Fri, 04 Mar 2011 19:12:45 GMT

Hi Jeroen,

Thanck you for quick response!
My case number with HP are: 4626776895.

Thancks