topic Re: VC Ethernet in BladeSystem - General

VC Ethernet

Rejeanc — Wed, 02 Mar 2011 19:08:06 GMT

Hi, lots of my VC in some of my c7000 are Failed (VC Status: Failed), plus they are trying to communicate with 49.48.46.49 on port 22.

I tried to reset them, but still failed, but stop to tried to connect to 49.48.46.49.

Re: VC Ethernet

Rejeanc — Thu, 03 Mar 2011 11:52:20 GMT

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02720395〈=en&cc=us&taskId=101&prodSeriesId=3794423&prodTypeId=3709945

Re: VC Ethernet

Viktor Balogh — Thu, 03 Mar 2011 14:02:58 GMT

Hi,

just curious if the advice in that document (disabling DNS) helped...

Re: VC Ethernet

dchmax — Thu, 03 Mar 2011 14:06:45 GMT

Regarding that advisory posted. We learned that if your OA firmware is 2.x then DNS needs to be removed from the device bays and interconnect modules. 3.x you should only need to remove DNS from interconnect modules. No OA or VC reset/reseating required. Just give it 5-10 minutes and it should clear up.

Re: VC Ethernet

Rejeanc — Thu, 03 Mar 2011 14:09:42 GMT

it resolved the error messages, but still tries to access 49.48.46.49 on port ssh.

Re: VC Ethernet

cjb_1 — Thu, 03 Mar 2011 16:24:29 GMT

Is 49.48.46.49 your vc domain ip address?

Re: VC Ethernet

Rejeanc — Thu, 03 Mar 2011 16:28:37 GMT

No it tries to communicate outside to 49.48.46.49 on port 22 once every minute.
dns gives mx-ll-49.48.46-49.dynamic.3bb.co.th

Re: VC Ethernet

DavidADav — Mon, 07 Mar 2011 13:51:43 GMT

We're seeing the same thing and ended up blocking our entire VC/OA network's Internet access ability until this is narrowed down.

While I'm glad to see that someone else has observed the 49.48.46.49 ssh connection, I'm still quite concerned.

I also had to run the same fix, disabling DNS via the internconnect IP config interface, however I had a nasty surprise after committing the change. Rather than having the interfaces refresh their config automagically after 2-5 minutes (as they had done in the past), my Flex-10 modules all spontaneously rebooted.

Very nasty surprise.

I've got a case open with HP for both the spontaneous reboot and the 49.48.x.x SSH connection - please post if you find anything out.

Re: VC Ethernet

DavidADav — Mon, 07 Mar 2011 14:40:23 GMT

Just a quick update.

The issue I noted, where the domain essentially reboots, is mentioned in the latest version of the DNS issue advisory, which was updated today.

I spoke with a VC engineer about the 49.48.x.x ssh issue, and here's what he told me:

- The "DNS bug" has existed since v1.8* of the VC firmware, but has not manifested itself until now.
- The "DNS bug" is not a virus.
- Some old section of code relied on the 49.48.x.x IP block not existing, and this code is somehow related to the VC's internal domain communication.
- The 49.48.x.x IP block was allocated in February (the WHOIS record and admin info was updated 14 February 2011), and reverse resolution began sometime thereafter.
- When it started to reverse resolve, the SSH connections also started.
- This reverse resolution actually _working_ is at the heart of the DNS issue.
- It will be fixed in firmware 3.17.

That's a summary of what I heard and I hope it helps. From my end it is technically feasible, as the SSH connections started at the exact moment my VC domain started to report an inability to connect to or manage the interconnects, and both issues cleared after removing the DNS server entries.

The BIG issue, in my mind, is that removing the DNS servers in the list can cause a full VC domain reboot, which is what I experienced. The engineer noted that there is no rhyme or reason to why this happens, but in a NO_COMM error status it is "very likely".

So watch out and don't update that DNS until you can secure an outage window, as we were completely down for around 5-8 minutes.

Dave

Re: VC Ethernet

BladeGuy — Tue, 08 Mar 2011 14:43:47 GMT

HP has identified a DNS related issue with HP Virtual Connect that does not impact data traffic but does impair the manageability of Virtual Connect devices. HP is acting promptly to help customers remove this issue in the short term with an interim resolution. A permanent firmware fix will be available in the next three weeks.

Customer Advisory Document ID: c02720395, March 7, 2011 is available now at the url below to resolve this issue. If you have questions or need assistance, contact HP Support in your home country. HP is committed to minimizing any impact on customer environments and to resolving the issue as quickly as possible.

Download Customer Advisory Document ID: c02720395, March 7, 2011:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02720395〈=en&cc=us&taskId=101&prodSeriesId=3540808&prodTypeId=329290

Visit the HP Support worldwide contact list:
http://welcome.hp.com/country/us/en/wwcontact_us.html

Re: VC Ethernet

mhenry_1 — Fri, 11 Mar 2011 07:48:22 GMT

Hello, I am encountering this issue as well and am about to follow the instructions to rectify it. However I am concerned that when the issue resolved that the virtual connects will attempt to synchronise configurations and changes that have been made to the active VC module while the issue was occurring will be overwritten by the old configuration. Therefore is it possible to take a configuration backup of both active and inactive VC modules before the change is made?
Whenever I attempt to access the configuration backup page a loading popup appears and will stay there indefinitely.

Re: VC Ethernet

BladeGuy — Fri, 11 Mar 2011 16:58:43 GMT

Please log a support case so you can get specific help with your issue. You can find contact info for support at
http://welcome.hp.com/country/us/en/wwcontact_us.html