Status of Flex10 Technology.

[The Brit]

Is anyone out there (other than us) concerned about the status of the Flex10 technology.

We were relatively early adopters of HP's BladeSystem, and Virtual Connect technology because it seem to meet many of our corporate needs,

1. It could accommodate all of our OS's (Windows (VMWare), Linux (Oracle Clusters), and OpenVMS).
2. Offered improvements in Power and Cooling costs.
3. Allowed Server consolidation to a greatly reduced footprint.
4. Simplification of Server management,
5. etc., etc.

The initial setup worked well and as advertised (with a couple of relatively trivial problems).

Based on our experiences at that point, we were quite open to the idea of upgrading to the VC Flex10 technology. After all, HP assured us that our proposed configuration would work, and the upgrade procedure was documented (in "HP Virtual Connect for c-Class BladeSystem Setup and Installation Guide, p47-48), and was so straight forward it only took up ~10 lines.

So, given these assurances, we "bought in". We purchased 8 Flex10 modules with the intention of upgrading Bays 1 & 2 in each of our 4 enclosures, retaining the 1/10 modules in bays 3 & 4 (as mentioned earlier, an acceptable and supported configuration).

Luckily, we were able to test the upgrade procedure in a "spare" chassis, unfortunately we only had a limited number of spare blades which we could use in this test chassis.

We performed the upgrade, exactly as documented, and our initial thought was that it had worked. We discovered on further testing that it had not. With the exception of 2 blades, we could not get any network connectivity. And the problem seemed to be between the Device Bays and the Flex10 modules.

We repeated this upgrade attempt at least 4 times over the following 3-4 weeks, with endless suggestions from HP. We opened 3 support tickets related to the upgrade, and had at least 3 teleconferences with HP Engineering. All to no avail. The upgrade just would not work. What we did determine was that
a) All profiles should be deassigned from Server Bays (not in documentation)
b) All servers should be powered down (not in documentation)
c) Backup VC Configuration (before swapping VC modules - no problem)
d) Physically swap the modules (no problem)
e) Log in to VC Manager and import the new Domain (no problem)
f) Restore the saved configuration (BIG PROBLEM...)

The act of restoring the saved configuration file completely screws up the mapping between the Device Bays and the VC Modules. They, in fact, get reversed, so instead of mapping d1-d16 to Bay1-Bay16 you end up with d16-d1 mapped to Bay1-Bay16. You remember my comment earlier that 2 of the servers worked?? Well it turns out that they were in Bays 4 and 13 (mapping d4-Bay4 and d13-Bay13). After the upgrade, the mapping became d4-Bay13 and d13-Bay4, which, because they were identical servers with identical configurations, and mutually acceptable profiles, deceived us into thinking that they were OK. In actual fact, they were masquerading as each other.

Finally, during a conversation with a particular HP Support Rep, we were told that to his best knowledge, this upgrade had never worked (... the technologies are too different for this to be compatable! (his words)). At our last teleconference (last week) which included VC Product Manager, VC Engineering, and sundry HP and VC personnel, the question "...has anyone on the call, ever seen this upgrade work???" just product an embarrassingly long silence.

Having said all that, we also discovered that if the Flex10 environment is built from scratch, i.e. not from an restored configuration file, then everything seems to be (eventually) fine (but see later). I say "eventually" because every single server/OS/Application combination required specific NIC drivers, Firmware version, or OS configuration options to make it work. None of these were easily identified, obtained, or documented.

The reason that this upgrade procedure is so important to us is that we have fully populated,very heterogeneous enclosures containing both Proliants and Itaniums, and multiple OS's. We run standalone system, oracle clusters, OpenVMS clusters and VMWare, all in the same enclosures. The prospect of having to rebuild all of our VC domains, with the associated problem of trying to match up VC-assigned WWIDs in new profiles, is not very appealing.

It just strikes me that HP did a piss-poor job of testing (at least) the documented upgrade procedure. We would have been accepting of a statement saying that the upgrade, as documented, does not work, if that statement has been forthcoming in the beginning, preferably before we invested a bundle of money in purchasing these modules. Even in the case where we built the VC domain from scratch, it took ~2 weeks of constant research (and additional support tickets with VMWare and HP) to get the necessary information about F/W and Drivers.

I have to say that although we were early converts to VC technology, and big fans of HP generally, (we are almost a completely HP shop), our experience with the Flex10 technology, and with HP support generally, has greatly damaged their credibility, in our eyes. 8 weeks have passed, and we are still not on Flex10's yet.

Finally, (but no longer surprising) we see the Flex10 alert issued today regarding compatability of Flex10 with other VC modules, and blades using Flex10 Mezz cards.

I know this might sound mean spirited, but did HP test ANYTHING on Flex10 before they started selling them??

I will be interested in any responses this post receives.

Dave

[WFHC-WI]

We found the exact same thing occurring in an enclosure we retrofitted with Flex-10 modules in place of 1/10Gb VC-Enets. (I just added a note about this in your other post).

I'm glad HP has finally seen enough of these go wrong to issue an advisory, do you have a link handy?

Other than that issue our experience with Flex-10 modules has been benign... of course in every other case (and eventually in this one) we rebuilt the entire domain.

HP needs to recognize the implications of releasing instable hardware and asking us to consolidate into it! Perhaps it's a ploy to get more customers to purchase support agreements?

[The Brit]

Sorry, no link. In fact no advisory. The admission came during a teleconference.

I would hope that an advisory will follow before any more people make the mistake of attempting this upgrade.

Dave.

[WFHC-WI]

Page 46, Virtual Connect Setup guide:
http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01732252/c01732252.pdf

"if you are removing a module that is in interconnect bay 1 or bay 2 of the primary enclosure, you must first delete the domain, and then recreate it after the module has been removed."

Page 49, same guide... the instructions for migrating from VC-Enet to Flex10s has no mention of domain re-creation. Seems like a consequential oversight unless maybe I just have misinterpreted it.

Suffice to say all should now know that you CAN NOT reliably replace your VCE with F10 without recreating your domain.

[The Brit]

The import of the domain is unavoidable, but this is not the problem.

When you first log in to VCM after installing the new Flex10 module, it takes you straight to the Domain Wizard. On the first screen you have two basic options,

1. Import just the bare domain, or

2. Inport the domain and restore a saved configuration file.

If you perform step 1, you can ignore the remainder of the wizard and still have 2 additional options;

A. (Re)Build the new VC Domain from scratch, or

B. use the Backup/Restore screen to restore a saved configuration file

The only combination which works is Step 1 followed by Step A.

Step 2 on its own creates a broken VC domain.
Step 1 plus Step B also creates a broken VC domain.

the 1/10G VC configuration file is apparently incompatible with the Flex10 module (no matter how you do it.) It is the act of restoring the old config which breaks the domain.

Dave.

[dchmax · Accepted solution]

Solution was to reverse engineer how VC builds out profiles and assigned WWID's. Once I nailed that down I was able to create a script that I ran through the CLI which built the VC Domain, Imported Chassis, Created Profiles, and assigned them to the correct blades in under 30 minutes.

As The Brit stated, the documented upgrade path from HP does not function, and if someone did get it to work its not worth the risk.

[Mike Schreiner]

wow. This is a nightmare. We have a 9 enclosure HP Blade matrix that we are very delicate when we make any move on it.
We experienced the same sort of thing and had to have the firmware upgraded completely from the blade to the VCM modules.

[The Brit]

Thread was for information only.