Oneview Profile annoyances

[csouthy]

Does anyone else have problems and frustrations with how HP Oneview handles profiles.

1) Applying a profile takes 10 minutes.  This is rediculous - is this normal behaviour im experiencing?  Virtual Connect was pretty much instantaneous

2) Profiles locked to specific hardware models - If i copy a server profile that is for a Gen8, i cant apply it to a bay with a Gen9.  What advantage does having a profile for a specific model actually give us?

3) Profiles defaulting to Virtual Addresses - this is more an annoyance.  I would prefer that profiles default to their Physical Addresses on creation (but this might be just me).

[ChrisLynch]

Hello, @csouthy.  I'm sorry you have had such a diffferent experience with HPE OneView (versus what I can presume is Vitual Connect Manager). Let me address each point, and hopefully answer your questions.

1. You have to understand, there are significant differences between VCM and HPE OneView.  What can seem 'instantaneous' to you, is actually not.  VCM isn't multithreaded, and takes roughly 2 min to apply a Server Profile to a Server Resource.  A VCM Server Profile is also a  'lowest common denominator' approach to how it applies.  VCM doesn't provide any way to control or map specific connections to a given hardware device, let alone the other features found in an HPE OneView Server Profile (BIOS, Local Storage, Firmware, etc.).  With HPE OneView Server Profiles, you have to specify the Server Hardware Type (SHT) and Enclosure Group (EG), if you didn't specify a specific Server Hardware resource.  The SHT and EG provide the ability to properly model a Server Profile with Expected Connections, Network Resources, BIOS Settings and external 3Par Storage.  When you create a Logical Interconnect Group and Enclosure Group, you are actually modeling how Servers can and will map to network and fabric resources.  If a resource isn't provisioned to a LIG, and subsequently the associated EG, the Server Hardware resource isn't expected to be able to access that resource.  So contrast that with VCM; you cannot specify which Connection is assigned to which Physical Interface on the target system.  The SHT provides this policy (amongst a few other policy capabilities, like BIOS and Firmware).  Plus, all of this happens Asyncronously, which isn't possible in VCM (which is a syncronous-based platform, where you cannot perform another Server Profile operation until the prior one is finished).  Which leads me to your second question.
2. Prior to HPE OneView 2.0, Server Profiles were locked to both the SHT and EG, as I explained the relationship above.  We relaxed the restriction to allow customers to move between EGs and SHTs, and would remove settings or network/fabric resources if it isn't possible to assign in the destination EG and/or SHT.  You have much more flexibility now with the release of 2.0.
3. VCM required the Administrator to specify how they want to manage Addresses for Ethernet, FC and Serial Numbers.  This was limited to the VCM Domain.  You could specify VCM Managed Addresses from either Pre-defined pools or Admin provided pools.  You could also default to Hardware Addresses.  If you chose VCM Managed, there is an override within the Server Profile.  Each VCM Domain would need to be configured seperately.  If you wanted to aggregate address management, VCEM was the preferred method, and had a number of management drawbacks.  VCM and VCEM are now combined within HPE OneView.  When the appliance first boots, it randomly generates 1 million MAC and WWN addresses each, which uses the Link Local Administrator bit.  There are a lot of customers that use 'Virtual' Addresses, and would rather not manage the physical.  FC storage connectivity is certainly a primary case to use it.  DHCP, PXE and other Ethernet bootstrap mechanisms are another.  There are administrative benefits as well, if say a system board or mezz is replaced.  The NICs and/or FC adapters no longer need to have their policies and configrations modified because the adapter address changed.  This also helps in your sitatuation where you'd like to migrate a Server Profile from a Gen8 to Gen9.  Imagine needing to reconfigure the NIC or FC ports in the OS because the addresses changed.  Or worse, you needed to go modify FC fabric zoning (or your SAN Admin needs to).  I wouldn't classify 'Virtual' Address Management as an annoyance.  It certainly is a benefit to quite a number of Virtual Connect customers.

I hope this helps.  I look forward to your comments to this discussion.

[csouthy]

HI Chris,

Thanks for the in depth reply.

In relation to #1...i guess the problem is general maintanance on servers have a considerable increase in time.  Say for example the profile in VCM took 2 minutes to apply (which in experience is much quicker), this is much quicker than Oneview.  Please look at this example

I had a Gen9 BL460 in which i had to shutdown from my ESXi environment and remove from the enclosure to perform some maintanance.   After this was completed, i inserted the server back into the enclosure, however i had to once again reapply the server profile.  This process is recorded in my logs as taking over 15 minutes.  So i am twiddling my thumbs for 15 minutes while the profile is assigned.  Say i need to do this to multiple servers, you can appreciate the length of extra time this adds.

Now say i need to evacuated an enclosure populated with hosts into another, we are talking about a 16bay x 15min profile apply which if done one at a time (which may be a case sometimes with VMHost resourcing), we are looking at an extra 4 hours of additional time compared to VCM.

In VCM i would have an identical profile set up on another enclosure, then switch the server location and it would be booting almost instantaneously

However this may be just the method that i go about things.  If you can suggest a way to perform profile work that reduces the need or time to apply profiles that would be welcomed.

On #2, can you point me to some documentation that may help me understand this a bit more?

On #3 point taken.  However this is more a change on what VC did by default, so i was perplexed as why Virtual was defaulted over Physical...but thats fine.

[ChrisLynch]

There are fundamental differences between VCM and OneView as I have pointed out.  If you are simply comparing the time it takes for a single VCM Server Profile to apply, and compare that with OneView Server Profile, that isn't enough, and yes you will have a completely different opinion.   The Average Server Profile operation in VCM is 2 minutes.  The Average Server Profile operation in OneView is going to be different depending on what you are setting within the Server Profile.  If all you are doing is configuring Connections and nothing else, that operation typically takes 7 minutes.  Boot times with various RAM configurations will skew that.  Let me put it into another way:

If I were to create 16 Server Profiles for a single Enclosure, it will take 2min x 16 (32 total Minutes, which is serial BTW), versus 7-14 minutes of total time for 16 servers as OneView is an Asyncronous Task based system (16 Server Profile tasks will run in parallel, versus 1 by 1 with VCM).  Again, when only configuring Server Profile Connections and nothing else.  When you start to add in the other options avialable to you in HPE OneView Server Profiles (which are not there in VCM), the process will take longer as the server(s) will boot into Intelligent Provisioning and run the policy configuration from the Server Profile.

When OneView provisions a Server Profile, it is making sure the configuration is deployed to the target system, instead of the 'set it and forget it' method VCM takes.  When you model and deploy a Server Profile, that is the configuration policy you are deploying.  Not just Connection management that VCM provides.

If you want to stop the Server Profile from re-applying (which is automatic BTW), you can change the Server Hardware Affinity from Device Bay to Server Hardware.  This locks the Server Profile to the target system (based on the servers Serial Number and UUID), and prevents the Server Profile from being assigned to another physical server.  So in your case, the Server Profile would not have been re-applied just because the Server Hardware was removed from the Device Bay it was populated in.

Also, I'm curious to understand what maintenance tasks you are referring to?  I'd love to better understand the frequency and the tasks you are performing, so I could bring that back to the development team as another data point on how our customers use our product(s).

Please review the HPE OneView 2.0 Release Notes (look on Page 5, Section 1.2, bullet point #3: Profile mobility – across generations, models, enclosure groups) for the changes updates to the Server Profile, which not only include the option I mentioned, but also improvements to Local Storage and Online Firmware/Driver remediation for Gen8 and newer systems.  I would highly suggest you bookmark the HPE OneView Enterprise Information LIbrary Vanity URL (http://hpe.com/info/oneview/docs) for future product documentation needs.

There are a lot of things different in VCM when transitioning to HPE OneView.  I would also highly suggest you review the Introduction to HP OneView concepts for HP Virtual Connect customers whitepaper.  We also have a training class that goes into a great amount of detail around the entire HPE OneView product.  I helped develop most of the course that is avialable.

[csouthy]

HI Chris,

I can can see now the case you are presenting.  However a few other points.

The example of the Asynchronus task, whilst it great, shutting down 16 ESXi hosts at a time would not happen in our environment.  We would be doing 1 or 2 at a time in order to keep enough resources in our environment.

For the boot time.  I only have connections added.  I have no other options selected.  I could live easier with the 7min rough apply time, so not sure thy mine is taking the extra 8 minutes if its a barebones profile

Thanks for the suggestion on the Affinity type, noted.

As for the maintanance tasks...... we have had a lot of trouble with our latest Gen9 hosts and specifically their 20Gb Flex LOM cards.  This comes down to both firmware and ESXi driver compatibility.  So we are performing higher than usual updates and restarts on these hosts.

We are also at the stage of upgrading our switch infrastructure and also VC modules.  We are migrating our hosts out of the enclosure to perform this work (reduce risk etc)

[ChrisLynch]

I totally understand that you aren't going to take 16 servers out at a time for maintenance.  I wasn't suggesting that.  However, with newer capabilities does come change in how things operate.  OneView is MUCH more than just Connection and Identity management, which is all that VCM provides.

Reboots of a server to apply a newer driver and the Emulex/Avago firmware doesn't result in any additional reboots than say VCM, as the Server Profile isn't being modified (unless you are assigning a new Firmware Baseline to the Server Profile).  Also, OneView doesn't re-apply the entire configuration if you were to modify the Server Profile to add another connection, or re-insert the original server into the device bay.  But, when the Server Profile applies, it is deploying a policy, and it must wait for devices to configure into the state they need to be, unlike VCM. 

We continue to look at ways to improve how devices are managed and configured.  As UEFI evolves over time, and HPE adopts the newer versions, it will open more paths to configure devices differently, and improve the overall experience.  I can't discuss more than that in an open forum.  But just know we (HPE) continuously work to improve system boot times.  It's not always HPE OneView causing increased boot times.

[csouthy]

Hi again,

Another perfect example.  2 servers with as simple a profile as you can get.  2 network connections, thats it.  Nothing else selected

I had to relocate 2 servers this afternoon, but one at a time as they are a redundant pair.  If this was VC, i would shut down the server, relocate it and it would start up within 30 seconds.  This whole task of 2 servers would take 10 mins tops (boot time).

Im now stuck here after hours as this process has blown out to over 50 minutes

5 mins for Oneview to "Discover" the blade in the enclosure 

15 mins for Oneview to "Apply the Profile"

5 mins boot time

x2

= 50 minutes.

Of all the good things OneView might do, it's definately not worth the extra waiting time.  15 minutes to apply a profile is absurd, i don't care what it's doing behind the scenes.  I will be moving the VC administration back to the modules starting tomorrow.  Oneview is good for monitoring, but administration is a nightmare if we have to put up with these overheads

Surely i'm not the only one that has this issue?

[ChrisLynch]

15 Minutes is not what any customer should experience with that type of Server Profile configuration.  I have been able to reproduce this behavior in a lab, and am looking into why this is happening.  Just to get some additional data points from you, can you please provide the following:

• iLO Firmware Version
• System ROM Version
• Hardware Platform Model

You can get all of this information by going to the Server resource object, and then selecting the Hardware subpage view.  I've attached a screenshot showing an example.

[csouthy]

Hi Chris,

It's great that you can reproduce and are looking into it.  Below are a few recent examples, but its really every hardware on any firmware version currently.  From Gen7s Gen8s and Gen9s i have had this issue.

BL460c G7, ilo 1.80,...ROM l27 01/29/2011

BL460c Gen9, ilo 2.30....ROM l37 09/24/2015

Let me know what you find

[ChrisLynch]

Thanks.  I have been able to see this with Gen8 and Gen9, which is all that I have in my lab.  Can you tell me what OA firmware you have deployed?  From some initial investigation, we added some enhancements for when the Server Profile is being provisioned (when the appliance talks to the OA and iLO), and it may be causing some unecessary delays in deploying the profile.

One more thing.  The forums here are not official means for product support.  What I would ask that you do for me is to open a support case, and then Private Message me the case number, so I can help track this to resolution.

[csouthy]

No problem, will do now.

As for the firmware.  We have two at 4.40 and one at 4.50

[csouthy]

Hi Chris,

Just an update for you.  I decided to increase the resources assigned to the VM.

HP reccomends the following

2 Virtual CPUs (i increased to 4)

10GB RAM (I increased to 26GB)

I did a test and my profile went from 15 minutes average, down to 42 seconds!  Clearly an absolute turnaround.  I am continuing to test and it would be good if you could see the same results if you increased the appliances resources in your test environment.

But from initial results, maybe the recommended specs on the Installation doco should be increased

Interested to hear how you go.

[ChrisLynch]

Please know that increasing the VM resources isn't a supported operation.  Yes, you can add more vCPU or RAM as you have.  However, the appliance will not scale its internal resources to utilize those additional resources it has been given.  When did add those resources, did you reboot the appliance?  If so, in some of our findings, rebooting the appliance helps, but overtime the issue will return.

[csouthy]

Hi Chris,

The increased resources was from a recommendation from HP support, so it seems the left hand is not talkng to the right.

A reboot was required and this in turn could have been the contributing factor as well.

From what you're mentioning, seems to be some sort of stability issue in which the applicaiton has issues after being up for a period of time?

[ChrisLynch]

The increased resources was from a recommendation from HP support, so it seems the left hand is not talkng to the right.

They should not have recommended that at all.  Inreasing the Appliance resource allocation is not supported.

From what you're mentioning, seems to be some sort of stability issue in which the applicaiton has issues after being up for a period of time?

Yes.  Because after a reboot of my lab appliance Server Profile operations were dramatically improved like your experience was, without increasing the default vCPU and RAM allocation.  Deleting a Profile prior to reboot would take 15+ minutes, after reboot took 46sec.

[ChrisLynch]

We released the 2.00.07 patch last Friday, 24/06/2016, which one of the fixes resolves the 15min Server Profile operations you were experiencing.  I would highly suggest folks start planning on upgrading their 2.00.02 appliances to 2.00.07.  As always, perform a backup and save it prior to upgrading.

[csouthy]

Thanks Chris, great news

[PatrickLong]

I don't want to make too big a deal about it, but I too was given the recommendation by support personnel to increase the resources allocated to the appliance.  Also interesting to note, this HP Advisory released 2016-06-24 gives specific guidance to do the same...

"To further reduce the risk of exposure to this issue, increase the number of vCUPs to 4 or greater and the amount of memory to 16GB for the OneView appliance."

http://h20566.www2.hpe.com/hpsc/doc/public/display?docId=c05185536&hprpt_id=HPGL_ALERTS_1910910&jumpid=em_alerts_us-us_Jun16_xbu_all_all_745865_1910910_ServersEnterpriseSolutions_critical__/

Not trying to tip the apple cart, just wanted to point out the discrepancy.

[ChrisLynch]

@PatrickLong, Yes, there are some cases where increasing the vCPU and vRAM to 4 and 16GB respectively will help avoid certain cases.  We are not advising that all customers increase these values.  Yes, they can, we don't prohibit.  If customers can support 4 vCPU and 16GB of RAM, we aren't going to stop them.  However, the appliance is not going to scale to utilize all of the new resources provided.  We are looking to provide that functionality in a future release.

We limited the appliance initially to those values so we couldn't impose unreasonable sysem resource requirements.  We are actively not only rethinking that approach, but will deliver the ability to scale the appliance resources when provided.

[markzz]

Hi All

I'm reading this forum and see your finding OneView poses some challenges.. i can only agree there.

15 Minues to apply a Server Profile.. I'm envious..

From the time I insert a new blade into a chassis, I wait for Onview to clunk along for 30 minutes or so I can then apply a profile.. Applying the profile will take an additional 20-30 minutes.

This is so very frustrating as I'm sitting here at midnight waiting for a profile to apply right now. (if it works).

I'm frsutrated.. 40 minutes later OneView thinks the profile is still applying. But I can see the blades console and therefore can confirm the server is up and running it's OS, it just can't see any NIC's or HBA's.

Yes Chris, I know your going to tell me to call support and log a call.. Excellent idea. Sorry I don't have the 4 hours I'll need to spend with with level 1 and 2 support so I can get to someone in level 3 who escalates to the engineering team..

OneView is not your finest piece of work HP..

None of these issues when you use DL series servers.

[ChrisLynch]

@markzz, from the time you insert a new blade into an enclosure, OneView should not take 30 minutes to discover it.  On average, and in the vast majority of cases, discovering a new BL Server after inserting it into an enclousre managed by OneView will only take 7 minutes..  If you are going to the discussion, it would be extremely helpful to better understand your environment.

• What version of HPE OneView are you using?
• What type of hypervisor host is the OneView appliance deployed to?
• What is the source blade server you are trying to manage?
• Is the OneView appliance and enclosure in the same Data Center?
• When you try to apply a server profile, what are you trying to configure?  Connections? BIOS? Boot order? Firmware? Storage Volumes? Local Storage?

All of the above can directly impact Server Profile management, including DL's.

Yes, I advise customers to open a support case, as the forums here are not an official means to obtaining product support.