Shifting to Software-Defined
Showing results for 
Search instead for 
Did you mean: 

Upgrading to HP Operations Orchestration 10? Here are Sentry Insurance’s tips for success


Dan Hyman.jpgBy Dan Hyman, Systems Engineer, Sentry Insurance


Editor’s note: This article is part of an ongoing series of guest posts by HP Software customers about Automation and Cloud Management use cases.


Like any important IT infrastructure, upgrading a critical automation platform like HP Operations Orchestration for task automation requires planning and careful execution — especially if you choose to do it in-house.


Here is how Sentry Insurance made the move to HP Operations Orchestration v10 successfully, with some tips for organizations that are considering an upgrade.


At Sentry Insurance, we were running HP OO v9 on a two-node cluster. Each node was running Windows 2008 R2 and hosted the Central/RAS/Terracotta services. The backend was one Microsoft SQL database (also Windows 2008 R2) and failover was configured on F5. We started looking at HP OO v10 for a few reasons. We encountered complications with Terracotta load balancing and wanted to move away from the software load-balancer and onto a hardware load balancer. OO no longer uses Terracotta in v10. We also wanted a better Source Control Management solution and HP OO v10 introduces SVN support.


In-house upgrade or outsource?

We’d been working with OO for a couple years already and were very comfortable with it, and after reviewing the various options for upgrading, we felt it was something that we could do in-house, given our expertise with the product. And if complications did occur, we had an existing agreement for third-party support that we could use.


Three tips for upgrade prerequisite planning


  1. Build new servers and upgrade side-by-side — You will need to decide if you want to do an in-place upgrade, a side-by-side upgrade on new hardware or install from scratch on new hardware. Upgrading side-by-side allowed us to build out our new HP OO environment while maintaining our existing v9 system. This gave us the ability to move at our own pace and cut over to the new environment without disruption in service.
  2. Back up production and non-production content from old system — When you create a content migration plan, consider that authors may have important flows developed in non-production that haven’t been published to production yet. When we first migrated OO content, we were only going to bring over production flows. However, shortly after our first migration, some flow authors questioned where their non-production flows were. We were able to recover from backup, but it would have been much easier to migrate content from both environments from the beginning.
  3. If HP OO servers are VMs, take snapshots prior to installation of new system — You will need to ensure you have backups of applications and content within HP OO v9, in case you have to fall back for some reason. If you choose to do an in-place upgrade, and your environment is virtualized, taking snapshots is an ideal way to lock in your current working environment.


Of course, you should upgrade in a non-production environment first and test the content migration plan.


Content Migration

You will need to take several steps for a successful content migration:

  • Ensure all flow authors have checked in all flows/operations
  • Check that there are no invalid flows (such as missing/broken steps, invalid data, etc.)
  • Eliminate any “sleep” scripts from HP OO v9 flows — these are not supported as scriptlets in HP OO v10 (Note: this is not in reference to the sleep operation, only the sleep scriptlet option on flow steps)
  • Consolidate all content into one repository for the initial migration if possible; it’s much easier this way
  • Once your content migration is complete, import the converted content into v10 Studio and fix anything that didn’t migrate properly (just make sure that prior to importing your migrated content, HP OO v10 content packs are imported into Studio; store all of your system accounts, system properties, etc. in one content pack for each)
  • Once the flows are cleaned up, re-organize content by separating them again into individual projects that pertained to their specific function — for example, one project each for networking, storage, server builds, decommissions, etc.


Post-upgrade system configuration

Today, Sentry Insurance continues to run a two-node cluster for HP OO v10, now running Windows 2012 and hosting Central/RAS worker in the central cluster. We now have two Windows 2012 SQL databases mirrored as a SQL cluster. Users access OO via a load-balanced Virtual IP configured on an in-house F5 Local Traffic Manager (LTM). The two OO node reference the backend via a Virtual IP with a fail-over configuration on an in-house F5.


Here are a few tricks we’ve learned regarding system configuration:

  1. Avoid OutOfMemory exceptions during WinRM PowerShell steps — increase the memory allocation per shell:
    • set-item wsman:\localhost\shell\MaxMemoryPerShellMB <value in MB>
    • Default (in PS 3.0) is 1024 MB — increasing to 2048 MB should be sufficient in most cases. Sentry uses powershell steps heavily, and we found that some of our flows were failing to execute when PS hit this memory threshold. Tuning your Worker servers to account for this memory allocation may be necessary as well.
  2. Avoid running out of sessions while running flows with PowerShell steps concurrently — increase the MaxConcurrentUsers and MaxShellsPerUser (the latter being most important):
    • if you use multiple service accounts, set-item wsman:\localhost\shell\MaxConcurrentUsers 30 ß
    • if you use the same service account(s) concurrently, set-item wsman:\localhost\shell\MaxShellsPerUser 100 ß
  3. If you use PowerShell operations heavily, you’ll want to set up a scheduled task on your HP OO Worker servers that looks for and clears out old sessions — This applies to PowerShell operations that leverage WinRM (as many of ours do). If written properly, PS sessions will be closed when scripts complete, but if the script fails mid-way through, or if the script isn’t written properly by an author, it could leave a stale session out there. Old sessions hang on to memory and count towards the MaxShellsPerUser wsman setting


Also if you have multiple departments that can author OO flows, create a separate SVN repository for each so that sensitive data, such as System Accounts, do not get compromised!


Working with HP OO’s new features — and what comes next

The upgrade to HP OO v10 was a great success for our organization. Several new features stand out to us as key improvements, such as the ROI focus on the HP OO dashboard and the modern look/feel of the Central interface that’s now based on the HP Experience style. From an architectural standpoint, we’re pleased with the SVN repository and being able to use in-house F5s instead of a software-based load-balancer.


Live scalability — the ability to build new Central instance and point to existing schema on the fly — has been key for us, as has the way that content is now packaged in Content Packs and delivered to Central on the fly, so the repository in Central is a thing of the past. Flow hand-off within Central works well, as does having the complete REST API to the entire set of Central functions. Permissions have also moved from Studio to Central and appear to be more stable than before.


Now Studio functions as a stand-alone IDE and no longer has a dependency on the repo, which makes for a smoother experience while developing flows, but Studio also maintains the same look/feel as v9, so authors will be very familiar with the new environment.


Of course, we’re not stopping there. Our plan includes integrating with HP Services Access Workbench and upgrading to HP OO v10.21. Also on our roadmap is integrating with the Amazon cloud for Sentry’s future IaaS work, and continued enhancements in automation, especially to increase ROI by expanding HP OO’s usage throughout our infrastructure.


But upgrading to v10 has been an important and successful step in our longer journey.


Learn more

Read more about HP Operations Orchestration


Read the other blogs in this series:

About the author:

Dan Hyman is an IT Engineer with ten years of experience in IT Infrastructure; specializing in automation and product integration. Certifications include RHCSA, MCITP, ITILv3, and HP-AIS OOv9

About the Author


Nimish Shelat is currently focused on Datacenter Automation and IT Process Automation solutions. Shelat strives to help customers, traditional IT and Cloud based IT, transform to Service Centric model. The scope of these solutions spans across server, network, database and middleware infrastructure. The solutions are optimized for tasks like provisioning, patching, compliance, remediation and processes like Self-healing Incidence Remediation and Rapid Service Fulfilment, Change Management and Disaster Recovery. Shelat has 23 years of experience in IT, 20 of these have been at HP spanning across networking, printing , storage and enterprise software businesses. Prior to his current role as a Manager of Product Marketing and Technical Marketing, Shelat has held positions as Software Sales Specialist, Product Manager, Business Strategist, Project Manager and Programmer Analyst. Shelat has a B.S in Computer Science. He has earned his MBA from University of California, Davis with a focus on Marketing and Finance.