Grounded in the Cloud
cancel
Showing results for 
Search instead for 
Did you mean: 

Culture shift: How Sprint manages people, process and technology to automate server provisioning

NimishShelat

By Chris S. Saunderson, Program Manager/Lead Architect for IT Automation at Sprint

 

Chris Saunderson.jpgEditor’s note: This article is part of an ongoing series of guest posts by HP Software customers about Automation and Cloud Management use cases. You can read the first and second posts here.

About the author: As program manager, lead architect and evangelist for Data Center Automation and Process Automation at Sprint, Chris Saunderson has been responsible for scoping and selecting technology, developing the business case and leading implementation of HP Automation technologies into Sprint IT.

 

 

I’m going to let you all in on a little secret: I hate doing repetitive work. I really do. I don’t mean that I hate working — ask my wife, she’ll not-so-politely disagree. What I hate is the idea of smart people doing dumb things over and over and OVER.

 

Fortunately, I’m in a position to address repetitive work. In this post I will share some of what Sprint has been doing to reduce the overall time for delivery of Windows and Linux systems, while increasing the quality and quantity of builds. But it’s a little bit more than that: I really want to talk about the “How” of moving forward.

 

First, a little background

 

Sprint has a long history of being innovators. Our history begins in Kansas as one of the first telephone companies. As an organization we continue to have innovative DNA, whether it’s in long distance, fiber optic networks, or high speed Internet networks, and a lot of innovation in wireless.

 

In IT, we’ve been keeping up with the pace of innovation as early adopters of virtualization in the majority of our platforms: first with VMware in 2003, then Power/AIX and Solaris in subsequent years.

 

When we took on the Cloud and Automation suite of products including HP Server Automation (SA) and IT process automation tool Operations Orchestration (OO) in 2011, we were pushing forward on our journey from standardization to virtualization to automation to orchestration. In mid-2014, we started to work hard on system provisioning as a means to remove hands from the delivery chain. As we’ve learned through the process that removing human intervention not only makes things faster, it also introduces a cultural change that must be carefully managed.

 

Building the system

 

We began by looking into the design artifact that describes a system to be built. Each of us has one of these things: essentially a napkin with pencil marks on it, it’s a rough sketch of a path that leads all the way up to a shiny Cloud portal. For us, it is a Service Manager Change/Task pair that documents all the metadata needed to build a system. At a very high level, it looks something like this:

 

OO Provisioning.png

 

Of course, block diagrams over-simplify things. The “Design the Server in SM” box is typically 4–8 hours of an engineer’s time, and it serves a very important purpose, it is a control point for consumption of resources inside the four walls of our datacenter. One of the goals of this IT automation is to multiply the value gained from that time spent — the fewer human interventions in the subsequent steps, the more value is available in this initial design step.

 

But that’s just one of the goals. The other one is trickier to get across in words, but it’s at the core of IT automation: humans need to trust that the automation returns the same result as their own actions. This is part of a larger cultural change within an organization. The majority of the time organizations spend introducing IT automation is rewiring people’s brains to accept that what they used to do is now being done with automation.

 

The hard part: changing people

 

It’s important for the process to be transparent. We ask people (like system admins to help design the very automated processes that will replace how they do work today. First, they must be able to see and trust that that what they are automating will achieve the same end result as if they had done it personally. Second, automation must be seen as being receptive to the ideas people come up with and build a feedback loop to take the good ideas they were coming up with and put them into the automation work stream.

 

Finally, I think it’s critical that their leadership team is open and honest with them about how automation will impact their roles. The more automation that we build means the less repetitive work has to be done, and the more time we can collectively spend on Interesting Things. They need to hear, “We’ve given you time back, now go use it and work on important things.”

The initial results

 

After we understood how a build progresses, we were able to work on each of those yellow boxes in the diagram above. We are very clear about what will be delivered through the automation, and what still needs to be done through human steps (incidentally, this is our punch list for addressing next steps in automation).

 

After many hours of workflow development, we have it pretty much nailed: we can deliver Windows and Linux systems in our virtual environment in an average of three hours — that’s down from five days. Our build quality is increasing, as we add to the quality checking and additional automation to complete each server build.

 

Let’s discuss the flow

 OO flow.png

 

There are two parts to the way the flow above works: a Pre-Check that makes sure that the system can be built (there are no logic errors or configuration inconsistencies), and then a Pre-Check+Build that actually executes the build and makes sure that we have sufficient capacity available for the system to be built.

 

There are some things that this flow uses that are probably worth a bit of discussion:

  1. WSDLs are a “nice to have” We took a solid architectural approach that went something like this: “WSDLs are relatively slow, so let’s just use them for create/update/delete functions, and use SQL for everything else”. Fortunately for us, our Service Manager (SM) team was very open to the idea of creating views on the SM database for us to simply select from to avoid joins and other hideousness. But when we need to update something, then we use the WSDL to make sure we have fidelity in writes and transactions.

 

  1. Boy oh boy, does iteration help! Over time, the good ideas started to come from our internal customers, and by looking at incremental releases rather than monolithic, we could give our customers incremental delivery of functions against a backlog of feature requests. (Wait a minute…that’s almost agile!)

Our use cases for the flow are fairly simple: an engineer vets their design prior to handing it off, and then the user of the flow (our system administrators today) use both the Pre-Check and Build using the flow, only coming back to add the finishing touches to what has not yet been automated. This way, there’s complete transparency from the design to the build and the value comes back into focus. We require people spend less time staring at bits on a screen, and more time on useful work.

 

What’s the future?

 

Most people reading this may have their heads cocked to one side wondering, “But why don’t you just implement a service catalog and be done with all of this?” Oh, believe you me, that’s coming. But inside my company, I walk a tightrope with my process partners. We certainly want to get to that service catalog approach, but in the meantime, I have to keep delivering efficiency in the world that I am in today.

 

That efficiency comes in many forms. One building block is the reduction in time it takes to provision servers, but another building block is having people trust automation — and that is what allows the Business of IT framework I live in to evolve to where we can truly embrace Cloud. Until then, I’m going to keep helping smart people work on smart things and helping my business move faster.

 

Learn more

Read more about HP Operations Orchestration (OO) and HP Server Automation (SA)

 

  • operational intelligence
0 Kudos
About the Author

NimishShelat

Nimish Shelat is currently focused on Datacenter Automation and IT Process Automation solutions. Shelat strives to help customers, traditional IT and Cloud based IT, transform to Service Centric model. The scope of these solutions spans across server, network, database and middleware infrastructure. The solutions are optimized for tasks like provisioning, patching, compliance, remediation and processes like Self-healing Incidence Remediation and Rapid Service Fulfilment, Change Management and Disaster Recovery. Shelat has 23 years of experience in IT, 20 of these have been at HP spanning across networking, printing , storage and enterprise software businesses. Prior to his current role as a Manager of Product Marketing and Technical Marketing, Shelat has held positions as Software Sales Specialist, Product Manager, Business Strategist, Project Manager and Programmer Analyst. Shelat has a B.S in Computer Science. He has earned his MBA from University of California, Davis with a focus on Marketing and Finance.

Comments
Mark Laird

Great article - many thanks for sharing what you have done.

Events
28-30 November
Madrid, Spain
Discover 2017 Madrid
Join us for Hewlett Packard Enterprise Discover 2017 Madrid, taking place 28-30 November at the Feria de Madrid Convention Center
Read more
See posts for dates
Online
HPE Webinars - 2017
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all