HPE Business Insights
Showing results for 
Search instead for 
Do you mean 

DevOps gurus track the value—and ROI—of DevOps

Guest Blogger (HPE-SW-Guest) on ‎07-23-2012 11:02 AM

Paul Muller-Gene Kim.jpgBy Brian McDonough, Discover Performance managing editor


HP Software Chief Evangelist Paul Muller just posted a podcast on his Cross the Streams blog—a conversation he recorded at Discover Las Vegas with DevOps luminaries Gene Kim and Patrick Debois. Kim and Debois are on the team writing The DevOps Cookbook, out later this year, and they shared their thoughts on the movement, what it really offers to business leaders, and how you get your ROI.


Paul’s got the audio, but only we would sit down and type you up a full transcript.  So pop over to Cross the Streams to launch the sound, then come back here to follow along.  After that, check out our Q&A with Gene Kim, and the rest of our new DevOps issue.


- - - - - -

Paul Muller:  I’m joined today by Gene Kim, author of a number of books in the area of IT management, and in fact an upcoming book, which I think is going to be quite interesting to CIOs: When IT Fails: A Business Novel. And by Patrick Debois. Patrick is one of the—probably the—world’s preeminent experts and thought leaders in the area of not just DevOps, but culture and change management around people, processes and the likes to deliver IT faster, better and more securely.


Patrick Debois:  Great to be here Paul.


When do you need DevOps?

PM:  It’s good to have you all. I thought I’d take the opportunity while we’re together to have a conversation that I’ve been having a lot with CIOs lately, which is how they should be thinking about DevOps [and] DevOps principles; what it means to them—and more importantly, what it means to them in terms of mobilizing their people. I’ll start by throwing out a question to both of you: What are the signs that a CIO or a senior leader should be thinking about or considering DevOps? Patrick, do you want to start?


PD:  Sure. I think in general the idea of DevOps is centered around getting faster feedback from your IT at the beginning of the market to where your users are using it. So if you see, as a CIO, a very large lag between having your IT challenged or checked within the market, I would say that’s a sign of some bottleneck somewhere between the idea and [delivering] it to the customer. DevOps will focus on whether your bottleneck is actually somewhere in the handover between Dev and Ops.


GK:  I totally agree. In fact, I think the goal is faster feedback loops, and I think the way most IT organizations are configured, it’s almost guaranteed to be the exact opposite. And so there’s this downward spiral that I’ve seen in my 13 years of studying high-performing IT organizations, where you have fragile applications in production. When things break, there’s a long mean time to repair, there’s a lot of firefighting. The business starts missing the commitments it made to the outside world, and something terrible happens: They start making even more audacious commitments to the outside world, which means now development sees even more urgent date-driven projects put in the queue, which means even more shortcuts are taken, more fragile apps, ever-increasing tension between development and operations, and all this means reduced throughput, higher lead time. If that’s something that CIOs see, well, the good news is that they’re not alone, and that there is actually a prescription for this.


What can DevOps do for me?

PM:  Following on from that idea then, it sounds like it’s somewhat related to their Agile delivery efforts, it would be fair to say. I mean, the two go hand in hand. The question is: What are the primary benefits that a senior leader would see from it? Changing people, changing process, none of that is cheap, Layer 8, [that is, the people who actually use the applications] is very expensive to manipulate. What are the benefits?


GK:  One of my favorite points that I’ve just seen very recently was in a talk given by Scott Cook, the founder of Intuit. He said it was all about how many experiments you can do in terms of maximizing the customer acquisition, maximizing conversion rates. He said one of his success stories is at TurboTax; they actually did over 40 simultaneous experiments during the peak filing season. They were able to significantly increase customer conversion rates and figure out exactly what it takes to get customers to sign up. So what is the ROI on that activity? When we talk about DevOps, these are the capabilities that you need in addition to Agile to be able to do that kind of fast throughput, fast experimentation.


PD:  Yeah, I agree there. It’s almost like the Lean Product [philosophies] would give you an idea whether your idea is feasible or you can put it into the market, and so development will tell you whether you can build this software and everything. But if you go the extra mile, meaning go through production and actually into your operations, in your production, this will give you an idea of whether the users actually will like it when you build it, and whether the idea was feasible. So that feedback, if you can turn that around from IT to production very fast, your return on investment is that you can adjust much faster to what your market wants. So it’s not like a death-march in a certain direction that will give you an idea after a couple of months; you can almost …. After a few days, once you get the whole automation [in place] and [start getting] … feedback, metrics, it gives you a way to be more flexible in making decisions on your platform.


GK:  What’s the conversation rate? What is the A/B test yielding? What functionality is actually being used? By the way, you might like this Patrick. I heard this the other day. Clyde Logue said, “Agile is what helped development to regain relevance with the business, but it left IT operations behind.” And so, so much about what DevOps is, is that missing piece that allows all of IT, the whole delivery chain to achieve the promise of what Agile brings to it.


PD:  Yes, and the way I visualized it last time, it’s almost like you as the idea owner, it’s like sonar …. So your idea from yourself to the production, and so the faster you can use sonar that prove the viability of the idea, that gives you actually a competitive advantage over your competitors.


It’s not “the ROI of DevOps” …

PM:  So we should be thinking about it then in terms of, at least as you described it, in terms of accelerated benefits realization, and so shortening that [cycle]. It’s really not the ROI of DevOps, it’s really more that the ROI of your original project can be realized sooner.


GK:  Patrick this could be an entire separate interview to write about, like—you know, you said something, Paul, that just blew me away: If we were really honest about how long it takes for the customers to get the expected value, for the business to get expected value, most projects would never even clear the internal hurdle rate. Their internal rate of return wouldn’t even—you know, the project never would have been approved, and I think that’s sort of a dirty, dark secret that a lot of the DevOps practices are specifically designed to diminish.


PM:  Yeah, and I think we should share that work-in-progress sort of cost model that we’ve built there.


GK:  Brilliant.


DevOps: Why haven’t we had it all along?

PM:  So here’s the big question then, and it’s the question of time immemorial: If this is all so obvious, you know, Dev should work with Ops to accelerate deliveryhello, my Dev and Ops teams are already working together, what’s going wrong? What’s different?


PD:  Well if they’re doing okay, this is nothing that you have to change, obviously, but in most of the cases what we see happening in companies because of the siloed management is that each of these groups are optimizing for themselves. The developers are optimizing to get new features out, but they care less about the whole stability or anything that you can measure in production. The Ops guys are more focused on keeping things stable, so they kind of go away from change and aligning the goals. It’s all about getting that idea out faster, but in a stable situation, that aligns the common goals between those two teams. If you already have that, I’m sure there are other things that you could optimize, but that’s one of the primary focuses you should be working on there.


GK:  I think just to show how good it can be, back when I studied high performing IT organizations, we thought a thousand production changes was fast in a given week. In 2009, the state of the art was 10 deploys a day. Now Amazon has gone on record saying they’re doing a thousand deploys a day to their customer-facing infrastructure, and that shows that the agility and nimbleness that was once—you know, what was competitive 10 years ago is not competitive now, and the bar keeps on getting higher. And so, if your organization can’t sustain those types of experimentation rates, then chances are, it’s going to be because of an issue between how well development and operations work together.


Align your goals

PM:  I think Patrick has touched on probably the most fundamental point, and we talked a little bit about this yesterday, which is: If my goal sheet says that my job, for example, within operations is to ensure 100% reliability, the natural place to go to is to cease all change, because we know that change is usually the reason for potential downtime or outages. I think your point there, Patrick, is if you don’t get the goaling aligned horizontally, you’re never going to achieve that accelerated flow.


GK:  Right, yes.


PD:  Yes, but to be sure, in most of these cases where there is this bottleneck, you will see these big changes being thrown over the wall, so if you want to go faster and do multiple deploys a day, it only makes sense if you start reducing the size of your changes, otherwise your risk will increase. If you’re doing big changes all the time, they’re going to be hard to understand, but if you make them smaller and more frequent, this has been proven as a way to kind of mitigate the risk of the increased change rate you’re doing.


PM:  Reducing batch size, much as you do with manufacturing, right?


GK:  Exactly right. What pains me when I hear this is that I’ve seen so many senior managers say, “The deployments are so painful and so catastrophic, we need to lengthen the deployment interval,” which is exactly the wrong thing to do. We need to keep shrinking batch sizes, and so whenever you make a decision that says, “The releases are so expensive, let’s do fewer of them” just think, “This is actually taking us in the wrong way.”


PD:  Yeah, and also you could say, “Okay, I increase that. I do smaller changes.” A lot of people are still scared of [adopting] that change rate. There are some things that you can do to mitigate that, like have canary releases, dogfooding, or …. Because there is actually a difference between doing multiple deploys and doing multiple releases. So that’s one important thing to note if you’re scared about increasing your velocity of change there.


Patterns of success

PM:  Finishing off then: What are the patterns of success and the patterns for failure? Just a couple of high level points that IT leaders should think about if they are considering a DevOps move.


GK:  May I share my favorite three patterns as we have been working on this DevOps Cookbook?


PM:  That’s a great idea, let’s do that.


GK:  And by the way, working with Patrick and the team has been one of the most fantastic journeys I’ve had in my life. The three [patterns] that are my current favorites out of the DevOps Cookbook work are: 1) How do you embed Ops into Dev? Get those nonfunctional requirements to a document and user storage in a way where development can build them in the earliest parts, and actually start building code and the environments at the same time in the earlier sprints. I mean that for me was sort of like the single minute exchange of die idea in lean manufacturing—I loved it. 2) The second is: How do you get Dev into Ops? Putting them closer to the customer, get them in level 3 escalation so that …. Patrick Lightbody had this quote: “When we woke up developers at 2:00 a.m. we found that defects got fixed faster than ever.” I was like, “Wow, fantastic!” So that sort of shortens and amplifies feedback loops. 3) The third is just the use of kanbans so that we can actually manage not just the work but the time between the work. Like when you’re dealing with handoffs more and more the touch time—you know, the task is spent in wait time not in touch time. Those are currently my three favorites.


PM:  Maybe Patrick can answer from a slightly different perspective, which is: Are there known patterns of failure? Like, just don’t touch this, this is the third rail; if you take this mentality you’re bound to have a bad day?


PD:  Well what can happen is that people get too focused on optimizing the Dev and the Ops problem and their bottleneck isn’t there. So we’re dealing with the same problem as we’re optimizing in Dev or in Ops, but it might be that the HR incentive is given differently, like one person is responsible for all the middleware and nobody can touch it because his bonus depends on it. You can try to optimize whatever you want, but this is likely a constraint outside of your team but within HR or financial that’s actually hurting you. So be sure that whatever bottleneck you’re trying to optimize that is actually hurting your problem is the correct one.


The value of small, effective moves

PM:  Wrapping it up, I’m just thinking now that in summary what I’m hearing is DevOps is really about accelerating the velocity, the potential frequency, and most importantly the reliability and predictability of application changes, and therefore innovation is really what I’m hearing. And that the business benefit from that is not necessarily the automation or any of those particular smaller elements, although there may be ROI there, the ROI is really about getting your business benefit faster as a result. The key thing here is getting your teams to collaborate, but also changing the behavior or the type of work that they handle from big projects to a lot of smaller, more manageable change, and making sure that in the process of all of that that you don’t optimize locally for goals, that you’re rethinking people’s goal sheets. Is that a fair summary of all of that?


GK:  Yeah, exactly, so that you work on small things. And then make sure that they make it all the way through the system, through Dev, through Ops, right to the customer.


PM:  Small is beautiful.


GK:  Yes.


PD:  What I want to add to that is that automation is obviously a key part. But you can automate whatever you want, [yet] if you don’t learn about what you’re doing wrong in production … Getting the metrics and the monitoring and all that feedback back within the decision making is crucial, because you can go automate whatever you want and go faster, but if it’s crap … it’s just automated crap if you don’t learn about it.


PM:  Automatic chaos.




PM:  Fantastic. Guys thanks so much for taking the time to join us today. I appreciate you getting in touch. If the folks want to get in touch with you, Gene, how can they do that?


GK:  Yeah, in fact you can reach both Patrick and me. We’re putting our best stuff up on the ITRevolution.com website, both When IT Fails: The Business Novel, as well as the DevOps Cookbook. Our goal is to affect the lives of a million IT workers in the next five years.


PM:  Sounds like a great goal, thanks so much for joining us.

About the Author


This account is for guest bloggers. The blog post will identify the blogger.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
1-3 December 2015
Discover 2015 London
Discover 2015 in London, the ultimate showcase technology event for business and IT professionals to learn, connect, and grow.
Read more
November 2015
Software Online Expert Days
Join us online to talk directly with our Software experts.
Read more
View all