Nov 1, 2018
Bob chats with Microsoft Azure DevOps Product Owner and author of Agile Software Engineering with Visual Studio, Sam Guckenheimer, at the DevOps Enterprise Summit 2018.
Connect with Sam and Bob on Twitter.
Transcript
Sam Guckenheimer ‑ DevOps Enterprise Summit 2018
Bob Payne: "The Agile Toolkit."
[music]
Bob: Hi, I'm your host Bob Payne. I'm here at the DevOps Enterprise Summit 2018 with Sam Guckenheimer. Welcome, Sam.
Sam Guckenheimer: Thank you, Bob. It's great to be here.
Bob: It's the first time we're really chatting. We chatted a tiny bit last night. My colleague Sanjiv Augustine said you were instrumental in hosting The Agile leadership network when it formed and came up with the declaration of Interdependence. How did that end up coming about?
Sam: Well, that was no what 14 years ago or something like that.
[laughter]
Sam: What we saw was at that time that this was of course way pre‑DevOps. The Agile community had fractured into many groups saying "More agile than thou." That seemed stupid.
Bob: That fracturing has continued and remains as stupid today or... [laughs]
Sam: Yes. Unfortunately, the fracturing has continued and it hasn't gotten less stupid. That was the reason for trying to get the interdependence declaration together to get these leading lights from what was then the Agile community working together.
In the meantime, the pure Agile has largely been eclipsed by DevOps. As you see something like this DevOps Enterprise Summit going on its fifth year roughly doubling every year in scale. I'm here now. Still there.
[laughter]
Bob: There are a number of things that I found at this conference that I haven't been able to make a ton of sessions because we have a booth. I've found that I haven't really learned, maybe this is my own fault, anything at the Agile conferences for probably about 10 years. It wasn't any substantially interesting information.
Sam: That's correct. I last keynoted at the Agile conference in 2014. That's probably the last time I've been there. It got kind of stale. The energy in innovation, in practice I think has really shifted to DevOps. That's come about, because the DevOps' definition of Dunn is not potentially shippable and promotes...
[crosstalk]
Bob: It's captured a value, enlargement value.
Sam: It's live in production with Telemetry that is demonstrating the value delivered.
Going from a world where you were effectively stopping at an intermediate activity that didn't reach the customer or end‑user to go to one where you have to reach the end‑customer and you have to measure the value delivered, is much, much more powerful for all the stakeholders, for the business, for the people involved.
It's much more satisfying. You disintermediate the development to customer relationship. You think of things as one engineering discipline, not as silos post the Scrums, so to speak.
Bob: Certainly there were a number of great Agile teams and organizations that fully believed that Dunn meant in the hands of customers and delivering whatever goal, that...
[crosstalk]
Sam: I do not mean to bash anyone. I certainly think there great Agile teams. A lot of what we do today has its roots in extreme programming, but things like XP at the time, had this notion of, for example, pair programming.
We have largely, as a community, moved to the notion of a pull request as a virtual pair programming. We have moved from the idea of onsite customer to measuring customer impact, which isn't to say onsite customer is a bad idea, it's a great idea, it's a rarely achievable one.
All of these seeds that were planted back then in the late '80s by the early Agilists were important seeds. The garden where I think they're really bearing fruit now is in this DevOps community.
Bob: The other thing that I think is probably the next wave that we will see in organizations that are not already there, certainly, many organizations have already integrated business into this flow. Without that DevOps is necessary but not sufficient to actually change the outcomes that businesses are seeing.
That's the next frontier for those companies that we're not sort of born in the world of IT as the fundamental driver of business outcomes.
Sam: That's correct. DevOps is the flip side of the coin from digital transformation. Digital transformation is the business term for taking your business model and turning it into one that can improve continuously in an Internet‑powered age. DevOps is the shorthand for the technical practices that enable that.
Bob: I see way too many organizations mistaking a DevOps transformation for digital transformation. They're fundamentally doing the DevOps practices, but they're not backing up into the initial value proposition to begin with. That will sort itself out.
Sam: This is a common thing of confusing means and ends. The ends are things around growing the business customer, acquisition customer, engagement customer, employee delight, all of these measures of happiness and success.
The practices are ways of getting there. The goal is to focus however on those end results. The clear sign of dysfunction is when you see people measuring the inputs, not the outputs.
Bob: If Deming or [inaudible 7:33] came back and saw that Toyota was doing the same practices it was doing 75 years ago they would drop dead after having just come back to life. [laughs] In real systems the practices and the processes are never the ends. They are all in service of maximizing flow...
[crosstalk]
Sam: Exactly. If you think about the evolution, the practices today are different because the constraints are different. One of the overriding constraints was for example infrastructure availability. You get all of the stuff around how to manage and schedule the infra.
Today with the public cloud that constraint is gone. It's a classic example of, in Eliyahu Goldratt terms, elevating the constraint or removing the bottleneck. Then you see the constraint shifting. As you're adopting these practices what happens is you have a continual shift of the constraint, and you have the next one to attack the next bowling pin to knock down.
[crosstalk]
Sam: Right. What DevOps says has basically taught us as well. You can remove infrastructure a constraint by using the cloud. You can focus on the value delivered to the customer and measure it so you can have both qualitative and quantitative view of that.
You can take the quality game and shift it left and right so that quality does not become this big testing bottleneck in the middle. It can become part of the pull request flow. It can happen before code merges.
Then you can in production gradually expose value to more and more of users so that the blast radius is something that's flexible, so you don't have the constraint of saying, "I need to master my MTBF in order to release."
You can say, "I need to maximize my ability to recover and may have the shortest time to recover, so that by controlling the blast radius and being able to recover quickly I can experimentally by increasing the rate of experimentation I can deliver and measure value delivered on a cycle that was never possible in the old days."
It wasn't possible before we had the Internet, it wasn't possible before it hit the public cloud, it wasn't possible before we had these practices of high‑quality, highly‑rugged automation that we do today.
Bob: Yeah it has been a sea change since I did Fortran on punch cards [laughs] .
Sam: There have been many sea changes yes. Mike Pearson gave a great talk yesterday, borrowing from Carlota Perez on the structure of industrial revolutions, and postulates that we're at the point of disruption from the period of adoption to the period of dispersion.
That would account for a lot of the changes that we're seeing, and it would account for a lot of the anxiety that you see among people who are saying, "How do I learn fast enough? How do I catch up fast enough? How do I get ahead?"
At the same time, what you see very clearly reflected in company success, company's market gap, and company's ability to innovate and pivot, is that the ones who have mastered the go‑fast‑without‑breaking‑things‑and‑adjust‑course‑as‑you‑go, are the ones that are winning in pretty much every sector.
Bob: I love Mark Schwartz's analogy of the battle of the Russians with Napoleon, and the speed of decision‑making being fundamentally out of sync with the reality of the battle.
Sam: Exactly, that was also true on Omaha beach in Normandy, that was true in Vietnam, that's been true in pretty much every military conflict, that the degree of autonomy and speed of innovation has determined the outcome in the end, and people who are great at enabling the next war instead of fighting the last ones, are the victors.
The latest example decide or...I don't know if that's politically correct to go there, but you see it now in...
Bob: [laughs] That have been substantially politically correct on this podcast [laughs] .
Sam: You see this in cyber. The Russian budget for cyber is less than the cost of an F‑35.
Bob: No one could argue that the F‑35 is more costly than it needs to be but it's... [laughs] .
[crosstalk]
Sam: Who cares? The point is, they're not trying to win the manned aerial dogfight. They are extending the notion of total war to a new battlefield and they've been very successful, but finding the place where there are no defenses and where it's possible to innovate quickly and it's proven to work.
You could also argue that as David Sanger does in "The Perfect Weapon", that the US started this cyber‑war arms race. In any event, we've not follow through on the consequences of what we started.
The military analogies, they turn some people off, but they have their value. We are, and the rest of society also, in a place where we need to be winning the future, not the past.
Bob: It's actually one of the analogies I quite often use when I'm talking to people that are OK with the military analogies.
The OODA loop, the Boyd loop of observe‑orient‑decide‑act. The team that can turn that loop the fastest, whether it's Amazon, Netflix, or a manned‑aerial dogfight, or a cyber‑attack, is going to win.
Sam: Exactly. In our world, the OODA loop results in some kind of software or service delivered. One of the things we know from measuring it is that about a third of the time, we get the results we'd want, a third of the time, we get opposite result from what we hypothesized, and about a third of the time, it makes no difference.
The implication of that is that you want to be able as quickly as possible, to double down on the successful third and fail fast or pivot away from the other two thirds, which means that you need to make the OODA loop as short as possible, which is what Boyd talked about in his idea of aircraft design and aerial battle.
That's exactly true in how we develop and that means not just using small batches which Agile taught us. That means not just breaking down the silos, but it means really focusing on time to remediate and focusing on quality to the left so that you have clean delivery and you have the mechanism in production to control exposure and to go faster and wider as you need to.
Bob: You mentioned the one‑third, one‑third, one‑third, I know that was a study that came out in Microsoft. Actually...
[crosstalk]
Sam: Ronick O' Harvey was behind that. Ronny is now a technical fellow, he wasn't back then. He basically took a very large sample of "improvements" that were delivered.
Let's measure, are these really improving, what we wanted? The result was a third of the time, in other words, I've confirmed the others' change is bad, unless is great. That was quantitative demonstration of that. I don't know if he published that before he did a stand for PhD or after, but it was a famous study and it holds up.
Bob: I also very much like this idea of very small batches, because without the small batches, it's hard to get attribution of what improved the customer experience and what was neutral or negative, because you're conflating way too many changes if the batch is large.
Sam: That's why the pull request flows becomes successful, because you can make the pull request a batch that is a few lines of change, it's possible to have a human‑code review on it, and it's possible to have extensive automation on it.
Again, an example of a practice that wouldn't have been possible pre‑cloud is when we do pull requests, we run the build‑in automation on them with typically 80‑some thousand tests before asking for the human‑code review. Human eyes are only focused on those things that automation has said looked good already.
As opposed to the way things were done, pre‑cloud in the XP pair programming model, where human eyes were first defense. That was very appropriate given the constraints at the time. The constraints of today are different.
Bob: That was certainly one aspect of pairing. The other is just as the design discipline getting the collaborative design quite often yields better results, but...
[crosstalk]
Sam: I totally think that people should collaborate on design. I'm totally for that. I'm not trying to..
Bob: I totally get the point about the quality. Is automation...we want lazy engineers [laughs] . We want engineers focusing on creative thought, rather than repetitive action.
Sam: Exactly. Another example of that that's possible these days, is you want a very high reuse, an open source. If you can solve a problem with 30 lines of code and reuse thousands, that's much better than creating 3000 lines of codes that need to be maintained.
In effect, we want to reward people for writing less code, which again turns on it's head, one of those classic input matrix and myths of, "Well, how much code did you write? How busy were you? How many hours did you put in?" As opposed to, "What result did you achieve?"
Bob: What are some DevOps practices that have really changed Microsoft fundamentally? I know you've got a couple of talks related to that here at the conference.
Sam: I bucket our lessons learned, usually in five groups. One is how we focus on value delivered to the customer, both quantitatively and qualitatively, and let that drive the way we think about what we're delivering and how we measure that.
Two is how we apply production‑first mindset. Our CEO tends to call this a live‑site culture, in other words, you build it, you test it, you run it, you secure it, you troubleshoot it, you improve it with responsibility residing in you, the creating team, not getting fragmented across these Silos.
Three is the idea of team autonomy and enterprise alignment that you want to let teams at the level of the feature crew Scrum, cream squad, whatever your favorite term is, you want to let these small feature crews work autonomously on their stuff, and control what they are taking into the next sprint or what items they're doing next.
You want them to support their stuff in production and you want the mechanisms to align their work up to the common business results, so that they know which needles they need to move by the work that they do and they know how to view those gauges.
The fourth is shifting quality left and right so that you can get a signal in the pipeline of green meaning green and red meaning red, and in production, you can see continually what is happening with every changes, you expand its exposure.
Fifth finally is using cloud to make infrastructure‑flexible resource.
That's how I bucket it. I did one talk yesterday with my friend Ellen Smith about how we moved our DevOps' ass. It's really a story about eight years of taking what's started as a non‑premise product and turning it into a cloud service and on‑premise product. That was an attempt to myth‑bust the idea that if you're going to the cloud, you need to start in the cloud and throw everything away.
It was an attempt to say, "Here's a proof instance where we had a business, pre‑cloud, with on‑prem product. We preserve that and made it better, and use the same codebase to go the cloud where the cloud is making the on‑prem better." Of course the cloud's the majority of usage and the fastest growing now, but it wasn't a throwaway, which of that story.
The other one which is similar, which I'm doing tomorrow, is a talk about Windows' journey to DevOps. Windows division is the extreme case of scale and legacy, and they have successfully moved to DevOps. There were a bunch of bumps along the way. For example, to get Windows to be able to use Git at their scale, we needed to fix the Git, and that took three attempts.
Bob: Really?
Sam: Yes.
When we started doing something like a Git clone of the main Windows repo, took 12 hours. That was if the network didn't burp, or your laptop didn't go to sleep, or nothing else wrong happened. If any of those things did happen, then the whole operation needed to start over.
Bob: Need to start again, yeah.
Sam: That now takes a couple minutes. We did a series of 300x or better improvements in Git performance with what is now open‑sourced as the virtual file system for Git. Windows motivated all of that to be able to support their scale of codebase, which was hundreds of times larger than anything else anyone was using.
Bob: That's interesting. I did not know that you guys were major contributors to the Git.
Sam: We're one of the top two. We're the largest open‑source contributor of any company, have been for about two years now. Git is a project where we have been very heavily in, and virtual file system is one of the latest aspects of that.
Come to the talk tomorrow.
Bob: OK, I may. What time is it?
Sam: 11:25, I think? 11 something. The times here are weird. All these weird five‑minute increments.
Bob: [laughs] . It is five‑minute increments and three hours off, because I'm an East Coast person. Are you out of...
[crosstalk]
Sam: Along Seattle.
Bob: Thank you so much, Sam. This has been great. Is there any one thing you'd like to close off with that you're interested in?
Sam: Yeah. There's something that I'd like to make our listener aware of, and that is I curate a website. The short link to it is aka.ms/DevOps. It's, DevOps and Microsoft.
What I try to do is to put up our experience reports there, not the high‑level marketing level stuff, but like, "How did you actually do the change in testing? How did you go to no downtime deployments? How did you start using service reliability engineering? Etc."
There're about 50 articles up there, but half of them with good video. They're just stories about how we work. I love people to use that as a...
Bob: As a resource?
Sam: ...open resource.
Bob: Thank you very much. It was very nice meeting you and chatting.
Sam: Thanks a lot, Bob.
Bob: Thanks.
The Agile Toolkit Podcast is brought to you by LightSpeed.
Thanks for tuning in. I hope you enjoyed today's show. If you'd like to give feedback or be on the show, you can ping me on Twitter. I am @AgileToolkit. You can also reach me at Bob.Payne@lithespeed.com.
For more free resources, transcripts to the show, and information about our services, head over to LightSpeed.com. Thanks for listening.
[music]