Today’s show deals with the sometimes confusing, sometimes misleading term ‘DevOps’. We chat to Brian Guthrie who has worked at a host of different big companies including Soundcloud and ThoughtWorks and has many years of experience in and around the fields that the term DevOps is supposed to bridge. Is DevOps a goal? Is it a job description? Is it a part of the work of both the development and operations teams? We try break this down and get to the core of the idea. During the episode we also try to imagine how DevOps can fit into a more holistic approach in companies and Brian gives us great insight into the way he envisaged the subject being useful. So jump right into The Rabbit Hole, let’s go!
Key Points From This Episode:
Transcript for Episode 47. DevOps with Brian Guthrie
[0:00:01.6] MN: Hello and welcome to The Rabbit Hole, the definitive developers podcast in fantabulous Chelsea, Manhattan. I’m your host, Michael Nunez. My co-host today —
[0:00:09.4] DA: Dave Anderson.
[0:00:11.9] MN: Our producer —
[0:00:12.0] WJ: William Jeffries.
[0:00:12.9] MN: And today, we have a special guest, Brian Guthrie. How is it going, Brian?
[0:00:16.4] BG: Good, how are you man?
[0:00:17.4] MN: Doing great, do you want to give a little introduction?
[0:00:19.8] BG: Sure, so hey, it’s great to be here on the podcast I've known Dave and William for a little while now in various capacities and just also to give you my background I was most recently Director of Engineering over Slice and then, I was at SoundCloud for a couple of years before that working with our product teams here in New York City and then I was - did my own thing as a startup persons that went on and off and I was ThoughtWorks for a number of years before that. So, I’d say, I’ve done a variety of agile style software development over the years and that’s I think that’s really sort of my passion. I come back to applying graphic feedback practices to the process of building great software.
[0:00:53.7] WJ: So, you’re pretty new to this it sounds like.
[0:00:55.4] BG: Yeah.
[0:00:55.5] MN: I mean, you know, we all – we all are in some ways, right? Like, I keep on rediscovering the things that I didn’t understand or didn’t absorb when I first started doing this, you know, many, many, many years ago and you have to bring that learning mindset to it, right? I think it’s humbling to go back and look at what you believe five or ten years ago and say like, “That guy I wouldn’t hire him.”
[0:01:17.5] BG: Yeah.
[0:01:19.9] WJ: It is.
[0:01:20.0] MN: Today we’ll be talking about the real meaning of DevOps and I'm hoping Brian you can shine some light on that. Anyone, have any thoughts on the real meaning, is DevOps not real enough like is there a real meaning behind DevOps.
[00:01:35] DA: People are not just keeping DevOps real these days.
[0:01:35.1] MN: Yeah, we got to keep it real.
[0:01:37.7] DA: It became popular and now what is it, cool kids want to do it.
[0:01:41.3] WJ: It will be real if you clap your hands and believe.
[0:01:43.1] DA: I heard that like the government is trying to DevOps.
[0:01:47.7] BG: I mean, Jez Humble was it a One F for a while, right? It was I mean that’s legit.
[0:01:52.0] DA: Oh, the man who wrote the book on DevOps.
[0:01:54.9] BG: Well, one of the people who wrote the book of DevOps.
[0:01:56.2] MN: Well, yeah.
[0:01:56.4] BG: It was a code reader.
[0:01:59.9] WJ: One F or 18F.
[0:02:02.0] BG: Oh, sorry, 18F you’re – oh, you’re right. I mean, let me take a step back here a little bit I guess because I want to emphasize, I don’t think no one owns the term DevOps, right? Like, no one in this room came up with it. I would never – I don’t want to lay claim to its origin or invention or whatever. I have a particular frame that I bring on that conversation that I think resonates and just to sort of contextualize that a little bit. I went off on this a little bit on Twitter a couple of weeks ago and, you know, hearteningly that expression of meaning got a much bigger response than I think I anticipated and it’s been really gratifying to see folks try to take back that term and recontextualize it but I’m not trying to apply a new shade or meaning to it. I’m just trying to understand how the mindset of the people had when I think they were attempting to bring about that culture shift in technology, still has resonance and still has meaning for folks who are doing that work today.
[0:02:53.9] DA: Yeah, yeah, more like it doesn’t have to be a complete specialization where you DevOps guy is in the darkest room in the building all by himself, just with the bunch of blinking servers like he can be out with everyone else like learning code and being a productive member of the society.
[0:03:09.9] BG: Yeah, I think, that’s one of the - I think it’s a concrete example one of the sort of more troubling patterns that has come out of this move and just to take a step back I think the wonderful thing that’s come out of the notion of DevOps is that people are aggressively scripting and automating their path to production and taking that journey much more seriously but I think the way a lot of enterprises treated that journey is a bit like magic enterprise fairy dust higher in a particular kind of DevOps expert to sprinkle a little bit of bill automation and will make everyone better.
And, I think that probably worked out at a few organizations but I struggle with it because I think it tends to recur, it tends to repeat some of the same patterns that led us as an industry down to the path to need that, kind of, DevOps support to begin with and when I say DevOps what I mean is a cultural shift. I think I – if you look at the definition as a [inaudible + 00:04:03] wikipedia, it’s sort of the set of practices that teams follow to help bring their software to production safely and quickly and the way that we use in this rating developers practices is by building cultures that transcended the individual incentives of the different teams.
[0:04:19.7] WJ: What are other people’s definitions? So, we now have the Wikipedia official, according to Wikipedia definition, what does it mean to everybody in the room, DevOps?
[0:04:28.6] BG: To me it’s like, you got this guy or a gal, Bobby and who’s just turning out things in Docker and looking at a terminal all day and making sure that the website is up and running like, I feel like people who have been called DevOps engineers in the places that I worked are – I got this we’re going to deploy to production it’s going to be great and I’m going to handle the flow of everything but I have no idea the skillset or the tools that a DevOps engineer would actually use and manipulate to do that process. It’s just like, magic, wizardry and all sorts of like cool coding and that aspect it’s really, really like, like it’s a part of my mind that I’m not familiar with so I’m really interested in the entire conversation we would have.
[0:05:16.7] DA: Yeah, or like that person maybe like put in holding to being the guy with the fire hose who's like putting out all the fires in production like just keeping the lights on and just concentrating all the pain over there.
[0:05:25.3] WJ: So, to me while I don’t know Dave, what is your definition of DevOps?
[0:05:28.7] DA: I mean, I worked in a couple of different context but I've really enjoyed the people who have brought to the DevOps title or the DevOps philosophy like the idea that they want to be working alongside of their fellow developers and feel the same pain that they’re feeling and help them get over those challenges to release code faster or build code faster. Be it like local environment setup or in their deployment pipeline with during inter sort of [inaudible + 6:01.0] or even production deployments and support logging, whatever.
[0:06:06.6] WJ: So, to me DevOps is a philosophy. The philosophy is that you don’t have a segregated ops team and the segregated dev team that work in opposition to each other where developers are incentivized to shift features and ops are incentivized to keep environments stable because those two things are in opposition and it creates tension. And so, instead what you do is you unite the two teams and you use dev tactics to improve ops and ops takes on ownership and responsibility of writing code the same way that developers take on ownership and responsibility of handling deployments even though they are allowed to continue to be specialists.
[0:06:44.7] DA: That is the definition that I share, that resonates with my journey and my understanding with term.
[0:06:48.0] WJ: It seems to me though that the industry like the people who bought in to that philosophy, ended up creating a bunch of really powerful tools by applying development strategies to operations problems and now those people are associated with these new tools like, Chef or Puppet or Kubernetes and so, when people want to hire for that kind of skill set they use that term as a job title. And so, what you found is, what we’ve seen is that nowadays when people talk about DevOps they are talking about role and that role is the person who has that magic fairy dust for you. Not the person who is going to unite your ops and your dev teams.
[00:07:27] DA: And, let’s be clear I love Kubernetes and Chef has been my friend and occasional secret lover over the years.
[0:07:33.0] MN: Your secret is safe.
[0:07:33.9] BG: But, I don’t think of myself as someone with predominantly an ops background and I think I would say, come from a predominantly dev background but I’m a dev who has had to go through the journey of trying to understand the deeper level, “What is my softer doing in production like why is this have meaning?” I think it’s important to regard that as a skillset that as part of your journey towards more senior development responsibilities you can and should own. You can understand how your software is running you might never have the same depth of all the context as the operations expert that you’re hopefully sitting next too. But, that doesn’t mean that you can’t apply a quality mindset to the operationalization of software and production.
[0:08:14.3] MN: Right, it doesn’t have to be like turning off and turning it back all again every single time like you can make things better.
[0:08:19.8] BG: I think that one of the wonderful things that’s happened to DevOps is that it’s provided a rich career path for people to follow who are interested in that intersection.
[0:08:30.6] WJ: It’s now the most highly compensated area of software development according to the most recent spec overflow jobs reviews.
[0:08:39.0] MN: Yeah, sure, man.
[0:08:41.0] DA: Well, I take it all back then I’d I’m a DevOps engineer. I love this stuff this is great.
[0:08:51.0] MN: So, as a software engineer learning things like, Kubernetes and Chef are extra skill sets that any software developer should learn or should have a grasp on to help with the operative side of the company, since the industry has made this assumption that we need to hire DevOps people on people who just know these things. How does a developer become a developer with ops on the side or do they just go full on its DevOps or the true meaning of that like, how do you separate yourself that I’m, oh, I’m an engineer and I happen to know Kubernetes.
[0:09:34.0] DA: Yeah, I think the challenge there is different like having buy from an organization be like, hey, we are going to let you guys like have that time to cross pollenate and like, you know, collaborate in a good way to build each other up.
[0:09:46.7] BG: I think that’s really key, I want to emphasize that point that it is the cultural shift that requires organizational buy in.
[0:09:46.7] BG: Yeah.
[0:09:54.0] DA: How would you say you characterize your primary well, when you think about like, I am an ex-developer, do you have a noun that you -
[0:10:01.1] MN: Oh, I’m going to use another one that gets, full-stack developer, well so there you go.
[00:10:06] BG: That's perfect.
[0:10:06.0] MN: But, it’s like front-end and back-end I would like at the moment I’ve been that clients that are predominantly asking for like React front-end experience so that’s like the one that I’ve been, you know, I’m a front-end engineer, front-end developer, kind of, thing.
[0:10:19.4] BG: What I would say to you is that if you characterize yourself as full-stack developer and I share that definition I’m on the same camp then understanding the operationalization, the build and deploy structure of your software is part of, is part of that journey.
[00:10:32] DA: Part of stack.
[00:10:32] BG: Yeah, and you know, I’m old enough, you know, just barely to remember when a CI server was something that like we spun up on a Dell Box that got shelve under someone’s desk somewhere so you couldn’t requisition hardware for this stuff.
[0:10:43.0] MN: Yeah, right next to rubber chicken.
[0:10:46.5] BG: Yeah, and there’s a wonderful James Shore blog from back in 2006 where he talks about doing Continuous Integration with the rubber chicken and a bell let it, hopefully we can tack this on the show notes later.
[0:10:57.0] MN: We were just talking about James Shore like very recently about the Batman.
[00:11:02] DA: Oh, cool.
[0:11:02.1] WJ: The Batman role, right? The article about James Shore, are that James Shore wrote about the deployment process, sorry.
[0:11:11.3] BG: Yeah, that goes in to the definition of continues integration which I also have strong opinions about things about that is maybe some other [inaudible + 00:11:17] topic. Well, KTLDR and that’s the continues integration is not building your future branches on observer somewhere. Like, continues integration is the act of continuously integrating and production and see the extension that continuous delivery is continuous integration of changes in to production, on preferably a daily or better than daily basis and DevOps is the set of practices that bring you to a shared understanding of what safe good deploys look like in production. It’s the guide that brings you to a place where CD is a safe and natural process where software is being deployed all the time.
Where you feel good about change set and where you’re working collaboratively with the team because everyone is incentivized similarly it’s not – I think that there’s a very old, I shouldn’t say, very old. But, there’s a particular school of thought and software that I characterize that sort of the separation of powers, school of thought, there legislative and judicial and executive and they all exist in tangent, you know, we and that involves spinning up centralized teams who have separate sets of metrics and incentives.
But, when you centralize things that have team ramifications that have implications for how your software flows to production than those incentives can come a conflict and some people would argue that there’s virtue in that conflict but I guess I tend to believe that if you look at the past production if you look at the time for like when we start working in the future doing well I think that you’re finished. Like, that’s the metric that you’re looking to optimize for and that the way to get there safely and joyfully is to bring everyone together like everyone on the team get them on the same page.
[00:12:46] DA: Everyone, has stake in that process like you can’t be just a bystander. I really like the idea like you’re talking about before about like, continues integration and continuous delivery like driving the development of the DevOps practices there’s like adage like, if something hurt you should do it more often so like, if deployment is hurt and they suck then you just keep on doing them and eventually you will arrive at the same conclusion that Jez Humble on many other people have come up before you’ll end up writing the book on DevOps.
[0:13:16.7] BG: I don’t think Jez was the first person to arrive there but I think the he did a good job in collaboration with other people in the industry just to articulate what that meant to try to put it a term on that had resonance and then bring that to people away that made sense. But it is so often that in the tech industry all we take out of that is the concrete process we say, “Well,” we look at the end of the journey and we say, “Well, there’s a book so if we just do it in the book then we’ll have it.” But, it ignores the mindset that brought people to that place in the first place. I shouldn’t say ignores it but it minimizes it or tends to sweep it under the rug.
[0:13:50.2] BG: I guess it’s too complicated like I want a package solution, I want like, agile like give me a scrum master.
[0:13:59.6] DA: Give me an agile, I would like one agile please.
[0:14:02.8] MN: One agile, one unit of DevOps and then, you know, profit but like -
[00:14:07] BG: There’s a great trigger about DevOps right like saying you want to hire DevOps engineers like saying you want a hire collaboration engineer to do all your collaboration in process.
[0:14:18.2] MN: Just don't want to deal with that myself.
[0:14:19.1] BG: Right, if the culture you want to build is one of growth and conjunction if you’re trying to bring people together then articulate placing the burden for that journey in on one person or one team it’s really hard it’s unfair to that person to expect them to bring organizational change when the organization maybe isn’t thinking about build and deploy holistically.
[0:14:37.2] MN: Right, and is like a journey also because like you don’t end up at the final chapter like you have stages that you go through and there are times that you are doing manual deploys and there are times that you’re, you know, having a more robust pipeline but you have to take steps to get there and you have to prioritize what’s important for you at that moment.
[0:14:56.2] WJ: Yeah, I will say though as much as we would like to not have to advertise for DevOps engineer because it is like advertising for collaboration engineer, doing all you're collaborating. The reality is that's what people are advertising and that's what people are expecting to see advertised and when you see a job posting that says Ops engineer it attracts a different a group.
[00:15:22] BG: It bums me out there, right? Because I think that it minimizes the fact that scripting and development has always been part of the ops, tool set and the ops approach to things and ditto for SREs that is sort of folks working on that side but there's a history of bash scripting that goes back to the dawn of - I should say, bash scripting, shell scripting, right? It goes back to the dawn of Unix, right?
[00:15:39] MN: Right.
[00:15:39] BG: I mean even that is a reductive way of thinking it, right? Like, scripting has always been part of the journey and the path to production and what DevOps brought us was a set of tools at the conjunction of dev and ops that nudged in the direction of a certain sort of practices that were maybe conceptually like of greater interest first, certain set of problems. I think it's unfair to say that people coming from an ops background don't have that skill set or don't think that way.
[00:16:04] WJ: But I think if you advertise for ops rather than for DevOps you're much more likely to get people who are not familiar with modern deployment tooling.
[00:16:13] BG: And I'm okay with that as long as they want to go on a journey. If I can bring someone with that, if you're looking f for a specialist making some of that, that have knowledge to my organization and pair them with someone who doesn't have that and bring them together that's the culture.
[00:16:25] WJ: What if you're in a team that doesn't have anybody with any experience with that tool set who's never touched Puppet, who's never touch Chef. He has never touched any sort of automation tools then your ramp up time is going to be a lot slower, right? I mean sure, you could pair that apps engineer with a developer engineer and they could reinvent a lot of these concepts or teach themselves but it seems that what most hiring managers want to do is go out find somebody whose got the skillset already and let them teach the rest of the team.
[00:17:00] DA: Right. I think a challenge also with like growing in to that position is that like with anything that you're learning you're going to make mistakes and like with this really robust tool sets that are available like you can make some really crazy mistakes and then that's your life line and like you can't - you paint yourself in to a corner then feel challenged to get out of there. Well you got to - I guess you can learn from failure and fear constantly making these mistakes and hopefully you learn from that and don't do it again.
[00:17:31] BG: Yeah, if you tear everything down and ready again then it's going to be better.
[00:17:34] DA: Right.
[00:17:35] BG: I think that's when you're - a lot of people say they want to build a learning organization right that, that's the growth they want to bring their team on and that's part of the conversation, if you want to build a learning organization you have to be ready to commit emotionally to the idea that you're going to make this, mistakes and that you're going to give your folks the space and safety to explore that problem space and pick up the tools that they need to in order to become the developer that you need for the problem that you're trying to solve. And that's just as true if you're trying to pick up a tool that fall in the DevOps category as if you are pairing on a programming language or a framework for the first time.
You know no one starts these jobs with a 100% knowledge that they need in order to become effective for the whole set of problems that you're going to solve. You have to pick up new stuff along the way and so if your organizational stands is learning first then I think it accelerates that journey rapidly and it get you in to that mindset where you're being creative about those problems sooner than later.
[00:18:28] WJ: I'm just trying to get you to cave and use the word DevOps has been bastardized.
[00:18:35] BG: I think there's a really substantive piece of push back I got on these which is that it is important to understand and contextualize the why, right? Like, if there's a reason why the meaning of the term has evolved and this is also the way a lot terms grow and evolve that a set of early adopters of the learning mindset hit upon on something that they think is a compelling way of thinking and then that, if it's successful becomes absorbed and adopted and then ultimately commercialized in to a specific category by the wheels of the industry and that's okay that's part of growth and change but I think the best thing about that is that if you always stay in that learning mindset and if you refuse to buy in to the - to the somewhat hardened definition then you're always your space to be inventing new things and tackling problems creatively.
[00:19:24] WJ: But if you let people go crazy then the definition of literal becomes figurative.
[00:19:30] BG: Oh my goodness there is a great - oh I can't remember which dictionary it was but they post an article about how much flack they get about that but they are able to point back to examples in 19th century literature in which people were already twisting the definition of literally so this is not a new - this is not a 20th century corruption.
[00:19:46] WJ: And I think you can argue that the term agile has under gone a lot of transformations that we're seeing with -
[00:19:53] MN: Well, it's a marketing term, it's a buzz word that strikes a nerve and like there are certain ideas at its core that resonate to people and that's why it works. I mean, you see this like big data and like deep learning as well. I've been learning more of deep learning and like what is deep learning different from a neural network? it's just like one extra layer you just have three layers then it's deep and what is that? It's ridiculous but it's the marketing and in some ways it's good, in some ways it's misleading.
[00:20:26] DA: Agile.
[00:20:29] MN: Agile, Scrum.
[00:20:32] BG: I think if you want something to cone and define agile here you're going to be - I hav been doing what I would characterize as agile software development for somewhat north of a decade now but if you need somebody to define that, you're going to need someone smarter and wiser than I am.
[00:20:45] MN: Yes it has been used and so many different organizations. They have different concepts and I have air quotes you may not see me but agile and what that means and every company has a different agile definition of that and that's also quite insane.
[00:21:01] WJ: So maybe you'll end up being the person who is old enough and wise enough to give a full definition of DevOps after that term becomes as thoroughly capitalized upon as agile.
[00:21:12] BG: I think where we landed on this and how I sort of ended my commentary on it. I try not to use the word rant but it's Twitter -
[00:21:21] WJ: Yeah it's a rant.
[00:21:21] BG: It was definitely a rant. The philosophical origins of the advances on software development I think I hope that we've made over the last 20 years all revolve around mutual respect type feedback loops so without getting in to the terms agile and DevOps specifically I think the way I try to characterize this, I hope it's a more - it's a frame that I think it gets to the root of what I'm trying to get at but it's also sort of a useful form of indirection because it doesn't directly attempt to tackle those terms as I sort of the way I think about the advances we got from high software development, they revolve around the idea that incremental change is important that we can change incrementally and that we should change incrementally; that we need to establish tight feedback loops that we understand whether or not what we're doing is working or not that is better to share knowledge than to hoard it, that to pull ourselves out of ourselves and by extension by out of our comfort zones a little bit with the belief that in sharing that knowledge we gain more from the sharing than the separation and mutual respect, right?
You don't go anywhere even if you have all of those things you can't establish the goals of respect between people then you're never going to build that sense of safety and trust in your team and I think that lies at the heart of my understanding of what agile is. I think if you take out any one of those things you don't get to quite the same place and DevOps is an extension of that mindset in to the operational space in the way that makes all of software development safer and more effective and hopefully more fun.
[00:22:55] MN: That resonates with me like that kind of comes to like any good team that you've been on like you need to be open to be able to admit what you don't want to know and admit that you want to learn more and you need that kind of understanding in order to grow together.
[00:23:10] BG: I guess I would also say that I totally agree but the changing of the definition makes me sad a little bit because I want to believe that I'm part of that journey, that I'm part of a culture that's trying to build those bridges and so just sort of be told “Well, you know you're not a DevOps engineer and so don't touch the continuous integration build.” I maintained a major continuous integration for years and I've been told that and it's hard to hear like I don't want to feel as though that, that knowledge is shunted away because I engage in a particular task in a team that is not directly operational. I think we should all be part of the journey. I think it's on people from both sides of that isle to grow in to the skill set to build software better.
[00:23:56] DA: You hear that Mike, you gotta get on that. I got a goal to the DevOps channel I'm like, hey I'm one of you now.
[00:24:03] DA& MN: Yeah, that's it. Absolutely.
[00:24:04] MN: I'm learning how to do this.
[00:24:06] DA: Where were you at the Docker workshop man?
[00:24:09] BG: No excuse.
[00:24:11] MN: I was driving at the side wall towards Philadelphia. That's a whole -
[00:24:16] DA: I need to unpack that. Let’s deep dive on that.
[00:24:20] MN: Awesome. Right thank you so much for coming on down and sharing knowledge on DevOps. Bryan how can people find you on the internet?
[00:24:27] BG: How can they find me? It is possible to geo locate people via their cellphones signals?
[00:24:34] MN: We're almost there, almost I think. There are microchips in the water.
[00:24:38] WJ: They work in the NSA, they can.
[00:24:39] BG: Just tune in to my Alexa.
[00:24:41] MN: Great, and buy the things on your wish list.
[00:24:45] BG: This has been so much fun, I'm really thrilled to be on the show and to hang out with you all. I am @bguthrie on Twitter. I don't know if I dropped any knowledge. I'm with the rest of you with a set of frustrations and joys and things that I'm trying to articulate for myself along the way right? What does it mean to be adopting this term and how do I bring that to the work that I do and I hope that I don't know, some of that journey was interesting.
[00:25:06] WJ: Humility is a core value.
[00:25:08] MN: Yeah, awesome. Again, Bryan thank for coming on down, I really appreciate having you here we had a great time. Our co-host Dave always a pleasure.
[00:25:17] DA: Thanks man.
[00:25:18] MN: And our producer best times, when you're recording, always a good time.
[00:25:21] WJ: Absolutely.
[00:25:23] MN: And I'm Michael Nunez, feel free to reach us at twitter.com/radiofreerabbit and this is the first episode you're listening. Please give us a five star in iTunes and subscribe. This is the Rabbit Hole we'll see you next time.
Links and Resources:
