Welcome back to another episode of The Rabbit Hole! On today’s show we welcome our friend Dane O’Connor, who is here to tell us why we need to stop using the term ‘tech debt’. As a commonly used phrase in today’s developer community and in so much business, there can be a lot of sensitivity and potential issues that are connected to the term. Dane believes there is a fundamental problem with the way it is being used in company and client communication and this dynamic needs a complete overhaul in order to remove these problems. In our discussion, we introduce the issue and how it manifests in different settings before looking at the false choices the conversation presents and how ‘tech debt’ as it is understood currently could be relegated from its important position in professional settings. Dane goes on to detail the way in which these discussions should be held and how decisions around code and the investment in it should always be progressive and servicing everyone. For this insightful chat be sure to tune in!
Key Points From This Episode:
Transcript for Episode 76. Stop talking about Tech Debt with Dane O'Connor
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast, in fantabulous Chelsey, Manhattan. I’m your host, Michael Nunez. Our co-host today -
[0:00:09.8] DA: Dave Anderson.
[0:00:10.8] MN: Today, we have to stop talking about tech debt, once and for all.
[0:00:15.6] DA: That’s it?
[0:00:17.0] MN: Yeah, this should be the last 15, 20 minutes we spend talking about tech debt.
[0:00:19.6] DO: Please, please.
[0:00:20.6] DA: Oh wait, someone just burst into the room.
[0:00:27.9] DO: I have such a reaction to this, I had to just jump right in.
[0:00:29.7] MN: There you go.
[0:00:31.1] DA: Where’d you come from, Dane?
[0:00:33.1] MN: Dane, wait, yeah, you just appeared, we talk about tech debt, you’re mad. Before we begin, we have a guest today, Dane O’Conner. How’s it going, Dane?
[0:00:43.1] DO: Hey, thanks for having me guys?
[0:00:43.9] MN: You sound really upset.
[0:00:46.0] DO: Yeah. So I’m a consultant, I recently relocated to New York. I’ve been doing full stack development and dev ops engineering for about 15 years. I’ve also managed teams as large as 30 and I’ve worked for a large variety of companies form venture backed startups to Fortune 50 companies.
I’ve seen a lot of different organizations and their approach to this problem and I am thoroughly convinced at this point that it is causing more harm than benefits.
I think we absolutely need to eradicate this from our vocabulary. I hope to convince some people, by the end of this to never use the term again, be triggered like I am.
[0:01:29.1] DA: You had some conversations. I mean, we’ve had some conversations too, we have two episodes actually, we talked about this so we are continuing to talk about tech debt.
[0:01:39.2] MN: We need to stop.
[0:01:40.7] DA: Number 67 on tech debt and trust with Madelyn and Sam which was –
[0:01:46.8] DO: Yeah, this is interesting. I’ve actually talked to Madelyn and Sam. I love those guys. I always say that you know, it’s interesting talking point in that podcast, talking about trust because I think tech debt, the way we currently use it today, fundamentally breeds mistrust on teams.
People often have like – they’ve been taught or the theory is that it actually helps create trust and not just think the reality is today, it’s the opposite.
[0:02:14.6] MN: Right, because I mean, how many – how often have we had built out the feature and then go to our project manager and said, “Hey, we have to build this thing but it’s going to accumulate some tech debt.”
That’s probably all that project manager hears all the time that there’s tech debt everywhere and we’re doing it on purpose like us.
[0:02:33.5] DO: Right, yeah. In my experience, what ends up happening is tech debt is used as a place holder, a convenient place holder to make excuses. On many sides of the table, you know? In an organization, you have a lot of different departments trying to deliver products to customers and tech debt can become this thing that people talk about that is the reason for all the problems.
Like tech debt is the reason why it’s slow to deliver something or it’s the old developers who created this tech debt like it’s a way to abdicate responsibility and it’s weird that we promote it to the first class citizen like we do because the conversation revolving around tech debt is so one sided, the engineers understand what we mean when we say tech debt, but the term is so loaded now that none of the other people, like the people who are least qualified to have a conversation about tech debt are now all of a sudden involved in the conversation and feel that like they’re actually making the call, that’s another weird thing too, right?
[0:03:41.6] DA: What is like the worst case scenario look like.
[0:03:45.5] DO: Yeah, man. Worst case scenario is the engineering team is trying to deliver feature, this is like, I feel like almost every organization, where you want to hit a deadline, you’re taking on explicit tech debt because you think if you write it down somewhere, you’re going to get the opportunity to invest in it in the future, you write down tech debt, you talk about it like a first class citizen, you plead with the product owners or the different stake holders in the organization to actually tackle it at some point and then you never do.
That alone can, we’ve created this thing that can ultimately destroy morale because it’s like removing the agency over the technical product from engineering teams.
[0:04:31.5] DA: Yeah, it’s like cognitive load.
[0:04:33.7] DO: Yeah.
[0:04:35.7] DA: Just building and all the time.
[0:04:39.9] MN: Always building, never refactor, always building.
[0:04:43.5] DA: Yeah, although, I did work at a place where we actually - we came in and we are working with them and we looked at the tech debt board and we talked with them and more like, are we going to do any of these things? No.
[0:04:58.1] MN: No.
[0:04:59.6] DO: Maybe that’s even worse. Where everyone knows that it’s not going to be done, right? You’re writing it down and there’s not even the perception that you’re going to get to it.
[0:05:11.0] MN: I think like a lot of the time, I find that if I’m working on say, a feature that is accumulating some debt and potentially to hit this deadline and let’s say we have a V2 for example. If I have to go back and touch that code, that has the tech debt that I wanted to tackle, then my estimations will be included in me taking on that debt as well.
[0:05:35.0] DO: Yeah.
[0:05:35.2] MN: I feel really dirty, knowing that yeah, this could be done in X amount of time but I want to do this one thing, so I have to say why and come up with good reasons for that and play that card.
[0:05:46.6] DO: Well, and why is that? In my experience, the reason for that is that there is an implied assumption on how long something might take.
[0:05:57.2] MN: Right.
[0:05:57.2] DO: Which is devoid of the context of the current problem, at hand. This is what we would talk as accumulated tech debt. Now something’s going to take longer because we took on some more complexity.
[0:06:09.9] MN: Right.
[0:06:10.5] DO: I think that that is a really bad way to reason about it. It’s fundamentally past oriented perspective. You know, even the term debt, right? It’s like a decision was made in the past and we need to pay it off.
[0:06:26.6] DA: Right, there’s interest.
[0:06:27.3] DO: Right, there’s interest, right? That’s the common framing and like, I think a lot of us agree too that some cost is not a good way to reason about what you should do next.
Creating this list of things that you may want to do because you know it makes it more complicated, it’s going to take longer to do things, it feels painful to reason about it that way because that’s not what anyone really cares about, no one cares about making things a little cleaner for developers, they care about outcomes. The developers also should be oriented around that as well and I think what you were getting at Bobby, with the whole, the sense of feeling dirty.
Like patting your estimates to tackle the tech debt and I feel like that it’s clear that tech debt is a problem because you feel dirty doing that when you know, you have the technical knowledge, you know that that’s the right thing to do and you can make the judgment about the timing.
The estimate really – it’s not patting it, that is the estimate.
[0:07:30.0] DA: Right.
[0:07:30.7] DO: I would love for the industry to be treating that as the real conversation and not creating some artificial line between delivering a feature and delivering the feature well.
[0:07:40.9] MN: Right, I think the like, even when that – the problem that exists right now is if I make a certain estimate and like the project manager knows that I’m doing it because of tech debt, then like, every single estimate that I make could be done because I want to tackle the tech debt. So then there’s like no trust there either.
Wait, is that estimate real work or is that because you want to go do some tech debt?
[0:08:06.6] DO: Yeah.
[0:08:06.3] MN: That whole thing and it’s just like, mistrust happening in that regard and then it gets really awkward. The estimate.
[0:08:12.4] DA: It’s like something, it’s like well, tech debt is something that developers want to work on or like –
[0:08:17.9] DO: It’s a false choice, right?
[0:08:20.4] DA: It’s like tech debt is, okay, tech debt is a time that we have set aside that we can work on just these technical things.
Sometimes the – I’ve had sprints where we’ve worked on tech debt and I feel very empty at the end of it. I don’t feel very satisfied. Maybe some things like you know, reorganizing the folder structure as the project rose and then okay, now what do I do with these folders, right? Fill them up.
[0:08:53.5] DO: I’ve definitely been on that side of the table and I will say, that it’s because of the framing, it creates this false choice where it’s like, you either work on forward momentum for the thing that the valuable thing that you're creating or you work on tech debt.
It’s like, do we invest in one or the other when the reality is, you’re always investing in tech debt and you can also make the argument that every line of code that’s written is tech debt.
[0:09:22.9] DA: Yeah.
[0:09:23.8] DO: Right?
[0:09:24.1] DA: I love that argument. The best line of code to maintain is the one that you never wrote.
[0:09:29.0] DO: Exactly, yeah. It’s like, this is just your product that you’re working on. The thing that’s weird is that like, I feel like tech debt implies that there was a perfect implementation. You saw it and you chose not to do it, because why? Because you were lazy?
The reality is, you saw an alternative implementation but it would take longer and that’s what the business needed. The immediate is shorter - and that’s like, that’s okay. Then, when you go back and say, this is tech debt, it’s like no, that’s a sunk cost, you made the smart choices, you’re not blaming the previous engineering team, you’re not blaming your previous self.
Like that’s another thing that sucks with tech dent. You have to go to the product owner and you have to be like, yeah, this thing that I made has like all this problems that’s going to make it terrible for us to work on this going forward unless you give me like free reign to do whatever I want.
[0:10:27.2] DA: Right.
[0:10:27.4] DO: What a terrible position.
[0:10:28.9] MN: it’s like wait, “You did the mistake too? How do I trust that you are not going to do that again?”
[0:10:33.3] DO: Right, as supposed to like everyone –
[0:10:35.0] DA: he probably won’t also.
[0:10:36.6] DO: Yeah, exactly. Yeah. It just seems like everyone could be on the same side of the table and like even promoting this to be a first class term in our language creates a barrier and a reason to mistrust each other.
[0:10:53.3] DA: Is there a light at the end of the tunnel? It sounds like there is no trust.
[0:11:00.1] DO: Yeah, there are things that people may, I do not want people to think that I’m arguing for because in my conversations with people, they immediately suggest, “What should we call it?” It’s like, no. The problem is making this term a first class citizen, there is a path forward and that is to change the frame under which we have these type of conversations.
I think both engineering teams, design teams, product teams, need to be all focused on the outcome and be okay with the fact that previous decisions may make it difficult to deliver new features in a particular area of the code base. With the understanding that we made those decisions smartly.
It was not a mistake, it was using the best information that we had at the time to make this call. Now, going forward, when a product owner comes to you and they say, “Hey, we really could use this feature, it would help drive some metric,” right? You just tell them, honestly this part of the code base is complex and I think it’s going to take longer to deliver the feature than you think.
When you lay out all of those things, the team can agree as a unit on which features are the right thing to invest in as opposed to the product owner getting in their head. That, I can just keep on accumulating tech debt, I could accumulate a little bit more and maybe I’ll get this out the door.
[0:12:28.6] DA: I guess also, like America’s not like known to be good with physical response.
[0:12:36.4] DO: Those get a little bit more, we can just print a little more money and we’re fine.
[0:12:40.5] DA: Yeah. Just bump that insurance rate down a little bit you know.
[0:12:46.0] MN: I mean, I think that – yeah, too big to fail. I think that a lot of the time though, I mean, you, Dane you brought up a good point earlier about looking at a particular choice, seeing an alternative that’s like faster and then which ends up being tech debt in the code base.
I think that part of the issue was - can be like pressure from the organization and that you need to keep your word with the estimation that you said you were going to - that the team said they were going to do.
If you see something that’s going to take two days, you would want to finish it in two days but if you realized that after you know, getting all the context of the particular feature in and seeing that it’s going to take more than two days, if you did it this right way. “But I have this tool in my toolbox. That’s going to be future Mike’s problem, not present Mike’s problem.”
And then you do that and you keep building it. But I think you brought up a good point of just like letting the product in the project manager know like hey, we got to touch some piece of the code base, it would be pretty hard. But we can do it.
[0:13:59.3] DA: I like looping product in on like the tradeoffs that I’m making because there’s always tradeoffs, there’s always many paths forward and like the story I was working on, very recently, we’re like looking at like grouping different sets of items together based upon properties.
I could do this by ID or I can do this by the string description of the ID and it was like okay, the ID is more correct but I feel like the string one’s going to be faster and maybe it would actually be better for the business too in some ways, in a weird way.
[0:14:34.7] DO: Right.
[0:14:35.4] DA: It’s like, okay, this is kind of tech debt, I could label it as tech debt but that doesn’t feel right because it’s just like, this is a decision I’m making right now.
[0:14:42.1] DO: You’re making like a good call. Why are we calling that debt? Why are we calling that a mistake?
[0:14:49.1] DA: I save present me like a day’s worth of effort and you know, got that thing done.
[0:14:54.8] DO: Right, that’s like another thing related to outcomes, right. So let’s take the scenario where we do get that magical time to invest just in tech debt. First off, I’ve got to say that already bothers me because - and I may have, I think I said this earlier but I don’t like the idea that we would delegate a decision about when to invest in tech debt to someone who is not really qualified to make it.
It’s along the same lines of what you were just saying Dave, like looping product in is great but having product make a call on that is really like in your court. And it is great that you made the good call.
But let’s take the scenario where we get a full week or so to invest in tech debt. The real question is why are we investing in the tech debt if we are not expressing the reason for investing in tech debt, in terms that the rest of the organization can understand, it is kind of like we are going off doing our own thing.
It’s the same exact thing as product asking for a feature that doesn’t seem to make any sense. No one wants to work on it, we don’t think customers are going to use it. It’s the same deal and so this is part of the framing that I think needs to change.
Everybody needs to be focused on the outcome and if you can’t express why working on this tech debt is going to move the needle on something that we care about, in relatively near term or long term, but in some way that a different part of the organization will understand maybe it doesn’t make sense to do.
[0:16:25.8] DA: Yeah, that’s interesting for me because when you start thinking about that it is really about waste and minimizing that and being customer focused and you can think about ways like you are saying, product can be not customer where always be like, “Oh this is going to be the best possible thing and it doesn’t matter what the customer thinks.”
Which work sometimes for Apple or design like, “This is going to be the best possible design. It’s going to have the best toggles and animations,” and whatever.
[0:17:00.8] DO: But what basis?
[0:17:02.3] DA: Yeah, right.
[0:17:02.6] DO: And do other people believe in it or is that your pet project and I think this is not about trying to shift the responsibility to someone else. I actually think the reality is that tech debt is a way for a lot of engineering teams to relax some of their responsibility. I think they really need to take on, if I have someone coming to my house to build a load bearing wall, do I want the person coming up to me and saying, “Should I use this type of wood or this type of wood or this type of wood?” I want the wall to work.
[0:17:35.8] MN: I want the wall to do wall things.
[0:17:38.5] DO: Right, don’t tell me you’re going to come back next year and you know there is a little bit of allowing this conversation to happen is also kicking the can on the responsibility, talking about facing the reality that the estimates could have been wrong and just owning up to that and you know, there is an element there.
[0:17:58.9] MN: Going back to your example Dane on the house, you want to build the walls of your house. You want to build the walls of your house with materials that are going to do wall things, right?
Keep people out of your house, hang things up on the wall, that kind of stuff. And if you choose the crappier wall materials then you are not going to get good things, right? Your wall is going to fall apart, essentially.
I have a question though, a lot of places and this will probably right true to a lot of listeners, they are working at a particular code base that requires some form of framework update or upgrade, right? I’ll use – you want to update, this is not out right now at the moment but say, Facebook react now they figured out a way to make function component or stateless component to actually render faster, right? Because Facebook is saying, “Hey everyone, use stateless components whenever necessary because in the future we’re going to render things faster by doing that,” but right now it’s not happening.
It’s a dream, right, what do I tell - and then I have all these components that I haven’t been using stateless components for, how do I sell that to the organization like, “Hey we need to go back and make all of these components stateless or functional.” What do we do? How would that work?
[0:19:21.9] DO: Well all right, a couple of things. First off that may, I might not even agree that that would be a good sell. So let’s start saying that maybe this is a thing we should do. I think an important onus on the engineering group or even just yourself is to present stakeholders with choices. So rather than it being simply, “I see the ideal way and there’s a way to get it done faster if we cut corners.”
Instead presenting, “This is a way and this is what I could deliver to you quickly. This is what I could deliver to you with a more heavy investment and these are some of the maybe future benefits that come with it.”
So when you are talking about a faster render time, you know I would in that case, if this is going to be a significant investment, so it actually merits doing some of this analysis, you’d want to look at how that moves the needle. How does the app, is performance a key thing that anyone cares about right now?
[0:20:20.2] MN: Right.
[0:20:20.9] DA: Yeah, other particular pages that are very slow.
[0:20:24.6] DO: Right, is this an initiative and if it is then you can kind of express this investment in those terms. If it is not then maybe it is actually just – it would be something you’d like to do but it doesn’t really makes sense for the business and I think developers need to be more aware of that.
[0:20:40.3] MN: I think the example I gave, could be flipped in a way that we can present it to the business and say, “Hey if we spend X amount of time making these functional components, we will increase or decrease the render speeds by 40%,” right?
“Oh wow our page loads 40% faster? That would be great!” Here’s one, suppose I am working on a codebase that’s in Java 6 or in Java 8, it’s just harder. I haven’t done any Java in a long time listener but I suppose like Java 8 –
[0:21:10.8] DA: I think they’re on 11 now.
[0:21:11.7] MN: Oh yeah, wow. Okay, maybe we’re not. We’re at 6 just to give you an idea, we’re at Java 6 and we need to upgrade because the application has been serving customers for many years now.
Would you also have to sell that with, “Oh there has to be some key performances?” We have to speak the language of the product manager, is that what you’re saying like all of the business?
[0:21:32.3] DO: Yeah and I think what you are getting at is fundamentally these are primarily technical problems, right? They don’t necessarily have a direct customer interface.
You know, how do you have a conversation about those things? And I think the reality is that you can even just taking your example right? There are things that would be valuable to the business by upgrading to a more modern version of Java not only having an easier time onboarding new engineers if the company is growing.
But maybe the company is concerned about security vulnerabilities like I think looking around for the touch point that actually matters to the business and getting everyone aligned is very valuable. If you can’t find the touch point, it is probably not valuable but I think and tying it all back in to tech debt, I feel like tech debt becomes a placeholder for that conversation that I think absolutely needs to be happening, you know?
We’ll say, “Oh this is tech debt. We have to invest in it at some point.” And the choice laid out for people who are making decisions about this is, do I invest in tech debt or do I move forward?
And it’s not an either or like investing in the tech debt should be moving forward but how do we express it in those terms and I think engineers need to be a lot more involved in those conversations and they are not today because we have convenient terms to throw everything in a bucket.
[0:22:53.1] MN: So throw away with the tech debt you’re saying, just have the conversation straight up. “This is what we need to do with, these are our options.”
[0:23:00.6] DO: Yeah, I think that is a really important strategy is sitting down yourself and figuring out, “How can I frame at least two options, out of these decision?” You know? “If I am doing the quick choice, let me frame the long choice completely.”
I think that is important. I think anything less, anything on a smaller scale that you think is not worth doing that for, you should just do.
That shouldn’t even be a conversation. That’s like the guy or girl building the house. If he has to go buy nails like go buy nails, you know? Don’t ask me for permission to go buy nails.
[0:23:39.7] DA: Yeah, right and I guess like also in the tradeoff of considering what’s the quick way or the right way, it is often like, oh I can do this implementation like in a really small chain set. I can do it in a one line fix but it’s going to be gross and it is going to make complexity increase at this one point.
Whereas I could like refactor and spread out that complexity and make future changes maybe a little bit easier but you may not know what those future changes are.
[0:24:10.5] DO: Right, so that type of thing - so that is an interesting point because that is something that is really difficult to promote to having a general conversation about. Other people can’t really weigh in on that at all, right?
[0:24:23.8] DA: It’s pretty personal.
[0:24:25.1] DO: That is your craft right there. I think the reality is that sometimes when there is a really high amounts of uncertainty and we have these convenient tools that we reach for, it is easy to try and delegate that decision that I think that happens a lot whereas the reality is sometimes you just got to make the call.
[0:24:43.4] MN: Got to make that call. Do it for the sake of the business.
I think one thing that Dane that has been mentioning is just try to frame whatever this tech debt is for the sake of the organization and see if either there was money or those customers and that kind of stuff or as you mentioned before like retention.
You know if you want to increase individuals to consistently stay in this codebase and you want to ensure that we’re still providing value to the customer, we also have to find reasons why these tech debt and I am doing the tech debt.
[0:25:25.8] DO: There’s the air scare quotes.
[0:25:26.6] MN: You can’t see it, oh I am doing the quotes but this tech debt has to get paid down but right then and there not like leave it for future Mike to deal with it because I always think of tech debt as is this decision going to cost the company something. I mean the product manager, project manager C Suite individuals, they talk money. That is what it comes down to, “Are we going to make more money doing this or less money doing this?”
And if this is difficult to understand or if we’re using this really obtuse framework, it is going to be hard to get engineers into the door to actually enjoy working here to actually crank out these features. I think we need to do that because we’ll lose the amount of time it takes to do something and money is important. Time is money.
[0:26:11.6] DO: See now, I am already loving hearing you talk about that because now we know why we’re doing it. A lot of times the tech debt it really does turn into this thought that it is just engineering having fun or making your life easier, right?
That kind of sucks and one thing that you touched on too when you are talking about like for a lot of organizations I guess it really should be about money or they should be reasoning about how they deliver maximum value to their customers.
But I think for all the listeners out there, you are probably in organizations that are at various stages of development on that cycle and sometimes the organization itself doesn’t even reason about their problems as well. And so I think when you are in that type of environment, one of the things that will be interesting to try, this is an experiment, just try it. If you find yourself reasoning about tech debt, find a way to frame it in a way that is valuable to the stakeholders closest to you.
So you don’t have to go all the way full hawk to, “This is how we move the needle to the business.” That is the ideal but you know, have that conversation with – talking about building trust, have an honest conversation with a product owner about what they really care about right now, how you can deliver most value and then talk to them about the problems you are facing.
And to work with them to find the reason, the drive to do it. So it is not just like when we get permission we go do it.
[0:27:41.5] DA: Right, yeah. I like that too and it brings it back to tech debt and trust thing. So it’s like just trust or something.
[0:27:50.1] MN: Just trust.
[0:27:51.3] DA: We’re not saying that word again from this point forward in the podcast, it’s that word.
[0:27:55.9] DO: Stop destroying trust.
[0:27:56.7] DA: You know we are not going to say that word. It’s just –
[0:27:59.9] MN: You got to do this thing. It is going to be really complex because of the “hmm-hmm” there you go. You can fill in the words. We’ve got to stop it. We’ve got to stop saying the word all together, let’s do it together.
[0:28:12.1] DO: I need your help. Please, it’s going to be so great.
[0:28:15.5] DA: If you want to hear that word again, listen to Episode 67, hmm-hmm and trust or Episode 19, just plain hmm-hmm.
[0:28:24.5] MN: Just plain. Awesome and so we spent the last couple of minutes talking about the hmm-hmm and Dane, how can people contact you so they can share their experiences dealing with you know, stuff?
[0:28:40.0] DO: Oh man, yeah I mean I would absolutely love to have conversations about these sorts of things. Like I said, I have just recently relocated to New York. So the coffee is great but you can reach out to me on all the things LinkedIn, Twitter, GitHub, Stack Overflow. My handle is thedeeno.
[0:29:00.7] MN: All right, cool. Awesome.
[0:29:02.3] DA: Is that like rat pack?
[0:29:03.4] DO: It’s not, it’s a nickname from childhood.
[0:29:08.0] DA: Oh okay.
[0:29:08.3] DO: Yeah, it is not quite as cool.
[0:29:12.2] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five star review and help developers just like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast.
On behalf of our producer extraordinaire, William Jeffries, and my amazing co-host, Dave Anderson, and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
Links and Resources:
