Today on the show we will be talking about Git Workflows. It seems like everybody is always using Gitflow or Trunk Based Development. Gitflow defines a strict branching model designed around the project release. It assigns very specific roles to different branches and defines how and when they should interact.
Trunk Based Development is a source-control branching model, where developers collaborate on code in a single branch called ‘trunk’. Advances to source-control technologies have made Trunk Based Development more (and sometimes less) prevalent. However, it has been a branching model that many have stuck with through the years. In this episode we’ll be getting into more of what we prefer, whether it is Gitflow or Trunk Based Development and we’ll get to some of the pros and cons behind the two.
Key Points From This Episode:
Transcript for Episode 30.Trunk Based Development vs Gitflow
[0:00:01.5] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast. It’s nice for a change in weather in downtown Manhattan. I’m your host, Michael Nunez. I have my co-host here today:
[0:00:10.4] DA: Dave Anderson.
[0:00:11.7] MN: And our producer:
[0:00:12.8] WJ: William Jeffries.
[0:00:13.6] MN: And our today, we’ll be talking about Git Workflows.
[0:00:16.3] DA: Man, so many Git Workflows, but it seems like everybody is always using Gitflow or Trunk Based Development.
[0:00:24.9] MN: Today we’ll be getting into more of what we prefer, whether it is Gitflow or Trunk driven development and we’ll get to some of the pros and cons behind the two.
[0:00:36.7] WJ: Yeah, I think that Trunk Based Development is sort of the new kid on the block and I am kind of excited about it but I think it would be interesting to get some spirited debate going about pros and cons.
[0:00:47.9] DA: Yeah, although if you read trunkbaseddevelopment.com, which we’ll link in the show notes, you will find that Trunk Based Development is one of the oldest develop the methods for version control because it doesn’t use branching.
[0:01:01.1] WJ: Yeah.
[0:01:02.1] DA: And you can be lazy.
[0:01:03.5] WJ: Just commit to master, push force, no lease.
[0:01:067.6] MN: Yeah, no just place. Create, you could do what I do. I have an alias yellow, which is a GitPush-Force with a message and it’s like YOLO and it will whoop, push it straight up whatever, let it push to production. I find, I mean I’m personally a fan of Gitflow because of that very same reason as we discussed. At times you’re working and you’re working on a feature that you don’t realize when in some way, shape or form that will introduce a bug, and it’s sending in production and then things could go on fire and like what do you do and how do you mitigate that with Trunk driven development? It’s something that I have been curious about.
[0:01:50.0] DA: Yeah, so I guess there is a couple of different strategies with that like you could do over commit, revert the thing that you just did. You could fix forward if it’s something easy just go in there and fix it. YOLO force push.
[0:02:06.2] MN: Well force push for the win not for the loss right? You have to do it for the win.
[0:02:10.8] DA: Right, yeah but maybe we should define a little bit better what we mean by those different workflows because I guess you can – yeah, they are both using Git. It’s the same version control system that we all know and love and I think we all agreed now by 2017 that we all love Git.
[0:02:30.1] MN: Sorry people, we still use SPN. This conversation is about Git.
[0:02:36.9] WJ: So Nunez, how would you define Gitflow?
[0:02:41.2] MN: All right, so for Gitflow you have separate branches. We use three branches in this example. Master is currently what’s in production. It will reflect all of the features that are in production and that is the pipeline that will preserve all of the features that are out and online and live. You then have develop, which is a copy of master when it’s created and from develop, you then create other branches called feature branches.
So when you add a feature, Git pulls request by whatever system you’re particular organization follows and then that feature branch gets merged into develop. You do that many a times until you’re ready for release. From the developed branch you then create a release branch, make sure that everything runs well and whatever QA testing, automation testing that you have and when everything is set and done, you can then take that release branch and merge it on to master and develop and tag it so that you know that hey, we have this new thing out in production. We’ll call this 150 tagged, boom, done.
Now master and develop are the same because they now have the very same things in production and then from there, you know that master will always be clean because it doesn’t get merged or rebased often only when a big feature or a big release has happened.
[0:04:16.0] DA: So how do you deal with long note feature branches in Gitflow? If I have a feature that I am working on and it’s bigger than I expected and it’s just not ready to go out with the rest of the things that are in the develop branch, how do you work around that?
[0:04:32.4] MN: I’m thinking of two different particular situations. One is you can potentially have, I hate that I am saying this, but you can have a toggle that will allow the develop branch to exists with this feature but not fully implement it by having the toggle off at that particular point in time. Or I have worked on a long live feature branch where like every night you have to make sure you rebase or merge, develop into your feature branch to make sure that you are constantly updating.
So you’re not like in that painful merge back into develop and you get all the code updated in a clean way. But sometimes you want to make sure you can get those featured branches merged in as fast as possible, which is why I would probably wrap things in a toggle state and make sure that it is always off until you’re ready to release it and then you can toggle that particular feature on and then take care of it from there.
I can’t really think of a third option besides just trying to maintain that long live feature branch. That is one of the most painful things you have to deal with in Gitflow though. Long-lived feature branches are the worst thing possible.
[0:05:40.2] WJ: How do you deal with hot fixes?
[0:05:42.4] MN: Oh suppose you, in the example I mentioned before, let’s say the release had an issue that needs to be fixed before we merge into production, that release you cut a branch from the release, make the hot fix and merge it back into the release branch and then all of those changes will then be captured when they get merged onto master and develop but the hot fixes would happen when the release branch has been made.
So it’s like a lot of tracks and trees. I imagine if you Google Gitflow, you’ll see the first image in Google images you will see like the little, what looks like Skittles that will describe the very thing that I was explaining. We definitely should drop that in the show notes to kind of help people just click and go.
[0:06:30.2] DA: Yep, help them taste the rainbow.
[0:06:31.6] MN: Oh yeah and there’s a lot of dots, like I mentioned before. You’ve got master, you’ve got develop, you got release, you’ve got hot fix, there’s one more I can’t really remember at this particular point in time. But that’s the way you would handle a hot fix. A hot fix would kind of exist when you’re about to release something. So it should be really fascinating to just merge back in and then release that. Make that the release candidate and then merge that or rebase that on top of master and develop and then you’re good starting clean, continue your features from there.
[0:07:03.3] DA: What if I told you, you never had to rebase along the feature branch again?
[0:07:09.5] MN: Is that possible?
[0:07:14.6] DA: You know, that’s something that is a core tenant of doing trunk based development is avoiding those longer feature branches and avoiding that merge process.
[0:07:23.9] WJ: Yeah, it sounds like something that would be really untenable with a large development team, which is the first thing that I hear people say when I tell them about trunk based development. But actually, a lot of big companies including, Google, use this with literally thousands of developers all committing to master at the same time.
[0:07:41.4] DA: Yeah, I have talked to some developers that have been on teams of 80 plus doing this, which is not Google. Like Google scale is Google scale. You know, 80 is like — I can imagine what 80 developers look like.
[0:07:55.1] MN: Which is they have a hard time with the Gitflow with like seven. Then you have a branch and then it’s constantly being merged. You got to make sure of that your features are in before you merge them into master. We have an interesting Git etiquette rather. We have Git etiquette at the client that I am currently at. One of the features we have is like the squash and merge.
So if you have a million commits and you squash them, it will automatically squash and merge them onto master but before you do that, you have to rebase so that it just puts that one commit of your entire feature onto right on top of master, which is great. But even then you have to make sure that your code is up to date and what not, which gets really, really hairy with Gitflow.
[0:08:39.4] DA: Yeah, I’ve had problems with migrations with Gitflow because with Jango working on the rails over the past couple of months has been eye opening because this is not a problem in rails. Rails uses timestamps for their migrations but Jango is like, “This is migration number two,” and if I am doing a migration over here and I have a whole new feature ranch and then someone else does a migration over there and they also have migration number two on their feature branch, then which one is really migration number two?
[0:09:11.7] MN: Which one is two? Oh my god that would hurt. So if that the benefit of using like trunk driven development where like at that point in time, you made a migration and this person made a migration but if the migration happened and then they merge it onto master, when you pick up that branch from there and you’re on migration, you will have number three.
[0:09:31.9] DA: Yeah, so like the strategy with trunk based development would be to do the smallest of the work possible. So maybe your commit might just be the migration piece of that. So you wouldn’t squash it down with all of the other things related to it. Once you are done and you’re happy with it, you would just make the migration and then you’d commit it and then you’d push it to master and then now deploy and run the migration and then people would rebase and run off of that.
[0:09:59.4] WJ: It encourages people to write code in a way that is backwards compatible because you have to do things piece meal because everything has to be broken down into these tiny little snippets and I think that actually helps with architecture. It creates an incentive for people to make their code modular and to break things out into new files or new classes and new methods. It helps to avoid bloat.
[0:10:25.5] MN: So by having a big team all thinking in these small little comparable moving parts to ensure that things are backwards compatible, how often are you on GitHub? I imagine that would have to change the way you review code amongst developers or get your process into master a lot faster so that you can continue doing the work in this small bite-sized changes. My question is, how does that affect the efficiency of doing trunk driven development with the big team?
[0:10:59.7] WJ: We just do everything in pairs and there’s at least once senior developer in the pair and so that constitutes code review. If you were coding by yourself then we would ask for a code review but we just don’t do that.
[0:11:12.2] MN: Right.
[0:11:13.7] DA: Yeah, so that’s one way that you can reduce that overhead just to pull code review completely out of the process and then if there is a refactor required then a re-factor is a tiny unit of work that you’ll just add to the head of the repo, just keep incrementally working towards it.
[0:11:32.5] MN: That sounds like an episode in itself that we would have to get into poor request and code reviews and how to get down and dirty on that on the GitHub and whatever Hub you guys are using?
[0:11:45.7] DA: Yeah, I mean one of the tricky things about it is that you need to be like pretty disciplined and have like pretty high level developer maturity in your team and also like the tooling used to be good too because it is really supported by having good automated checks on your code like the code quality, running the tests, having a CI pipeline so you can immediately deploy the code and you get instant feedback and hopefully having zero downtime deploys. So you are not making all of your stakeholders angry.
[0:12:17.1] MN: Yeah because they want those changes out fast, so make sure you can keep up.
[0:12:22.3] WJ: Yeah, I think Facebook calls it the trunk tests. They’re the core tests that you absolutely need to pass and they are all unit test so nothing slow and then those run every single time you deploy, which is a lot.
[0:12:36.1] DA: That’s pretty cool. So Facebook is also using trunk based development?
[0:12:40.4] WJ: I think so, not positive.
[0:12:42.3] MN: That is pretty cool. I mean to know that these big engineering teams are using trunk driven development. I mean just going back to my concern earlier, you have some work, you make a small change, you push it out to master and then suddenly the website is down. Whoa.
[0:13:02.2] DA: Did I do that?
[0:13:03.0] MN: Yeah, that’s a good old Steve Urkel “did I do that?” moment and things are crazy. Like you have to make sure, correct me if I am wrong, but you also have to make sure not only is your deploy pipeline like snappy enough to catch or to get those new features, but to be able to pull them right out as well.
[0:13:24.3] DA: Yeah, so I mean there’s definitely like a little bit of GitFood that you need to get down on. Like I have never done a revert commit actually when we are doing Gitflow development before. Like it’s very rare that you need to do that but I’ve done quite a few of them. You’ve seen trunk based development because you put something out there and you’re like, “Okay let’s not do this right now. Let’s pull this back.” So the best way to do it is to instead of rebase against the earlier commit, which will scrub everyone’s re-page to a GitRevert and throw a revert commit in there.
[0:14:00.0] MN: Oh interesting.
[0:14:01.3] WJ: Yeah, you want good monitoring and production. You want good alerting as well and you want a really easy system for rollbacks like one-button rollbacks.
[0:14:13.6] MN: Okay guys, it’s on fire. We need to do something about this.
[0:14:18.5] WJ: But it does incredible things for your pace of development.
[0:14:21.3] MN: Oh because you are not like caught in the fear of breaking something.
[0:14:26.6] WJ: Right, if it hurts do it more. Eventually you just get really good at it.
[0:14:31.1] MN: Well the whole.
[0:14:31.8] WJ: Then it’s like, “Okay, well you know you can’t stop people from occasionally breaking things so just have that happen more often so that people get better at dealing with it,” and then you get to the point where you get automated smoke tests in production and it just immediately roles back your code if something is wrong.
[0:14:47.1] DA: Yeah, I like that. That saying you can move fast and break things or you can move slow and break things but it’s not an option, you’re going to break something.
[0:14:58.7] MN: You’ll still going to break things, unfortunately. That’s the rule ladies and gentleman, you’re going to break stuff, just saying. You will break things. Cool, I mean one of the things I have that I currently deal with in Gitflow is just the amount of either long live feature branches or everyone and their mother’s pairing or working on their own thing that’s like 80 different features that are out and then they all get merged back in at some point in time, and you have to work together to ensure that that’s the case.
But I see the benefit in Trunk drive development where things are just broken down into smaller pieces of work, even smaller than the features themselves and then everyone can then work at a sustainable pace where they can move fast and break things because you will break things.
[0:15:48.2] DA: Yeah and the nice is not having to merge other people’s code. Like doing that developed branch merge and rebasing on develop like that’s a pretty painful process sometimes. Like you can get merge conflicts on things that you’ve not worked on in someone else’s code and then you’re sitting there staring at the screen like, "What is this, why is it hear? I need to do like a get blame, I need to get this guy over here like what?” You changed this configuration file line over here, is it supposed to delete this or is this supposed to keep it? You know, it gets tricky.
[0:16:21.6] WJ: Yeah, I think instead of having a branch that that then has to be rebased the idea is to create a branch by abstraction, that’s the phrase they use in trunk based development. I think that makes a lot of sense. That in combination with feature flags.
[0:16:37.4] MN: Oh interesting, so do you find yourself merging into master a lot more in trunk driven development more than Gitflow?
[0:16:44.8] WJ: I find myself committing directly onto master.
[0:16:46.9] MN: Oh whoa. Hold on, this whole time I was thinking about like branching off making the PR going back, no, no, no.
[0:16:57.5] WJ: None of that no. You just commit right to master yeah.
[0:17:00.4] MN: Oh like no.
[0:17:01.6] WJ: Sometimes I will cut a branch afterwards and then push that up so that I can get CI to run and show that branch to somebody else or maybe push it to a staging environment and then be able to come back to it later. But generally in trunk based development, you commit to master.
[0:17:20.0] MN: So like straight up like even right now at the client that I have right now, you can’t even get push up to develop it will say, “No, like that is not allowed at all.”
[0:17:29.9] DA: Oh you got some hooks?
[0:17:31.1] MN: Yeah, no, no, no, yeah it’s like a GitHub. It is like, “No.” We have it in GitHub where this branch is not over writeable by regular people, only admins can do it. So even if you did GitPush to develop any change, GitPush force. No, no, no that’s bad, don’t do that, stop. It’s so scary here that you can actually just commit straight to master and trunk driven development. I think I might just do the trunk driven development just so I can push straight to master for like a first world anarchist here. Just making changes, straight up all to master.
[0:18:08.8] DA: Oh I like that, yeah first world anarchist? Yeah, I did feel like that in the first week when we switched our from that, like because we were, we are not doing like a full Gitflow model. We are just doing featured branches but when we switched over to trunk driven development, it definitely felt like something is very wrong but then after like a day or two, it’s like, “Okay this is fine, this is all right.”
[0:18:35.0] WJ: I found myself accidentally shipping things because we have it set up so that if you pushed to master it will automatically deploy.
[0:18:41.1] MN: Oh my god.
[0:18:41.9] WJ: Yeah but we have a good test wait and so if I push something that is broken then I just make the build right and then I have to fix it, it’s embarrassing.
[0:18:51.8] MN: Oh yeah, I imagine. I’ve heard people have alarms when the bell goes red or all sorts of weird things.
[0:18:59.6] WJ: We don’t have any. On this team there’s no shaming though if the build goes right, it’s very supportive and what happens is like –
[0:19:05.9] DA: Well just in front and your back you know? Behind your back there’s so much shame.
[0:19:08.7] MN: Oh yeah.
[0:19:09.6] DA: Oh yeah.
[0:19:10.4] WJ: And then you don’t even realize like a lot of times it would not have occurred to me that I could just to deploy it already and having trunk based development encouraged me to ship code, working code sooner and sometime you don’t need the entire feature like there’s some portion of it that’s already useful.
[0:19:30.8] MN: Right, so I think that for trunk driven development to be a thing, the infrastructure around the actual code base has to be like highly maintainable or the structure for having test and having test before it gets pushed and deployed and having CI being able to ensure that everything is good. These things have to be in place even before you can move onto trunk driven development.
[0:20:03.7] WJ: Absolutely. You need good infrastructure and you need a good platform.
[0:20:06.0] MN: There you go. Those are the words I was looking for. Yeah, cool so the conversation we just had right now pertaining to Gitflow and Trunk Driven Development, I hope that there are people out there who have been doing one and want to try the other. I am actually looking forward to an opportunity where I can use trunk driven development after being a Gitflow individual.
Is there any teach and learns that we have out in the midst? Because I have one maybe even two.
[0:20:37.6] WJ: I have a learn, I recently learned that you can schedule maintenance in Pagerduty. So that if you are doing load testing or something that is going to intentionally bring your servers down in a way that would normally set off your alerting, you can prevent it from paging someone if it’s in the middle of the night.
[0:20:54.2] MN: Oh nice, yeah interesting.
[0:20:55.9] WJ: They have an interface where you can go in and schedule maintenance through the website and then they also have an API that you can hit, which is cool because then you can set whatever job it is that is going to run that’s causing the downtime to trigger the maintenance on pager duty and then when it’s done and has finished spinning things back up, it can make the call that will start it back up again which is cool.
[0:21:21.4] DA: Like my books.
[0:21:21.8] WJ: Yeah.
[0:21:22.4] DA: Nice.
[0:21:23.1] WJ: So that minimizes the amount of time that your pager duty is turned off for and you can even set it up to check and see if things are currently valid and then delay for 15 minutes or however long you want before you turn pager duty back on. So that you don’t get a false alarm when the system is just getting settled.
[0:21:44.6] DA: I actually have a learn as well recently. So with the state of the Java Script community, there are so many tools that are great like web pack and Pebble that allow you to write the code that you want and not really have to worry about compatibility that much. So I’ve forgotten that you do need to worry about compatibility when you’re writing embedded script. I really wanted to use this really slick object that values method to get the values out of an object in an interview with them using math.
It was great, this code looked wonderful and unfortunately like you and I pushed it, I realized that there were errors happening and a lot of devices and it turned out that that’s just not supported at all in Safari. Just no plans for it to be supported in the future, just not happening.
[0:22:40.5] MN: It’s not. It doesn’t work at all in Safari?
[0:22:42.4] WJ: Why would they plan not to do it? Isn’t that in the spec?
[0:22:46.6] DA: It’s a draft spec right now so. It’s a wonderful draft spec but it is just a draft spec right now. So don’t use that, use keys and index in.
[0:22:55.4] MN: There you go. Talking about Java Script, I think I mentioned before in the past and I have been dabbling in react native, I kind of stopped but then jump back in and things look a lot cooler now the way that you can spin up a new react native app as pretty cool and they have this app called expo that you download separately. You can have your application running on your machine and when your MPMs start, it gives you a QR code or one of those little bar codes that they have for the computers.
And you can scan it with the expo app and when you scan successfully, it will download depending – you have to be on the same network as your computer or laptop and when you scan it correctly, it will download the app, build on your phone and then hot reload any changes that you make on the computer onto your phone. It is insane, it’s so cool to do react native work because you can know immediately the minute you save, you see your phone spin, build and then boom.
Again, no connections, no nothing just scan and reload that’s it. I was like, “Oh this is cool” there’s no reason for me to not be building things like react native. That was so cool, the feedback loop that you get on errors is just snappy like if you have an error, you make the change and you save them and immediately your phone is already building again, the changes when you see it. So I am just trying to figure out how to do the react native.
So it seems to be a little weird like the app doesn’t look good at all, so I am trying to figure out what packages I would need to download to make things look nice, any form I have to do. But yeah, react native is fun. I am trying to see if I can figure out how to make that app that I have or my face on it that just yells anytime you click on it. That’s what I am trying to do right now so just manipulating sounds on the phone for both android and iPhone is really cool.
[0:24:52.1] DA: Nice.
[0:24:53.4] MN: That’s the episode ladies and gentleman. I like to thank my co-host, Dave thanks for coming on down and our producer William, always good having you here and thank you for listening. Feel free to hit us up on Twitter.com/radiofreerabbit. This is The Rabbit Hole, we’ll see you next time.
Links and Resources:
