On the show today, we’ll be talking about testing pitfalls and avoiding common testing blunders. We wanted to get a word that felt like the things we deal with in testing. “Blunders” is that word! Blunders are those common mistakes we find ourselves making when we are testing. I’m sure you’re thinking of some right now. To join us in discussion of this topic we welcome Dennis Liao, a new consultant at Stride. So stay tuned as we jump into talking about testing and the pitfalls that come from them. Keep listening to hear it all!
Key Points From This Episode:
Transcript for Episode 88. Testing Pitfalls and Blunders
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast in fantabulous Chelsey Manhattan. I’m your host, Michael Nunez. Our co-host today.
[0:00:09.8] DA: Dave Anderson.
[0:00:10.4] MN: Today, we’ll be talking about testing pitfalls and avoiding common testing blunders.
[0:00:17.3] DA: We’ve all had them.
[0:00:18.1] MN: yeah, we’ve all had them. I wanted to get like a word that felt like the things we deal with, with testing.
[0:00:23.9] DA: Yeah, definitely hit home.
[0:00:26.2] MN: Blunders is the one. I think so. Yeah, this will just be common mistakes we find ourselves when we’re testing. I’m sure you’re thinking of some right now. Before we begin, we have a guest today. Dennis Liao, how’s it going, Dennis?
[0:00:39.7] DL: Doing fine man.
[0:00:40.5] MN: Awesome. Could you tell us a little bit about yourself before we start?
[0:00:43.8] DL: yeah, sure. I’m a new consultant at Stride, before that, I mean, I’d been an engineer since like 2009 but I’ve gone through the school and the boot camp route and I liked to boot camp route so much that I stayed to teach. I’ve got a bit of an academic side of things and the practical side of things.
[0:01:01.7] MN: Awesome. What boot camp did you go to?
[0:01:04.1] DL: I went to General Assembly, yeah.
[0:01:05.2] MN: Awesome, shout out to General Assembly.
[0:01:08.4] DA: And the family.
[0:01:09.6] MN: Yeah, friends of the show. Awesome. Yeah. We’re talking about testing and the pitfalls that come from them, does anyone want to start us off with the testing pitfall that we’re currently dealing with as of right now or in the past?
[0:01:22.9] DA: One I’ve seen is maybe sometimes people try to mock too much. You can get really aggressive about like mocking, I’m mocking that and then when you change the code, the test doesn’t fail.
[0:01:34.7] MN: Yeah, no. You mock things that may change in the future but it was mock so the test would pass anyway?
[0:01:41.3] DA: yeah, false positives because you know.
[0:01:43.3] MN: Those are the worst.
[0:01:45.1] DA: Yeah.
[0:01:45.2] DL: You fully orchestrated the happy path just to make it green and didn’t realize, you know, it’s broken.
[0:01:54.2] DA: I wrote a test today where I mocked date and I’m trying to verify that the updated date is happening properly. First of all, it was a huge pain in the butt to mock the right date and where it was coming from in python, who knows? He used a library and then actually ran the test and I’m like, there it is, going to see red test and then it was green. What’s going on?
[0:02:18.1] MN: That’s the worse, right? When you expect it to fail and then it passes and you’re like wait, why?
[0:02:24.4] DA: Yeah, exactly. Question everything. If I hadn’t written the test first then I don’t know what it would have done.
[0:02:31.2] MN: Right, Java script leads to a whole lot of that too. It’s just a whole ton of different libraries, you could probably have to mock out something from go to the dictionary and pick a word and this is probably the package that you have to mock out, broccoli JS, confetti JS, whatever JS. I got to mock it because I want to make sure it’s going to return this nice thing for me and like that’s always fun.
[0:02:56.9] DL: Yeah, for sure.
[0:02:58.3] DA: What might be an alternative to mocking like if I – is there some other approach that I could do besides mocking Broccoli JS.
[0:03:08.4] MN: The broccoli JS.
[0:03:09.9] DL: I don’t know, I think we’re stuck with it for a while, especially in Java Script, it seems like you know, that’s the way people are approaching it. I just wish that they would kind of standardize their approach with the libraries that they’re using. One of the things I’ve seen in the past, you know, something I wish that people would do better is maintaining the mocking libraries that go out into the community.
One problem I’ve seen in the past is a library going out to mock and how to interact with these mocks but the documentation is just not there and you get all these magic methods that they don’t belong to your code base, they’re specific to the implementation of the mocking library. If you don’t’ document it correctly, you’re just confused and stuck in the water.
[0:03:51.3] DA: Right.
[0:03:52.4] MN: I think we mentioned before in time, I think it’s Martin Fowler’s Testing Pyramid for those who are unaware of the testing pyramid, everything is either a pyramid or a quadrant, right? Always happens. The pyramid at the bottom, very similar to your daily nutritional value pyramid. Might have changed and will might change the next time you listen to this.
[0:04:15.6] DA: The daily bread.
[0:04:15.4] MN: The daily bread, yeah. The very bottom and the most that you should have in your code base are unit test and that should be like things that only exist very close to the implementation than up and then as we go up in the skinnier part of the pyramid would be the integration test which will lead to some mocking. You should definitely have way more unit test than integration test. Then last but not least, at the very tip, the small amount will be a full end to end test that probably calls like either staging environment or like the real deal.
Those should not be written so heavily and should like take up a very small amount of your testing suite.
[0:04:59.7] DL: Right.
[0:05:00.4] DA: Each time you're getting closer and closer to like the true end to end. If something breaks, if something changes, they should break.
[0:05:08.5] MN: Right, yeah.
[0:05:10.3] DA: Unlike my weird test.
[0:05:12.0] DA: Just so you know, before I started talking. I did google to see if there was a broccoli JS and there is a broccoli JS. That’s why I was cringing.
[0:05:20.6] MN: Shout out to broccoli JS for the show. There’s a lot to mention out there. Sticking around on the java script realm, A sync weights testing and waiting for callbacks to happen, it’s definitely one that I find myself like that is hell, pretty much for me.
[0:05:42.4] DL: One of the challenges right now is you know, kind of the different syntaxes that we’re currently having to deal with in Java Script which makes dealing with legacy super challenging. What some of you may experience when you go out there, you know, you’re dealing with a code base that has test that are still working with promises rather than a sync wait.
[0:06:03.0] DA: Or, even just regular old callbacks.
[0:06:04.5] DL: Yeah, then what do you do in that situation, right? You do a massive refactor where you rewrite everything or do you keep rolling with it?
[0:06:10.9] MN: Right.
[0:06:11.1] DL: How much can you tolerate?
[0:06:12.2] DA: Find a new job.
[0:06:13.3] DL: Man.
[0:06:15.5] MN: It depends, if it’s legacy and you're not, I think we had a conversation about this before in time but if it’s legacy that just will exist and not a lot of action happens in there, not everyone’s going to jump in. Then you should stick to the syntax that’s there and hopefully in the near future, deprecate that particular service so you can update the syntax.
If it’s here to stay for some time, then you could put in some of the work. Function by function, I wouldn’t say look at the file, change everything.
[0:06:44.6] DA: Yeah.
[0:06:45.3] MN: I follow the, not the boy scout’s rule but I call it that. It’s probably not true. The rule is, leave things better than how you left it. If I’m going to make a change in a particular legacy function, I’ll try to you know, update the little syntax here whenever possible. If it’s ES5 and there’s no babble, I’m not going to go and install web pack just so that I could write six functions.
[0:07:11.8] DA: You got to get that arrow function.
[0:07:14.5] MN: Fat arrow’s awesome.
[0:07:15.7] DL: Yeah.
[0:07:15.8] DA: Yeah, A sync definitely is a lot more clean to reason about them, anything else?
[0:07:23.2] DL: If you do find yourself with the time to do it, I would totally suggest it, this is just night and day when you look at the get diff behind it.
[0:07:31.3] DA: Yeah, I’m pretty excited like, there’s been a lot of recent developments in the python language and A sync away has made it’s way in python as well. I think maybe inspired by java script but yeah, just seems a lot nicer than anything that we had before.
[0:07:47.6] DL: Do you have a gigantic transpiring step in between?
[0:07:51.9] DA: No, you just have to upgrade from python two to python three.
[0:07:55.3] DL: Wonderful, okay.
[0:07:56.3] MN: Is that difficult right now?
[0:08:00.1] DA: That’s happy birthday for python three and years. There you go. That’s a different podcast.
[0:08:08.0] MN: Yeah, there’s a whole other episode.
[0:08:09.7] DA: Yeah, what about like multiple assertions?
[0:08:13.3] MN: I don’t like those.
[0:08:14.1] DL: Yeah.
[0:08:15.3] MN: Here’s the problem though. I totally understand why it exist sometimes.
[0:08:21.3] DA: I mean, I got so many things that I need to tell people.
[0:08:27.1] MN: Exactly.
[0:08:28.0] DA: I don’t have that much time.
[0:08:29.9] MN: Yeah, it’s just like often times, you don’t want to rebuild the same functions to assert some things so I imagine some people just like all right. I insert all the things and it’s fine and it’s like, no, it should be one at a time. One insert, one test, that’s it.
[0:08:41.7] DA: Yeah. Well, sometimes like things could be one assertion and they just stretch it out, they drag it out. Assert the first element of the array is one. The second element of the array is two. No, we could do this better.
[0:08:55.7] DL: You can pick that apart, right. Generally, you know, you want to stick to one, there are some times where you do want to test two things that are directly related to each other and that’s fine. You know, if you see this all over the place then it becomes problematic then you’re going to find yourself in a lot of pain later on.
[0:09:15.2] DA: Good rule of thumb.
[0:09:17.1] DL: One task, one assertion.
[0:09:19.5] DA: Do that testing Haiku. Arrange, act, assert where you get your data setup, you do the thing you’re going to do and then you do want assertion.
[0:09:30.6] MN: yes, testing Haiku, I like that, that’s pretty dope.
[0:09:34.1] DL: Do you find yourselves able to write the test before you write your code?
[0:09:40.0] MN: It depends. What I tend to do is whenever possible, yes, I find that in Java Script, that never happens. I usually write the implementation I want to do and they’ll say, “okay, I kind of know how to test this" and then I delete everything and then I write the test and then I implement it. Then now that I understand how to write that test, then I can continue on what any of the edge cases.
[0:10:03.6] DL: You're a boy scout over you.
[0:10:05.3] MN: Yeah, I’m really trying. I’m trying out here. I mean, it’s hard, I’m not going to lie, it’s dumb – especially in Java Script, I’ll find it.
[0:10:11.1] DL: It is painful a little bit, yeah.
[0:10:13.1] MN: Because it’s just like, in Java Script.
[0:10:18.6] DA: Yeah, I guess that’s like to do that, that would be like pretty strict about it, it’s a good way to avoid test that don’t fail when the code changes.
[0:10:26.2] MN: Right. That’s n other thing that we could talk about, you get those test, you make changes to the implementation and somehow everything is still passing, it’s like wait a second, if I change other things. Have I changed other things then how is this not going to catch any of the arrows that would expect if we made a change.
[0:10:46.2] DA: Right, that’s just like, that’s just being smart, right?
[0:10:51.2] DL: We’ve all been guilty of that though.
[0:10:53.5] DA: You know? In testing something against some constant and then you know, making the test dynamically update
[0:10:59.3] MN: Yeah.
[0:11:00.1] DL: When you do edit that value and you think you are being slick, right?
[0:11:04.7] MN: Yeah, no don’t do that. Don’t do it, don’t try to be slick. Things are supposed to fail then you fix them and that’s it. That’s great.
[0:11:13.1] DA: Yeah sometimes you’re going to feel a little bit like to launch your bookkeeping where you are just like testing that this thing that I wrote over here is equal to this thing that I’m going to copy and paste and write over here but maybe that is a sign that you need to pull the lens out a little bit and test something that is more close to the actual business value that you are deploying.
[0:11:32.8] DL: It’s definitely useful to have something fail just to signal to yourself and have that extra step of caution. Another thing that you should be looking out for is that your test should not bleed into each other. They should be atomic right? So they are all independent from each other which means you can randomize their order and then you can paralyze, parallelize their running.
[0:11:57.8] DA: That totally makes sense. So in the worst case scenario, you have a set up that is shared between multiple tests and value that has changed in one test. It is used in the subsequent test.
[0:12:12.0] DL: Right, so if your test all affect the same collection and you are counting on the length of this collection being incremented every time you run the spec, I mean that is cool but that means your spec all run sequentially and they all depend on the same dataset which can be dangerous and really gnarly to work with, right?
[0:12:29.2] DA: Yeah, I have definitely have some problems where factories might use global counters for sequences and then not only if the test round in different order where you get different result but if you add a test for the one that you are working on then that can also scrub all the test after it.
[0:12:49.4] MN: Just a fun fact in terms of finding solutions for those kind of test for people may run into, RSpec has a command called RSpec bi-sect and it will generate a seed that you can use so that it runs in that order and then you can actually see when a test fails. You can say “okay, the test fails and between these two files and then it tried to run them and then it goes down to the line as to what caused that error to happen in the first place.” Definitely check out RSpec bisect it is amazing.
[0:13:25.5] DA: Easier for Python, you know?
[0:13:27.5] MN: Oh sorry, I don’t know about Python.
[0:13:29.3] DA: Got to make py-sect.
[0:13:32.9] MN: Open source project right there, awesome. The next one for me personally, I ran into it embarrassingly often is when you test the wrong thing. When you find out that the thing you were testing on wasn’t the right thing you were supposed to test or a test that you’re just like overlooked the testing and then you don’t really understand why it happened in the first place. I don’t know if anyone feels the same way.
[0:14:00.3] DA: You are just testing in the integer as an integer?
[0:14:03.0] MN: Yeah, well that could be one for sure where you have a single tip and obviously the single is going to return but like the constant that we mentioned before. The constant is going to return the same thing because it is a constant that we have in the test.
[0:14:13.4] DA: Yeah, the constant is a constant.
[0:14:15.2] MN: Yeah, oh God, I mean this may bleed into different libraries that you may use that will cause this to happen and that kind of stuff which is really, really painful. It can drain your entire day.
[0:14:27.5] DA: Every time I am testing library code.
[0:14:29.8] MN: Yeah. You shouldn’t, I guess first you use libraries that are trustworthy and well documented and spec so you don’t have to test the library itself. You’ve got to trust it to a degree I guess.
[0:14:43.4] DA: We can just trust that part or will make a query if we ask it too nicely.
[0:14:51.6] DL: You know you don’t need to be testing your ORM and a unit test for another model or service object or whatever it is that you are in. You just need to assert that the behavior that this class deals with is being asserted and tested against.
[0:15:06.4] DA: Yeah, that totally makes sense. That sounds like a high value test.
[0:15:10.0] MN: I think we went over a lot. I mean I am sure there are a lot more testing blunders I am going to say. These are the ones that actually came up that we discussed because we run into them in our day to day lives and things that we’ve just seen in the past.
[0:15:24.6] DA: Yeah, please let us know if you have any more testing blunders.
[0:15:28.7] DL: Yeah I mean these are fun conversations, right? This is what we sit and just complain about all day so yeah.
[0:15:34.3] MN: Yeah, I mean we are doing the right thing by testing which is great but you know, we’ve got to follow through this problems and hopefully by listening to this you may find this useful and not fall into these blunders yourself. Cool, do we have any teach and learns?
[0:15:48.5] DL: I am reading a book called Simple Rules right now. It’s not really a technical book more than it is like a management book and leadership book and it focuses on how to run your organization or live your life in general by a simple set of rules and minimizing the different ones that you can apply when you are trying to make complicated decisions and it is really insightful. There is a lot of good stuff in there about how the most influential people in organizations have operated by using the paradigm of keeping your rules simple.
[0:16:25.3] MN: I see, do you have a simple rule you can share with us really quick? You don’t have to explain it just tell us what you like.
[0:16:31.7] DL: Yeah, you know you stop drinking coffee after 3:00. You go to sleep better, right?
[0:16:37.2] MN: Whoa, don’t tell me how to live my life come on. No but you’re right. I definitely don’t sleep if I have coffee late.
[0:16:45.9] DL: Right or write test before you write code, you know?
[0:16:48.7] MN: There you go.
[0:16:50.5] DA: It is a pretty simple rule I guess. It is not prescriptive. It’s just like keep it simple and keep it short.
[0:16:58.6] MN: Right.
[0:16:58.7] DL: And it’s back by really nice examples and I don’t want to get into it now but you know, it is definitely a fun read.
[0:17:05.7] MN: Awesome.
[0:17:06.3] DA: Okay that is for the next podcast.
[0:17:08.0] MN: Yeah, don’t drink coffee after 3:00 there goes a simple rule right there. Dennis it was awesome having you on man.
[0:17:14.4] DL: Yeah, thanks.
[0:17:15.1] MN: You should come by more often.
[0:17:16.0] DL: Sure, I mean you know, you all grab me kind of randomly today and I decided I’d have some fun.
[0:17:22.9] MN: Awesome, yeah it happens.
[0:17:24.4] DL: Yeah I am a long time listener first time contributor so.
[0:17:27.3] MN: There you go, to many more.
[0:17:29.5] DA: Right that goes for everybody out there.
[0:17:30.9] MN: There you go.
[0:17:31.4] DL: Yeah.
[0:17:31.7] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five star review and help developers just like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries and my amazing co-host, Dave Anderson and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
Links and Resources:
