Blastoff Rails 2026 - Fewer Tests, More Confidence by Ifat Ribon

5, 4, 3, 2, 1. We have lift off. >> Our next talk is from Ifat. And uh I actually had seen Ifat talk before at RailsConf 2023 um in Atlanta. So uh I didn't remember that. I saw her uh talk come through the CFP and I was kind of looking at it. I was like, "This one looks good." And then I I kind of made that connection. I was like, "Oh, I really liked uh her talk. I really liked her speaking style." And uh put her on put her on the short list. And I was waiting because uh this spring summer, there's three conferences right in a row. There's RBQ, there is uh BlueRidge Ruby, and then BlastOff. And I was kind of waiting for RBQ and BlueRidge to release their speaker lineups because I didn't want to overlap speakers too much um cuz you you know, you want to put on a a different conference. And uh you also don't want to like have speakers giving two different talks in two different months like Ifat is doing. Uh cuz that's that's a lot. That's a lot of work. And um RBQ released their lineup. I was like, "Cool." And then BlueRidge released their lineup and Ifat was on it. And I was like, "Ah, shoot." And she she submitted a different talk for for BlastOff. And I was like, "Uh okay." And I I kept coming back to it. And I was like, "You know what? I'm just going to send her an email." It just said, "Hey, I know you are doing BlueRidge. And I know that doing two talks in two months is a lot. Um so if you don't want to do that, that's totally okay. But I just thought I would throw it out there that we would love to have you at Blast Off if you're up for it. And she replied almost immediately like, yes, I'm excited. Let's do it. So, uh, thank you so much Ifat for putting in that work because that is a lot of work. And, uh, she did a great talk at Blue Ridge. I'm sure she's going to do a great talk here. So, I'm really excited and thank you so much. Let's give a warm Blast Off welcome 45 in 5, 4, 3, 2, 1, Blast Off. >> [applause] [applause] >> Well, first of all, thank you, Travis, for, yes, accepting me, uh, here. I'm very excited to be here. And thank you to all of you for being here. And thank you to all the speakers that have already come and our one more speaker yet to to come at the end of this day. Um, my talk will be maybe a little bit, uh, more specific, but I think we'll be touching on a lot of the themes that we've already gotten to hear about and and probably talk about in some of the breaks that you all have had. So, this talk, Fewer Tests, More Confidence, is going to focus on automated testing and of course using AI to do so. So, as we all know, AI can generate code and even better tests extremely fast. And, uh, if you're like me, you probably don't really enjoy writing tests, but as Neha pointed out, like, you really should write those tests. And luckily AI can come right in and do that for you. Amazing. However, all those tests aren't really free, right? Those tests come with more to review. Uh, even if you're not the one reviewing it. With more to reason about, tests our code, right? More to maintain, right? There's that test suite can grow and grow and grow. More to wait for in CI, which has been a problem even before AI. And more tokens or actual compute in CI. And maybe any individual test doesn't cost these things, but as they add up and as you all know, AI gets a little exuberant. It wants it to add up. It's incentivized to add up and show you just how much it can cover and just how much how much more it can do. It will add up to all those things on your mental tax, on your financial tax, whatever it may be. So, AI can generate more tests. It doesn't always generate better tests. So, how can we think about making that test suite better? Well, first thing I want to recommend today is to define a testing philosophy. And to start like any good philosophy, we're going to define our why and our values. So, why do we test at all? Well, we want to validate that code meets requirements, right? Uh if you don't know, this is the six-fingered man and uh Inigo Montoya had to check that it was in fact six fingers before he proceeded with his revenge to kill the six-fingered man. We want to protect essential user flows and business logic, right? Uh in any app that you may be working on, there's going to be some critical aspects where you want to make sure you get it right every single time, that your users never encounter an error, their data's always protected, that they always uh uh experience the app as you intended. You want to catch key failure states and critical interdependencies. Uh you may not always be working on a true monolith. You may uh be integrating with other things and you want to test that those contracts are always working as you expect because you may not be in charge of all the code that you work with, but you want to validate that it is always working. And oops, sorry. Lastly, enab- enable refactoring with confidence. Again, if you're anything like me, uh you may read code you wrote or the AI wrote 6 months ago based on your principles and your uh design practices. 6 months later, you went to an awesome conference, you learned a bunch of new stuff, and you're like, "I can do that better. I can tell the AI how to do that better." You want to be able to refactor that with confidence and hope that your test still pass. So, that's a little bit of the why. Let's talk a little bit about what even are tests. And uh maybe you you all are familiar with these uh test categories I'm going to go over, but I am going to just define the nomenclature that I like to use as uh sometimes people use slightly different nomenclature. So, uh there are unit tests. Uh maybe we all kind of agree on this one. This is when you're testing uh individual pieces of logic in isolation. Uh you may think a lot about your model tests, your service classes, uh very low-level work. Uh these are really great for those kinds of specs and for edge cases. The uh you want to be able to run these unit tests fast and frequently, so they need to be very discrete. Uh system tests, I use the word system test to talk about anything user flows uh where you're actually kind of loading up the browser or the session. Um other people may use other words, but that's why how I define system tests, the user experience, and things like that. Um these uh are certainly more expensive. Um so, you want to be a little bit more moderate about them. Use them selectively and also run them selectively, right? Maybe you run all of your unit tests uh right before commit, but system tests, maybe you only run feature change and in CI. And the last category uh that I'll name here are end-to-end or integration tests. And this is where I define um the times we actually want to make sure those components work together. Your external APIs are continuing to provide the contract that you agreed upon or expect. These are the slowest, most expensive tests because you may actually be hitting an API or something like that. So, run these periodically. In fact, you may only run them on CI or things like that. So, many of all have probably seen this illustrated as a bit of a pyramid which points to kind of distribution of tests you might want to have in your app. Unit tests because they're fast and cheap, you can have the most of those. This definitely costs less, feel free to run them up there. System tests sit in the middle there, right? Probably fewer than unit tests. And your integration tests are are probably few and far between, really focusing on those external dependencies. So, as we described, um you know, unit tests are very discrete, very specific. And as you move up the pyramid, you get a bit more broad. You might want to just hit the happy flows and things like that. So, maybe you all have seen this before, but this becomes a really good foundation for how we're going to teach an AI to also think about writing these tests. Another good part of a good philosophy is to define our principles and this is the how we're going to go about defining what makes good tests so that we can tell the AI. I've come up with or converged on four principles after reading a lot of There's a lot of articles and content out there. A lot of people have talked about this before. Key principles that I've converged upon will be illustrated here. So, the first one is to focus on behavior over implementation. Right? A simple use case of this is to avoid testing library or language code. Rails is phenomenal, very well tested. You don't need to test any of that. You know, you don't need to test that the enum methods work. They work. They've tested it. You can Uh, focus on public methods and their results. Um, I wanted to mention that uh I work at an agency or consultancy and we love to say it depends. I live in nuance and ambiguity. Uh, I'm trying to stick to some absolutes and guidelines, but of course there are always edge cases. Focus on public methods is probably the biggest one that uh people might be like, "Oh, wait." Totally fine. If there's something very complicated uh that's in the private internals, test it. Feel free. But in general, a good test will focus ultimately on the outcomes of your public contracts. Um, and then you want to be able to assert your test still pass after a factor. So, uh maybe a simple example is uh you know, this uh very simple contrived test is uh trying to test a very low-level private method that actually calculates what a discount is. That's great. Uh, there may be some math in there, but probably more importantly, you just want to know that the total that you're showing the user is correct. So, that's a good uh contrast of not focusing too much on implementation, focus on what is the outcome that you want to make sure is working every single time. The next principle is to keep test deterministic and this one has become obviously even more important uh as AI is notoriously non-deterministic. Um, so what that how that kind of bears out is to um do things like keeping your data local so that you can avoid having uh data created in different tests pollute your other tests. Much like a good scientist, you want to eliminate any confounding variables. Keeping data local is a really great way to do that. Uh, in unit and system tests, so not the integration test, go ahead and stub those collaborators. You're going to test all those pieces individually and you'll leave integration tests to make sure that those hand-offs work. So, to keep the other tests cheap and fast, just uh the collaborators, right? If your service class is calling a mailer, you don't need to assert that the mailer is going out. You can test that independently. Um, and then time obviously is a big one. Obviously it I'm sure we've all experienced a lot of issues with time and time zones and all sorts of things. So, there's a lot of beautiful tools out there for doing so. Um for instance, uh you know, if we just rely on the fact that this trial ends in 3 days, that test is going to fail pretty quick, in about 3 and 1/2 days. Um luckily, there's a lot of tools and other things that you can use out there to achieve your intentions of uh making it more deterministic. So, freeze the time for instance, so you're always remaining within the same time window. Uh the third principle here is to balance coverage with cost. So, we talked about the test pyramid and we kind of alluded to how you might think about moving up that pyramid. Uh but a bit more specific examples would be to choose those test types intentionally. Decide, what am I testing? Am I testing a very complicated edge case? Do I need to do all that setup for it? You find where you land on that pyramid according to those questions. You want to not only keep data local, but minimize it, right? It's expensive to make trips to the database. It's expensive to have a lot of just extra variables sitting around. You don't need that. Just create what you need. And uh again, in those uh system and unit test, avoid testing those external dependencies directly. You can stub those out and run your uh integration tests to make sure that that API is still looking the way that you expect. Um so, in a in a very simple case, and I would say this is the most common thing that happens to me with AI tests, is that I'll try to write a a request back or something like that, and I'll start writing system code. And I'm like, I don't care what the browser looks. I want to make sure that the controller's sending the data back right, or that it's calling the right collaborator. So, if you're ultimately just trying to assert that again, you have the right total for the user, you don't need to spin up the whole browser for that. Just again, test the method, right? Now, you can test that the total is showing correctly to the user and that would be a great system test, but if you're trying to test the logic, uh you can scale back on that pyramid. And the last one here is uh my favorite is using test design to influence and inform code design or vice versa. Um but oftentimes, if you find yourself struggling to write a test, it may be that your code is quite convoluted. And there's probably a way to simplify that to get a better test. Um so some some small heuristics or smells that you can check in on, test should have a single reason to fail. Small nuance here, system test because they're expensive, you may have multiple assertions and that's totally fine. Um but unit tests should be discrete. Uh results should be objective and discrete. You should be able to kind of see that a test failed and know exactly why and it should be easy to debug. And uh failures are actionable and specific to support that. Uh so for instance, maybe we've got a a user sign-up method and I'm asserting a bunch of things about it and it's possible that one method is doing a bunch of things, but uh really that either I need to break down that method or maybe I just need to focus on what that method is actually responsible for. Maybe I've actually circumvented some things in my code. So, if I can just hone down hone in on the one specific thing that we're doing, probably have a lot more confidence in that test. Um so yeah, good test design can definitely help inform code go design. Um also, if your code is well designed, it probably will be pretty easy to choose which test type and how you want to set that up. So, that was a lot of foundational work and um hopefully some of that is familiar and resonates with you all. I think it's important to come back to the basics now that we have incredible tools like AI that will just run amok. And we've heard a lot of things today. We learn heard about how, you know, judgment doesn't go away. We need that human in the loop still. And maybe that human in the loop isn't revo- viewing every line of code, like Kieran said, but how can you build up something like a harness to be able to achieve your values and your principles? So, how are we going to rein in the agent? So, we'll have a bit of a flow and um you know, I really enjoyed Kieran's talk this morning as he he talked about kind of the system he built and and we heard a lot of aspects about that throughout the day. But, if you're just maybe starting off on building your system or kind of coming back to the beginning, maybe we can walk through a bit of a flow about how to go about that and break it down. So, the very first step is to, well, just let the AI generate stuff. See what it does. Uh maybe with uh better tools and better models, like maybe Fable is getting it right more often or maybe as we get more Ruby code and Rails code indexed and available to the agents, we will see better results. But, for now, we don't really know. So, generate those tests. See what it comes up with. And it'll do it really fast for you. So, maybe uh with my order class here, it'll come up with four very quick tests. Amazing. Looks great. Looks like really great coverage, right? Well, the next step then is that we need to review that. And maybe at first you are doing that with your own eyeballs so that uh you can you later, you know, get a feel for what you're looking at. And the way that I think about reviewing tests is to land back on those principles that we talked about. And you can really just invert all those principles into flags and checks. Um so, all those uh bullet points I talked about earlier, just turn them around into uh flags that you can first do as a human to make sure you're going to feel for what you're seeing. And uh spoiler alert, you can later codify this for the AI agent. So, these are all the samples we saw before in each of those principles and uh making sure that your test uh pass each of these flags. So, let's take a look at these tests and uh for each one, we'll do a quick review. The first one here that it is checking that that discounted total, that's the outcome of our uh public method there. Great. That is core behavior. It's testing the outcome. It's small, discrete, just has the data that it needs. Amazing. That looks really great. Mhm. Here's maybe a small edge case here. We don't want to get to a negative number. Uh maybe we always want to cap the floor at $0 even if for some reason the discount is higher, right? That could be a real scenario. Okay. I think that edge case is probably worth keeping. Again, small and discrete, easy, cheap, fast. This last one, pending. Uh as I alluded to, my favorite example because the AI was doing this a lot to me. It wanted to test every little thing that I wrote in my model class, even the things Rails that I didn't care of. Not great. Uh I don't like that test very much. Um and then uh if you look closely, even though the description of the test uh that it returns a reduced total is accurate, is descriptive, it's and it even wrote the assertion a little bit differently, it's actually just testing the same thing as our first test. Like, we don't necessarily need to know that it was less than 100, we just care that we got the right total, right? So, I would say that that's a redundant test that is going to eat up If we have a lot of those, it's going to start eating up all those uh uh actionalities that we talked about. So, now that we've done our review and we've got a grip on whether the test meets our expectations or not, let's go ahead and prune. This is probably my favorite step of all time. I selfishly maybe love that AI generates extra stuff because first thing you need to do is cut the noise. Delete those tests we decided weren't worth it, uh not adding any value, not meeting our principles. Delete the the failed ones that just uh you know, didn't uh pass our flag gates. Merge any duplicate tests, right? Uh maybe you like how one was written or the other, that's great. Pick your favorite, but you don't need both that first test and that last test to know that you have a discounted total. And uh go ahead and and and shape what stays and start to think about how you like your tests written and things like that. And is my favorite step because I love to delete code. It's probably my favorite part of coding is actually deleting the code. I don't really mind miss as much the not typing. I miss the uh as the AI gets better, I'm going to have less to delete, I guess, but uh maybe that's a good thing. So, have no fear, delete delete delete anything that is not looking good or even start over, right? That's really good advice, as well. So, just totally start over, start capturing these things, and have the AI try again. So, that's about getting rid of tests that aren't very good. Um get rid of them. Great. Delete. Amazing. The next step then, as you work with the AI, is to harden the test suite. So, the AI is quite good at reading your code and trying to infer what should be tested and what you're trying to accomplish with your code or if the code that it wrote uh usually. But, it doesn't catch everything, right? So, use the AI to help fill in some gaps. Hunt for missing cases. What are some additional very niche edge cases, but very real critical ones that didn't get captured in the test suite? Have the AI do a suite for that. Generate variants. Try to slightly adjust some of your code. Uh a very easy one is to change um some of your operators. I'd say probably the biggest mistake I I used to make when I was hand typing was I would uh just accidentally slip operators all the time. A good test will help you find that and and catch that quickly. And prove that it catches bugs. Have the AI try to break your app and see if there was a test for that or not, right? So, now that you're you've cleaned your test suite up a bit, make sure you're filling in any holes that remain. So, with that, maybe there's a couple more tests we want to add here. So, these are the two that we kept that we liked. Um you know, maybe there is a scenario in which there's a code that we didn't uh capture somewhere in in some dictionary or logic, and we want to just assert that it doesn't do anything. Okay? Maybe that's that's specific to our app. That's an edge case, and um you know, even just uh receiving nothing. And maybe that is a slightly different edge case from the uh missing uh promo code. So, uh either one that doesn't work anymore, maybe it's expired, or simply uh somehow didn't make it into your code. You know, does our code still um exit and return gracefully? So, maybe there's a couple edge cases we wanted to add. So, now that you've gone through all this, and uh you know, at first uh you're pretty involved, you're looking at what's happening. Uh here's your opportunity now to start building your system and capture all those things that you learned uh using those foundations, using your own taste, your own judgment, and start building your harness. So, we talked about some of these elements uh earlier today, but um we have, you know, uh a files rule files, you know, different system call and different things. I'm just using the word rule here uh as the file that's going to actually capture those conventions, um codify those practices. Um a really good way to do that is to put in that checklist. Right? We talked um Abby told us about that yesterday. That's kind of the practice he's using. Um you know, for my rule MD, I'm including the principles and then the exact checklist, and I have the AI check is it meeting all these gates? And if not, you know, let's review that. Let's uh refactor and and revise those tests such that those gates pass. Um and then the skills um I I really liked what how Avi described it earlier this morning. Um teach it how it works, what to do. Um definitely the thin wrapper really resonated with me. Um but the skill, you know, tell it to invoke it when you're ready to write uh your tests and then point it to your rules. Your rules should codify and capture everything that you want it to actually do with your tests. This is all really fantastic. Training your AI, making your system better, that's really incredible. Um but how do you know that it's working and that it's any good? Um we learned about we saw a a a a good example of evals and that's a really great way to see that the AI is doing what it's supposed to do ultimately in the way that you built it. But what if you want another option for that? So, how do we know that we've done all this, we've written our rules, our skills, and we actually know that our test suite is better? Uh turns out we can use some deterministic tools for that. Um that also resonated with me earlier a lot, this split between, you know, leveraging AI for its strengths and then making sure that you have uh kind of a backup in a deterministic sense. Um I uh as I was giving this talk a couple times as a dry run with some folks, uh I was thinking a lot about this and you know, we had gone through this uh harden step and as I was reading these bullets, I was like, I think there's a tool that does this for you that I explored uh some time ago and wanted to come back to it. And that tool is Mutant. I don't know if anyone has heard of this gem or uses it regularly. Uh it does literally that. Um it will go in and modify your code um I'll talk about it um and try to help you understand if you're, you know, uh the quality of your test suite. I had used this some time ago. I came back to the repo and uh the author had updated this read me around the same time I had originally drafted this talk back in January and I was like, "Oh, this author captured my whole thesis better than I could, right?" And I can write a lot of tests, but we don't, you know, they're not always meaningful tests, right? Um so, I really like that. That really resonated. Um so, how Mutant works, I'll just give a bit of an intro here. Is uh it spawns tiny mutations in your code, so it literally changes your app code, um and it will then run the test suite against that changed code, and it'll spit out a whole report for you, and it reports, you know, whether um the tests were affected by its changes or not. And it uh calls them uh mutant survivors, and it's up to you to then review that report or ultimately send that off to the AI. Review that report and inspect those survivors. And what Mutant does, it helps you catch not only missing tests, but it also helps you catch um it also fills in the fourth principle around good uh test and code design, cuz it also can help you find overcomplicated code, dead code you're not using. I just thought that was a really nice uh bonus. Um and then it's up to you to go ahead and just kill those survivors either by adding the gaps or simplifying your code. So, um a couple examples of what that looks like. These are the examples that the the gem provides just to keep it simple, but you may have a very, very simple method here, you know, adult uh 18 or over, right? That operator looks uh correct to me. That's great. And you might want to test is a 19-year-old an adult? Yeah. Is a a 17-year-old an adult? No. That makes sense. Awesome. We covered our code. That line of code ran in both those tests. Well, what did we not cover? Mutant will help you out. What if you just dropped that equals operator, uh which I have done this mistake many times. Um you run your two tests and they still pass. Amazing. So, we changed the code and the test didn't change and this is a definitely a kind of a mindset shift. Um the idea here is that our our boundary, our change, the thing that we actually changed didn't affect our test at all. So, this probably points to we're missing a test that will capture why we have that operator. Or you might say you don't need that operator, right? That could also be something that you decide. Um another example on the flip side, a very simple example kind of with our order class earlier, you know, we have a guard here. If there's any line items at all, go ahead and sum them up. Um if not, just return zero. Um we'll write a couple of tests that it does the math right. Amazing. Uh we'll also test the guard. Amazing. We covered our bases, both our conditions. Well, uh mutant will go ahead and be like, "I don't think you need all that." So, what if we just broke it down simplified? Still get the same sum. Amazing. Still get that zero state. Amazing. Our test never changed and we simplified the code. We go ahead and keep that simplification. Um so, that's a I think a really nice bonus of mutant is also kind of points out where you might have dead code and maybe there were a lot of good reasons why that code was like that, right? Uh your code base evolves and it changes over time and um this is a good practice to use to keep reviewing that any uh tech debt or our changes you might you want to make. Um so, uh mutant literally calls it in the report an evil mutant, which I thought was a a nice nod. Um no relation as far as I know to evil martians, but I thought that was cute. Um and there's a a few heuristics you can use as you're reviewing the test report that it generates around uh which path it takes. So, if it modified your boundaries, you flipping operators is kind of its favorite thing. It removed some side effects that you might be creating a method. It outright changed the return value that you're sending. Maybe you're trying to send nil and it tries to change something else. Or if it deleted conditional branches and your test still pass, you're probably missing tests to cover off on all those things. If, however, it got rid of defensive code, it got rid of speculative branches, it got rid of redundant expressions. Maybe the things you're always passing in are of the data type that you need. Or just there was code that it never reached, right? You can probably have an opportunity there to simplify your code. And and a lot of linters will help you find stuff like that, too. But if your tests are still passing and it literally deleted code, amazing. Delete the code. You don't need it. So, with that, the kind of last step of bolstering AI Harness is to introduce these deterministic tools. Mutant is a really excellent one that I've really loved exploring. But there are a lot of other tools that already exist that maybe you all use to varying degrees. I just listed a few here. They each kind of serve different purposes. So, there's simple cough or just, you know, did this line of code run when your test ran? Okay, you have coverage ostensibly. This uncovered gem helps you focus in on the on the code that you just changed, which is really great for the test pyramid idea that you you know maybe you just want to run the test for the code that you change at that time. Mutant we just talked about and then there is this other gem I found Coverband which is pretty cool. It's actually a little less about tests, but it's more about good um upkeep for your code base. It tests whether your code is used in production, which is very cool. And if it's not, go ahead and delete it and delete its tests, right? Our favorite part, deleting code. So, a lot of deterministic tools out there and AI makes it really easy. You can write hooks to make sure you run these things. You can have AI read these reports and and help you understand and you know, even make the modifications that are suggested. So, these are really great way to to add on some actual objective measures that you can see if your test suite is generally aligning to the principles and values that you have. Uh that is the end of my talk. Um so, thank you very much. Um yeah, thank you. >> [applause]