Blastoff Rails 2026 - Don't Read the Code: Trusting AI With Your Rails Codebased by Avi Flombaum

5, 4, 3, 2, 1. We have lift off. >> Yesterday I I said that cliche of like this wouldn't be possible without you. Um but it is very literal. And uh I think it's it's very literal for Avi as well. Because uh in 2018 I was living in New York City. And I was I'd kind of been trying to get into to coding for a couple years. Um I dabbled. But I just never And I knew about boot camps, but uh I couldn't afford them. I was working as a Pilates instructor at the time. Uh different [clears throat] story. But uh So yeah, that was a career transition. Uh so I actually found out I think I was like on some mailing list or something for Flatiron. Even though I couldn't afford it. And then I got an email one day that they were starting a new program where you could defer tuition. They would loan you a laptop for cuz I had a Chromebook which couldn't code on anyway. So they would loan you a laptop for the the program. And um you didn't have to pay back your tuition until you got a job. And I was like you know, maybe I could do that. So I applied and I got into the very first class of uh the this program Flatiron School uh Access Labs uh in Brooklyn. And uh went through that. It was like 6 months of full-time classes. And uh Avi would stop by the the building in Dumbo every once in a while. And it was super fun. I made like friends for life. And um Yeah, I it's it's not a exaggeration to say that uh Avi is a very big reason why I'm a programmer today and why this conference exists. Um really funny side story is that uh when they deferred your tuition um you don't have to pay it back till you get a job and they will only take uh I think it was like capped at like 10% of every paycheck. They want you to pay that back. Um well, I uh I got an internship and then I actually left and I got my first job in Mexico City. And uh I was working uh on a Mexican uh Ruby on Rails engineering team and my salary was a Mexican salary and it was like $27,000 a year. And all of my classmates had gotten like six-figure $100,000 jobs in New York City and I sent the form to the the company that did the deferred tuition for uh Flatiron Labs. And uh I put 27,000 on there and they emailed me back and they're like "Did you mean to put 27,000?" Uh this we this looked wrong so we weren't sure if you just put it in wrong. Uh and I was like, "No, that's my that's my salary." And they're like "Okay. Uh So, yeah, that was that was fun. But uh thank you so much to Avi for uh starting that school and also for being here to speak. I think that's such a cool kind of full circle for for me selfishly. Uh so, let's give a warm uh Blastoff welcome to Avi for five four three two one blast off. >> [applause] >> Hey, uh everyone, how's it going? I'm Avi. Uh thanks Travis. Um Yeah, I started Flatiron School many many many years ago. Uh taught a lot of people how to code. I've been programming my whole life. Um also been in Ruby on Rails um basically since it came out. Um and uh yeah, uh I changed my talk a little bit because uh if you know, you're live today, you guys know that the nature of our field is changing every single day. So, don't be surprised. >> [laughter] >> Um but uh what I do right now is I work for a pretty small venture capital firm. Um, I'm the only technical person uh, on the firm and I both incubate companies that we spin out and then I also work with a lot of our portfolio companies training their teams on, you know, both their product engineering teams and their also non-technical teams on how to leverage AI. Um, so, uh, you know, actually Carina is one of my huge inspirations um, and uh, taught me so much about leveraging AI. Um, yeah, it's crazy like seeing how much our jobs have changed um, to the point where like, yeah, I mean, I'm sure it's Carina like, I don't write code. I barely don't read code really. Uh, I really mostly focus on like the AI portion on the gent of harness in the loops and that's kind of what I wanted to talk about. Um, so, what I started finding was that I'd go into these portfolio companies and I would work with their teams and then they would start, you know, vibe coding and they would end up with like four apps in Next.js and six apps in Python and like, you know, just you know, then they'd come to me and be like, hey, none of this works. And that's a problem. So, I had to kind of figure out a way to standardize how they could all vibe code effectively. Um, and I built a Rails generator like an app generator. Um, you know, but instead of like just kind of like a Rails template you know, that spins up an app, I had to do something a lot more because I really wanted to ship an entire AI harness with conventions inside of the app so that I knew that when they were vibe coding, it would be like when I was vibe coding to as much as possible. So, uh, I built this um, you know, as I said, it's kind of a massive uh, um, Rails app. Oh, it's actually massive Ruby app, right? That basically kicks off, you know, you just tell it to generate an app. Um, and has a whole bunch of flags of what you'd want to do, but uh it's going to start generating that app and by the end of it what you would end up with is um an app, you know, a standard kind of a a Rails app looks like this, right? Uh you know, the standard kind of stuff. Um where is it? There we go. Um yeah, just it's it's a Rails edge like, you know, I don't mind them run running on edge. Um but the important thing is that the app that was generated basically ships with a whole bunch of skills baked into it. Um so that I'm not expecting them to install all the right plugins and marketplaces, learn how to use Code Act or Claude. I just want that all inside the app, right? So that when they fire up Claude or mostly Claude, um everything is basically packaged within it, right? Local skills over like trying to share marketplaces and stuff. Um there is a way to fan out and create updates and things like that, but that was kind of like what made my life working with them a lot simpler. Um and the sort of part of the app that's cool is that once you've generated that app, um you can then load up Claude, not know anything about how to vibe code, and basically tell it to add a feature to the app. And I just want to show you what the app looks like first out of the box, right? Um this is kind of what you get. Um you can sign in. All right, so it has like, you know, standard kind of you know, admin section, right? So like kind of it comes with all the kind of things you'd imagine, like a whole payment system, whatever you want. Um and then it also comes with a whole demo of all the features the app has, right? So like, "Hey, here's how the app uses Turbo Streams." Right? Um it has a whole like UI kit, right? You know, things like that. Um, all how background jobs works, right? All that kind of stuff. It has a whole AI system built into it. Um, All right, let's well, yeah, with a whole bunch of different things of how to use AI, you know, even to the point of like integrating like, you know, vector stores and things like that with Postgres. So, it's a very, you know, not It's like It's a starter app, but like with a ton built into it. But again, the features that are pre-baked isn't what's cool. What's cool is how you can add something to this app, right? So, um, yeah, let's try to one-shot a feature we want to add to this app, and then I'll give the rest of the talk and we'll see what it does. Anyone want to What do you want to add to this? All right, let's say Cool. Right, so the idea here is what I want to know is with just that minimal prompt, what's it going to do, right? How's it going to use Stimulus? How's it going to use Turbo? How's it going to use Tailwind, right? What's it going to think of that you know, very minimal prompt? Um, and uh, it's probably going to try to ask me some questions first. Right? Let's see. Huh, shouldn't have done that. All right, I'm going to actually restart it cuz that was weird. Ah, demos live demos, right? There we go. Okay. So, that's one of the problems, right? Was that, uh, you know, I don't want it to use that internal cloud code brainstorming skill. I have my own planning skill that actually wraps compound engineering Kieran's planning skill, right? Um it's supposed to know that that prompt should have invoked that workflow and it didn't and that's a huge problem. Because again, I can't if I'm, you know, having 100 people at all these portfolio companies I work with vibe code, they're not going to know that like, "Hey, just be explicit. Type in {slash} RG plan, you know, right?" Um so how do you enforce that kind of stuff, right? How did I make it so that that was enforcing it? Because once it started going into RG plan, right? It that knew to kind of go into the right workflow, right? Um okay, what's the scope of it? Let's do a single shared board, um app facing. Yeah, sure. Uh native. Yeah, and okay, cool. All right, awesome. So I just made some choices and now it's going to continue on its doing it. I just want to make sure that it actually gets the coding and then we'll kind of go on. But again, like that's those were the kinds of things I was seeing, right? Now the next thing it does, right? It's going to start reading convention files, right? I very much care about how it's going to write the code. Uh I don't care how smart models get. You know, one of the big points that I kind of have found is that um I think that there's basically like um I want the models to write the code. I want my harness to define my process, right? And I I really want certain things to be deterministic and I want certain things to be non-deterministic, right? I want the model to shine on the non-deterministic things and then I want to make sure that my harness can really define the determine the deterministic processes of what it's going to do, right? That's kind of the way I think about my AI workflow is separating those into two buckets. What am I just going to let the model vibe out on because it's awesome at, and then where am I going to put guardrails and how am I going to enforce them, right? So, I very much like, you know, the conventions are have to be deterministic, right? Like, I still love RSpec, but now I use fixtures, I never did before, right? Um I use uh I use stimulus, I like hotwire, I I I use service objects, you know, a whole bunch of stuff, right? Um you know, how you use uh secrets and what the models can access is really important. Like model, you know, not you have to tell the model you can't edit production data somewhere, right? Um things like even just like even like when you're working with Git work trees, um you can let the model figure out how to make a Git work tree every single time, but it's going to burn tokens. Just give it a script for how to make do that, right? Um and then with the model the model's going to create artifacts like planning documentation, um you know, reports, whatever it is, those also should be convention-oriented and standardized, right? Um however, what the plans actually are, the sprints, the tasks it's building out, I don't care about that stuff. I don't even [snorts] review it. I trust it. I talk to it about it a lot, right? What the architecture of the code is going to look like, I don't care because if it's following my harness, it should be all right. It should be good. Um and then how it's doing it, I also don't care about, right? How many like the testing, you know, the turns it's taking, it's debugging it, I really just trust it to refactor and kind of just go on its own process. I don't even have to give it like, you know, design UI UX at this point. Mostly because a lot of the stuff I work on is like internal-facing, so it's not high-end design work, but it has enough uh to get like really good interface patterns, right, also at this point. Um so that's kind of the way I split kind of what I'm putting my time into. Um you know, and when I'm trying to work. Let me just make sure that Claude is going. Okay, great. It did. Okay, user creatable full crudsher card fields card UI. Uh model Right, but I mean yeah, this kind of stuff uh cards only. Yeah, whatever. Okay, cool. Hopefully it'll go. Um all right. Um so, one I hope you guys all know that the error of subsidized tokens is coming to an end. Um uh I very much believe in like what I call tokenomics. Um you have to pay attention to like where your model is running and churning on tokens cuz it's going to get very expensive very fast. Um so, one thing I think is is that basically your harness is going to help you make sure that your tokenomics and how you know, you or your developers or teams are spending tokens is going to be efficient. Um and uh the problem though is as you go down the model layer and you go to cheaper models, right? The worse they're going to be at code conventions, the worse they're going to be at parsing context from a big legacy code base and things like that. So, then again, the harness becomes way way more important, right? Um and basically the way that the app the harness works that I've built is that it kind of everything falls into like five buckets, right? I have like the instructions like my Claude MD. Um I have the skills I've written and or wrapped. And then I have docs and the docs fall into two patterns. I have convention docs which are like abstract code conventions about like turbo patterns I like. But then there's like a convention of how the AI system in this application works or how like the I don't know um you know, Kanban board system in this app works that would never go to a different application, right? So, system docs, convention docs, and then you a whole bunch of hooks and guards to basically make sure that the model is not going off the rails and not using something. Um and then pre-written scripts to make sure that workflows um are right. Um so, I'll show you kind of kind of like some of the examples, right? One, you know, when you're Claude MD'ing or Agent MD'ing, you actually want those to be as small as possible. Um and you just want to keep them really thin and just basically try to get it to piggyback onto like the main harness, right? And my harness basically starts with it needs to read conventions, it needs to read it need it needs to read docs, and then it needs to go through planning and working, and that's it. Like, I don't want it doing anything else, right? I don't want it like take off all of the rest of the things like brainstorming mode and, you know, planning mode and No, just always do that. Follow that workflow, and that's it. Um and then uh you know, the hooks really work. So, like I use a a session start hook to basically at the beginning of the session have it, you know, again, just reinforce this because in each session, as long as it starts right, it's generally going to go right. And like as you could see in the other session, the second I saw it load to the super power brainstorming, I was like, "What? That's not right." Cool. So, it's it's it's it's running. It's running the plan. I'm going to tell it. Um cuz it might not know to continue working. All right. Um okay, so when I build uh skills, um basically, I want my skills to be very thin. They themselves should not define any convention or any coding pattern. They always are going to point to reference files, right? And kind of always is kind of fan out, right? So, I tell it, here's where all the conventions are and here's a quick index of what conventions exist and go read them and when to read them. Um and then uh I always I love there's so many awesome skills out there. Um compound engineering is like my favorite thing in the world, but I basically build wrapper skills around it, right? There are parts of it I like, parts of it I don't want to use and how I want to use it. Um and you can basically build a little wrapper skill on top of that to say like, okay, here is the Rails planning skill I have and here's what you're going to do and when you hand off and what you hand off to compound engineering's planning skill. I found that to be very, very effective to like really split off wrapper skills and kind of piggyback them off. Um and then uh in terms of like reading conventions, you know, what I found is that basically like the models seem to pay attention to the beginning of the context window and the end and ignore the middle. So, in every turn, the model might think it read the conventions like 40 turns ago and be like, I'm not going to reread that file, I already read it, but it will have not or it will have forgotten and not care. So, I make sure that at every single turn, it's rereading the convention files, right? Um otherwise it's going to go off the rails, like it's going to take the path of least resistance. Um so, that's the first kind of thing is to make sure that it follows these every time. Um And as I said, like there are system docs. So, like I love Ruby LLM, it's my AI wrapper, but I've actually wrapped that in another object because I have to delegate how to use Ruby LLM in certain kind of ways. Um and that's kind of what I mean by like a system doc, right? Here are the service objects in this app, right? Um when I'm writing a skill for a convention, um I basically split it into three parts. Uh These are These are the reference docs for like my hotwire skills, right? The first is um and also all these files need to be around less than 200 lines of code. Models will get just start throwing stuff out after that. So, when I define like my stimulus patterns, it's going to be 200 lines. I have to be very, very disciplined about what I put in there, or I have to spin it out into a separate file. Um the first is kind of like a really brief reasoning. You give it one good and one bad example of what to do, and then you give it a checklist at the end. And that seems to work really, really well for the model, right? It understands conceptually what it's trying to do, has an example of good and bad, and then it can verify whether followed that. Um that is basically what all of my code conventions docs look like, which are all what the skills uh anytime it's trying to write stimulus, it's going to invoke the convention stimulus skill, which is going to read my that doc, and that's how I know that when it writes stimulus, it's going to use, you know, data values and not introspect on the on the DOM randomly. Um and then same thing with like uh deterministic workflows. Um the agent should not run RSpec natively. I don't want it to do that. I want it to run agent RSpec, right? Agent RSpec does two things. One is it runs a full test suite in parallel and puts out a JSON file that is basically here's what the tests were before, so that after they can compare to what they are after it did all that mess, right? That's going to help it. Um and then also just don't start a dev server. It's always running already. >> [laughter] >> Um so just things like that, you know. Um, and then also yeah, just closing like um, trying to uh, not let it use certain things I don't want it to is also really important in the harness. So I'm going to check it on Claude. Perfect. Awesome. Okay. Um, you know, work trees is what was a really interesting thing because when you use a Git work tree and all the Git work tree skills I've ever seen, they don't work with Rails out of the box, right? Because there are certain files you need that are going to be Git ignored in your work tree. Something as simple as that. And watching Claude spin on like creating a work tree so that I can work on two things in the same app in the same like, you know, code space at once, but skip all the credential files so that it can't even run the test suite was driving me nuts, right? Just like package that in like a bash script and don't let the model try to figure out how to use Git work trees correctly in a Rails app. Like it's that simple and you'll get so many fewer turns and token usage. Like it just runs faster. Um, so the thing is the models do get smarter and better. And one thing I look at a lot is given that I have this kind of hard limit that I only want X amount of conventions and I only want them to be X amount of lines of code, like 200. I have to make choices. I have to test. Is the harness working? Is this convention even needed, right? So what I built is an eval system into the generator. Um, and the eval system is basically uh, it takes a long time to run. It's like 40 minutes, but I give it a sample Rails app with a lot of actual code in it already. So not like a little Rails app because I need to have it needs to have enough code so that the model can get lost in the source code like a legacy app. So like here is the source app for the eval system. It has a ton of controllers already in it. Some are well-written, some are not well-written, right? Like you can't be a green field, you're going to be perfect. Um And then I basically spin up two Docker environments. One of them has that source code with the AI harness of the generator in it, and the other one doesn't. Right? So, control and the treatment. They're totally sandboxed, and then I use Claude P, the command line, to send it the exact same scenario prompt to both of them to build a feature to test each convention out. And this is what those look like. So, I create something like this. Right? Like uh we're going to create a feature called the warm intro. We're testing the front-end hotwire convention. And this is the kind of concept, right? Build a warm intro request feature model with a form, turbo stream submission. Here's the source app, right? Um this YAML file is what the eval system uses. So, now basically what's going to happen is it's going to fire Claude in SDK mode with that prompt to both apps, one with the harness, one without. And then I can compare what happened, right? And I have expectations of what I want. And in the end of that, what I get back is a report of how my harness is doing, right? What is being followed, what is not. Right? So, in uh the last run, which cost me 15 bucks, um uh the treatment got 28 out of 33 corrects. The control only got 21 out of 33 corrects, right? And I kind of break them into three areas, right? Where my chain, the harness of conventions, and how the workflow worked where I'm not sure and uh where it uh basically seems like it didn't need to have to have happened. And these reports kind of go into great detail of exactly where it broke down. So like this is a really classic example, right? That feature required like a you know, a modal interface. I don't know how many times I've seen the model recreate the exact same DOM for a modal in every single view. There is a shared partial. There is a helper file. Like just use that. Don't rewrite it. That is a classic failure in a large code base, right? And I've like that just failed, right? Like without the harness, it was going to rewrite it. There's the proof. >> [snorts] >> Um so that's how I can kind of evaluate how the conventions I'm writing and changing evolve over time and where they're still needed, right? And then I can also see the workflow. So for example, um also when this system run, it does it does it does three runs for each of them. Um this is basically testing that idea of if I said build a Kanban board, did you actually run RG plan and RG work? And here's where you can kind of really see the tokenomics of it, right? In uh the treatment, there were 122 tool calls for the front end hotwire, right? It read seven conventions, edited 35. It took 10 minutes to do to write its first line of code, right? Um and and costing me 372 in API token pricing, the control um started writing code faster, edited way more files, and ended up costing more, right? Like that's it. Like that's how I I evaluate the efficacy of this as it as models change and the landscape changes over time. Um so yeah, that eval system has been really helpful in kind of understanding like what models we're getting good at, what conventions are, you know, going to follow and kind of giving me like a north star, a way to just, you know, normalize like what I'm doing and how I'm doing it. Um at least I'd like, you know, does the chain help? Um is the model getting better or is it like need does it need tightening? Um Yeah, the you know, the the things I think are not changing every day are that skills need to be really thin. I don't care how big context windows get. There there's context rod is just so such a problem and anything I can do to keep the individual turn, whether I'm using parallel sub agents or managed agents, really condensed and only give it the right context, I really want to invest in constantly. I think we're always going to have to do that, especially as we can't just send everything to Opus 48 or Codex 55 or Fable and we have to go to cheaper models. Um checklists really help. Um Uh again, rereading the code, rereading the context after every turn really helps. Um make a lot of small plans, not big ones. Uh When I when I'm using AI, I always explicitly invoke skills. I never try to I would never say build a Kanban board. I'd say use RG plan to build a Kanban board. Um I think the like the guardrails and the delegating the scripts, I just don't get why people don't do that. It's so much cheaper and faster. Um and then yeah, you can really build a lot of measuring and evals. Like this stuff is easy to tell Claude to spin up and like understand you know, evaluate your sessions, right? Like get some telemetry on how your AI is working. Um I found that very very very very helpful. So, let's see how our Kanban board is working. Still going. Good. But, look. Oh, not happy about the board CSS, but we'll talk about that later. Um okay, I guess that is my time. How long has this been spinning for? A while. 20 minutes, okay. Well, 18. All right, any questions? Yeah. It's just a namespace for Rails generator. Yeah, so it's it's RG plan, like Rails generate plan. No, it's just a namespace of all the skills in the harness is like RG for Rails generator. Um kind of like compound engineering is CE, you know? Yeah. No problem. Yeah, um we can walk through it. I mean, at a high level it is uh Oh, yeah. So, the eval system and the architecture around it. Um the eval system is a folder within the the generator, the Ruby app that generates these, right? That basically um has uh you know, Ruby files that explain how to do things, like how to spin up the Docker environments. And it has um the Where is it? Right, the patterns I want to test. Um and then, so I basically write YAML for everything I'm trying to to test. So like we were looking at in when it's evaluating whether or not it did that shared model correctly, here's how it's going to know, right? Um, did the view code use render shared model or not, right? That's the file to look at, right? So that basically after the the the eval system runs and I have basically two Now I've the app with two source codes. I then have Claude run a third time and say, "Look at app one and see if there is a shared model or not or you know, in this file basically, right? So for every single layer I build one of those. Um, and then yeah, I mean it's basically yeah, I I don't know what else to say about it. It's two Docker environments. It's isolated and it's using this Claude SDK. Um, yeah, and then it creates a whole bunch of like forensics, right? Like I have um, you know, I'm basically logging everything it did in it so I can look at it to kind of improve the eval system and see what was going and things like that. See, still going. Jesus. I should have moved it to like low effort, you know? Yeah. Ah, doing too good. All right. Well, yeah. Nothing. I mean, I only interact in prompts. So like even when I want to write a convention, um, you know, I might like draft I use a I I mean, I dictate a lot, but you know, um, yeah, like for these for like figuring out like what was going to be sort of like my um you know, kind of like uh my scorecards. Yeah, I just worked with Claude a little bit to like figure out like okay, we need a YAML file that's going to define that after the eval runs how you're going to be able to introspect on the code and see, you know, whether or not we consider it good or bad, right? Um so it kind of came up with this and that's that. Um yeah, I mean I trust it more and more, you know. Um what was the another cool thing on this? Yeah, and then this scenario and patterns going to test, but yeah, and you know, I can keep on adding stuff to it. It's you know, easy, cheap. All right. It's never taken this long for It's just a common board. What are you doing? It better be awesome. Um all right. Let's see. I'm sure it works already. Uh so that server's running. Yeah. All right, let's see. I bet you it's there already. Just so that we can Okay, I'm back to app. All right, nope. Okay. I promise you it'll get there. >> [laughter] >> All right. Cool. All right, thanks guys. I guess that's it. >> [applause] [applause] >> That's That's my bad.