Interactive Skinner Box Simulator & Signal Detection Theory | BioniChaos

Okay, so we have another tool, a new tool on bony chaos.com. Everything we do is available on bykaos.com landing page should have a list of all the things we made in the last couple of years. So go check them out. Provide your feedback. We'll jump into this one. So I do have a PhD in this thing. So I should know a thing or two about it. So if a human actually ask a question, we'll more than happy to take it as a human in the loop. But uh we do have a demo mode that can actually explain what's on the screen. So the audio there is by notebookm or whatever it is called these days. I think it's just notebook Google notebook or something. Um, so essentially heavily relying on Gemini for this and listen to the explanation. It will actually uh if you go to the page bony.com/skinner box uh play the stand demo, it will actually reproduce the overview while also moving the sliders to show you around of what this tool is actually capable of. So, just listen to that and see how we go. >> Imagine staring at a slot machine. You know, uh, you know, the odds are against you, right? >> Yeah. You know, you're probably losing money. >> Exactly. But your hand still reaches out to pull that lever just one more time. >> It's so hard to resist. >> It really is. And today's deep dive is all about the invisible math powering that exact impulse. Oh, >> we're looking at how a piece of software actually lets you well engineer that behavior from scratch. >> It's wild. >> We are exploring the technical documentation and the visual interface of something called the Skinner box and signal detection theory simulator. >> Right. Which is Go ahead. >> I was just going to say it's a web-based fully interactive digital laboratory. It perfectly models animal behavior and decision-m right in your browser. >> It really is a staggering piece of technology. >> It is. I mean this isn't just some basic animation of a rat in a cage. We are looking at a live neurocomputational engine. It takes the fundamental laws of psychology and turns them into this interactive dashboard. You can literally watch the algorithms of habit formation update in real time. >> So our mission for you today is twofold. First, we're going to give you a clear overview of the century old psychological theories running under the hood here, >> the foundational stuff, >> right? And then crucially, we're going to focus on the simulators, controls, and visualizations. >> That's the fun part. >> It is. We want to see the exact sliders and buttons you use to manipulate a virtual mind. So, okay, let's unpack this because before we can twist the digital dials in the simulator, we really have to understand the physical box it's all based on. >> Yeah. You have to know where it started, >> right? And when I think of early behavioral psychology, my mind immediately goes to, you know, Pavlov and his dogs. >> Sure. The classic bell and food experiment. >> Exactly. You ring a bell, present the meat, and eventually the dog drools at the sound of the bell. But, and this is the key thing, the dog isn't choosing to drool. >> Right. It's an involuntary reflex. Biology just takes over. >> So, if we want to study habits, things we actually choose to do, how do you train a voluntary behavior? Like does the subject just do something by accident and you take it from there? >> Well, that is the exact problem that an American psychologist named BF Skinner wanted to solve back in 1930, >> right? >> He really wanted to move away from Pavlov's involuntary reflexes. He wanted to study voluntary motor behaviors. >> And to do that, he built upon Edward Thorndikeke's revised law of effect, >> which is pretty straightforward, right? >> Elegantly simple. Yeah. Basically, if an action in the presence of a stimulus leads to a reward, that action is more likely to happen again. >> Makes sense, >> right? And if an action leads to no reward or, you know, a punishment, the behavior faces extinction. It just gets suppressed. >> But Skinner needed a way to test this perfectly, right? Like a pristine environment. >> Exactly. Which led to the invention of the operant conditioning chamber, >> the famous Skinner box. >> The very same. Yeah. >> So, the physical box was designed for [clears throat] complete laboratory isolation. So, no outside distractions, >> none. It was acoustically dampened to block out external noise, had standardized ambient lighting, and inside there was a designated operandum, >> which is usually what? A lever. >> Yeah, a stainless steel lever for a rat or a pecking disc if you're using a pigeon. >> Got it. >> And the chamber was also equipped with automated food dispensers, visual lamps, acoustic speakers. Oh, and an electrified parallel steel rod floor. >> Wow. hook up to a shock generator. Right. >> Exactly. >> Well, I see your backdrop has actually transitioned to match this era. >> Yeah, I love this feature. >> It's very cool. You've got this dimly lit vintage 1930s laboratory behind you now with that heavy stainless steel operant chamber just sitting on a workbench. >> It definitely sets the mood. >> It paints a very distinct picture. But reading through the documentation, it seems like the physical box itself is almost secondary to how the data was gathered. That's a really good point >> because before this scientists used discrete maze trials. You put a rat in a maze, it finds the cheese, and the trial is over. >> Then you pick the rat up, reset the maze, and do it all again. >> Right. But Skinner completely abandoned that for something the documentation calls free operant rate tracking. >> Yes. And removing those artificial constraints of a maze trial, it changed the entire landscape of psychology. >> Because in a maze, you're really just measuring how fast an animal can solve your specific puzzle. Right. Exactly. It's a constrained linear event. >> It feels a lot like the evolution of video games to me. >> Oh, how so? >> Well, using a maze is like playing an old school rigid linear game level. You go from point A to point B, the level ends, you reset. >> But Skinner's Chamber is like moving to an open world sandbox game. >> I like that analogy. >> Yeah, you just place the subject in the environment, you don't constrain them at all, and you just let them hang out. But I guess I'm still trying to wrap my head around the analytics here. Why is watching an unconstrained rat press a lever better than timing a rat in a maze? >> Well, if we connect this to the bigger picture, think about what you are actually measuring. >> Okay. >> By leaving the subject unconstrained, the spontaneous frequency of those lever presses like how many times they autonomously decide to press it per minute, that serves as a pure realtime metric of their internal motivational state. >> Oh, I see. >> You aren't timing a puzzle anymore. You are continuously measuring the raw strength of their behavior over time. >> So the rate of response is just a direct window into their motivation. >> Exactly. >> Okay. So once you have that unconstrained environment and the subject learns that the lever equals food, >> you can start programming their behavior, >> right? You move from the physical hardware to the software, >> rules of the game, >> which the documentation calls schedules of reinforcement. >> Yes. Skinner and his colleague Charles Fer spent I mean thousands of hours mapping out how the timing of rewards dictates behavior >> because you don't need to reward every single action to maintain a habit do you? >> Not at all. In fact, it's better if you don't. >> Yeah. >> These schedules fundamentally fall into two categories. You have ratio which depends on the sheer number of responses and interval which depends on the passage of time. >> Let's look at the ratio side first. There's continuous reinforcement which the simulator labels as FR1. So every single time the rat presses the lever, it gets a food pellet. But you would think giving a reward every single time would create the absolute strongest habit. But the documentation says that's not true. Why wouldn't a guaranteed reward be the best? >> It comes down to expectation and what we call extinction resistance. >> Extinction resistance. >> Yeah. Continuous reinforcement creates incredibly fast learning. The rat figures out the game immediately, >> but the moment the food stops coming, that expectation is broken instantly. The rat just quits trying almost immediately. >> All because it's like, "Hey, the machine's broken. I'm out." >> Exactly. So, to build a resilient habit, you have to introduce scarcity. For instance, a fixed ratio schedule or FRN that delivers a reward strictly after a specific number of responses. >> So, an FR5 schedule means every fifth press gets a reward. >> Correct. And this creates high uniform response rates. But the psychology here gets fascinating because these bursts of action are always punctuated by a post-reinforcement pause. >> Right? The rat gets the pellet, eats it, and then just stops working for a minute. Why do they pause if they know exactly how to get more food? >> Because the brain is constantly doing this metabolic and mental costbenefit analysis. >> Okay? >> If you're on an F FR50 schedule, you know you have to press that lever 50 times to get one pellet. When you finally get it, your brain recognizes you are back at the absolute bottom of the mountain. >> That makes total sense. >> Yeah. The length of the pause is directly proportional to the ratio. The steeper the mountain, the longer the brain forces a rest period before starting the climb again. >> Wow. Now, contrast that with the interval schedules where the reward is based on time. >> Right? >> A fixed interval schedule like FI 10 seconds means the very first lever press after 10 seconds has passed gets the reward. And the documentation mentions this creates a famous FI scallop shape on the graph. >> The scallop shape is so cool because it reveals the subject's internal timing mechanism. >> How so? >> Well, when the reward is dispensed, the clock resets. The animal knows that pressing the lever right away is just a waste of calories. >> So, they pause. >> They pause. But as their internal clock senses that the 10-second mark is drawing near, they begin pressing frantically, >> hoping to catch it the exact second it's ready. Exactly. So the response rate accelerates in a curve a scallop shape right up to the moment of reward. >> Which brings us to the variable schedules. Variable interval changes the time unpredictably giving you a steady pace. >> Right? >> But the variable ratio schedule VRN is where things get kind of terrifying. The reward comes after an unpredictable average number of responses. Two presses then 20 then five. >> Yep. This is the exact math running in commercial slot machines, isn't it? It's just a relentless pause-free trap. >> It is. The unpredictable nature bypasses that metabolic calculus we just talked about. >> But you never know when it's going to hit, >> right? You never know if the very next pole is the jackpot, so you never experience that postreinforcement pause. >> You just keep pulling. >> Unpredictable rewards create ironclad habits. They are incredibly difficult to extinguish. If you move a subject from a variable ratio schedule to an extinction schedule, meaning no more rewards ever, they will just keep going and going. >> Though the text does note they often hit an extinction burst first. They get incredibly frustrated that their slot machine is broken, frantically varying their behavior, sometimes even attacking the lever before giving up. >> It's very relatable behavior, honestly. >> Seriously. But wait, how were scientists in the 1930s perfectly mapping out scallops and postreinforcement pauses without computers? >> Oh, they used a brilliantly engineered device called the Jer brand's mechanical cumulative recorder. >> Right. The paper strip. >> Yeah. Imagine a physical roll of paper unspooling horizontally at a constant speed and a mechanical inking pen rests on the paper. >> Okay. >> Every time the rat presses the lever, the pen steps upward just a fraction of an inch. So, because the paper moves steadily forward and the pen steps upward with each press, the slope of the ink line mathematically represents the real-time rate of response. >> Exactly. A steep slope means frantic pressing. A flat horizontal line means a pause. >> That's genius. >> The physical mechanics created a perfect mathematical derivative of the behavior. And when the pen hit the top edge of the paper, it rapidly reset to the bottom and just continued charting. >> It's incredible. Okay, getting an animal to press a lever for food is one thing, but a massive part of this simulator is designed to test what the subject can actually perceive in their environment, >> right? Which requires a totally different psychological framework. >> This is where the simulation layers in signal detection theory. >> Yes. Pioneered by David Green and John Sweatz. This theory merges psychopysics with operant behavior. >> Okay. Psychophysics. We are moving away from simple reward loops to ask >> how accurately is this brain processing sensory information. >> The chamber shifts to a going to go procedure. Sometimes it plays background sensory noise designated as N. Other times it presents a target signal superimposed on that noise designated as S plus N. >> So this could be like a tiny change in the pitch of a tone or a slight shift in the brightness of a light. >> Exactly. >> But the simulator has to do something incredibly difficult here. It has to mathematically decouple the physical ability to hear the tone from the sheer motivation to press the lever. >> Right? So in the underlying math, the pure physical sensory capacity is called D prime. >> D prime. Got it. >> That is the physiological discriminability index. It is completely uncorrupted by motivation. It just measures how far apart the noise and the target signal are in the animal's nervous system. >> Okay. But then there's a secondary variable, right? >> Yes. little C which is the decision criterion. This is the internal psychological threshold for actually taking action. >> And that decision criterion is highly influenced by motivation. Let me run an analogy by you for this. >> Go for it. >> Specifically for what the text calls a liberal bias, which happens when the criterion drops below zero. >> Right? >> Imagine you're waiting for an incredibly important text message. Maybe you're waiting to hear if you got a job and you were desperate. >> We've all been there. You'll start hallucinating your phone vibrating in your pocket. You check an empty screen constantly. >> Oh, absolutely. >> When the text actually arrives, you catch it instantly. So, you have a high hit rate. >> But because you check your phone 50 times beforehand, you have a massive amount of false alarms. Your physical ability to feel vibrations your D prime didn't get any better. But your motivation was so high that you shifted your decision criterion. You adopted a liberal bias. That analogy perfectly illustrates the math. A starved rat in the chamber, meaning it has extremely high motivation for the food pellet, shows that exact same liberal bias. >> It'll press the lever at the slightest hint of a sound. >> Yes, securing a high hit rate, but suffering tons of false alarms. No, consider the inverse. >> A rat that is completely satiated, or maybe a rat that has been repeatedly shocked for guessing wrong, they adopt a conservative criterion. So they wait until they are absolutely 100% certain they hear the signal before pressing the lever. >> Exactly. They successfully avoid all false alarms but at the severe cost of high miss rates. >> So they miss the signal even when it's physically present because their internal threshold for acting is so guarded. >> Precisely. >> Wow. Okay. We have all this dense history in math. Now we reach the focal point of the deep dive. How does this simulator actually translate these formulas into a visual dashboard? >> It's a really striking user interface. >> It is. It operates in a sleek dark mode. The documentation provides a thumbnail showing trial 57 currently running using a 1.4 kHz signal plus noise cue. >> The interface is cleanly divided into functional panes. On the left side, the physical apparatus is represented virtually. >> You see this beautifully rendered virtual white rat rat positioned near a lever and a food hopper. And you can watch visual cues light up dynamically on screen. >> Yeah, it's very responsive. There's a speaker icon emitting sound waves and these indicator lamps labeled SD and S delta. What do those specific symbols mean in this context? >> So those dictate the rules of the current trial. SD is the discriminative stimulus. Okay. >> It's the green light indicating that pressing the lever will yield a reward. Sla is the red light, >> meaning no reward, >> right? It signals that a reward is currently unavailable. So any effort expended is wasted. Right below those lamps, the chamber clearly labels the electrified grid floor, which is currently sitting ominously at zero volts AC. >> Good thing for the rat. Seriously, if we look over to the middle pane, that old physical paper Drew Brands recorder has been digitized into a live cumulative response recorder graph. >> And right below that is a real time graph of the signal detection distributions >> visualizing the math we just discussed. >> Exactly. You see two intersecting Gaussian distributions, which is just the math term for a standard bell curve, >> right? >> The blue bell curve represents the background noise and the green bell curve represents the signal plus noise. >> And there's a red line cutting through them. >> Yeah. Cutting vertically right through the intersection is a red dotted line. That indicates the subject's current decision criterion threshold, little C. >> In that same panel, the UI reads out a metric called beta, currently sitting at 1.57. What is beta measuring? >> Beta is just another mathematical way scientists measure the strictness of that decision criterion. >> Okay. >> It represents the ratio of the signal likelihood to the noise likelihood at that exact threshold line. >> So a beta above one means the rat is leaning conservative. >> Yep. And below one means they are leaning liberal. Here's where it gets really interesting though. The virtual rat on the screen is not a pre-rendered animation. >> No, it's not. >> Doesn't have pre-programmed loops. It is an autonomous agent driven by a live neurocomputational engine using what was it? Rascorlo Wagner learning kinetics. >> Yeah, Rascorlo Wagner. The simulator is literally calculating associative learning frame by frame. >> That's insane. >> It computes the strength of the association between a Q and a reward. It factors in the physical salience of the stimulus like how bright the Q lamp is and the magnitude of the reward like how caloric the virtual pellet is. >> Wow. But what makes this a true digital brain is that the engine is calculating temporal difference reward prediction errors. >> Okay, I definitely need an explanation for that one. What is a reward prediction error and why does the simulator care? >> Think of it as the mathematical gap between expectation and reality. >> Okay, >> let's say you expect a $5 tip and you get $20. >> I'd be pretty happy, >> right? That massive gap is a positive prediction error. In a real mamalian brain, specifically deep in the basil ganglia, that exact prediction error triggers a phasic spike of dopamine. >> So dopamine isn't just a pleasure chemical. >> No, it is a learning signal. It physically rewires the brain to repeat whatever action led to that unexpected surplus. >> Unbelievable. >> This simulator is mathematically computing that exact dopamine transient and mapping it directly onto the physical actions of the digital rat. It is modeling the literal chemistry of learning. And because this is an interactive sandbox, you aren't just watching this math play out. You are the one controlling the environment. >> Yes. Moving over to the right side of the dashboard, you have a vast array of telemetry control. >> You're given sliders to manipulate both the biological state of the digital rat and the physical parameters of the box. >> Playing God essentially. >> Basically, in the screenshot, the food deprivation slider labeled D is set at 77%. The grid voltage is at zero. Signal intensity is at 2.70 and the criterion shift is perfectly balanced at 0.00. >> And below all of that, you have your manual lab intervention buttons. >> Yeah, with a single click, you can instantly drop a meat pellet, deliver a grid shock, trigger the queue, or even manually press the lever yourself to see how the system reacts. >> And as you slide these values up and down, the metrics box updates instantly showing the shifting hit rate and false alarm rate. So those metrics are just a direct mathematical output of the cognitive state you've just engineered. >> Absolutely. >> Let me test the limits of this system, >> right? >> If I go into the simulator and just crank that food deprivation slider to 100%. Does the virtual rat just start hammering the lever at the speed of light? >> Well, the computational model is grounded in biological reality. >> So no infinite speed. >> No. Pushing deprivation to 100% would eventually simulate physical exhaustion. >> Oh wow. The response rate would spike dramatically at first as the animal adopts an extremely liberal bias and takes massive risks, but it's still bound by the simulated physics of a mamalian body. >> Okay, what if I introduce chaos? What if I grab the grid voltage slider, crank it up, and start delivering random shocks using the manual button? How does the math handle that? >> What's fascinating here is that introducing unexpected pain completely shifts the neural pathways. You aren't just tweaking a variable. You are triggering the agent's simulated basilateral amydala, >> the fear center. >> Yes, you are initiating a full-blown avoidance simulation. >> I see your backdrop is actually shifted again, this time to a glowing futuristic display of neon green, and blue telemetry data, mirroring the UI we're talking about. >> It feels very appropriate for the second. So, what does the digital rat do in my shot? It transitions entirely away from seeking food. The algorithm moves from operant positive reinforcement into operant negative reinforcement. >> So, it just tries to escape. >> Yeah. You'd watch the virtual rat exhibit species specific defense reactions. It might freeze in place or retreat rapidly to a corner of the chamber. >> And the graphs >> in the middle pane, that cumulative response recorder graphing the lever presses would instantly flatline, reflecting a massive behavioral suppression. Because survival overrides reward, >> the decision algorithms instantly rewrite themselves. >> When you look at this simulator, you realize you aren't just looking at the behavior of a virtual rat. You're looking at the foundational architecture of decision-m. >> Absolutely. >> Recognizing how easily a few UI sliders can perfectly map out a habit helps you realize how digital schedules of reinforcement are operating all around us. >> They are everywhere. That variable ratio schedule we discussed earlier, the slot machine math, >> that exact algorithmic structure is running right now in the apps on your phone. >> Yeah, it determines the unpredictable timing of your notifications to maximize your engagement, >> right? bypassing your mental costbenefit analysis just to keep you pulling the digital lever. >> The underlying mathematics of behavior are universally applicable regardless of the species inside the box. >> Which brings me to a final thought I want to leave you with. Drawn from a highly provocative detail at the very end of the technical documentation. >> Oh, the interconnected modules. >> Yes. It lists a few interconnected modules offered by the creators of this software. One of them is a neuro feedback laboratory >> specifically designed for live cortical mapping in operant brain wave modulation. >> It's intense to think about. >> Think about the implications of that for a second. >> If a software engine can perfectly model a digital rat's basil ganglia and we can condition its behavior by pulling a few visual sliders, what happens when humans are the subjects in the box, >> right? How close are we to using these exact same interfaces, these exact same sliders to condition and modulate our own brain waves in real time? >> It forces us to ask who is really engineering the habit and who is simply responding to the schedule of reinforcement. >> Exactly. Well, thank you for joining us on this deep dive into the algorithms of behavior. Keep questioning the schedules operating around you and keep exploring. >> Guess I'm not sure what happened there in the middle. It kind of didn't turn off the sound for the simulation. Otherwise, it did pretty well. Yes. So, interior different reinforcement schedules. It should reset the counters. Don't know if it does. And we can press deliver. Release a pallet. That seemed to work. I was jumping around for I think now it's scared or something. It's probably heard a sound and jumped. And yeah, it can deliver a shock. Yes. So it jumps on a shock. Uh there's not enough voltage. The grid voltage is currently 65. I can put it through 120 volts. It should do this animation electrified greed uh greed floor. Yeah, it will indicate the voltage level there. Yeah, we have to do more testing. Do more testing to this. Yes, we can go test the tool yourself. Yeah, that's what meant to do. Yeah, when the sound is on. Ah, that choke is only instantaneous. That's why um yeah, in the demo mode it does it continuously. So, I'm not sure if there's any issues with it. reinforcement extinction. Okay, there might be problem with the text. Yes, when you deliver a Q, the correct queue, it will respond by going to the lever. It will press the lay but that will be a well cuz we're doing it manually would it be counted counted would be affecting the prime didn't seem to should be some sort of reset those parameters option I think that's the default that we start save. Okay. So, we have this continuous CRF FR minus one. That's a standard reinforcement schedule. Why is this empirical D prime different from the actual D prime? I have no idea. Yeah, we'll have to do more testing to this probably next time. So, let me know if you tried it yourself or if you did behavioral studies before yourself. It will be really good to get some feedback and I'll see you next time. Bye.