Fundamentals of Active Inference (Chapter 8, Session 37) August 18, 2026

Right. Greetings everyone. Welcome to the textbook reading group. We're reading fundamentals of active inference by Sanjie Nam Jooshi published this year. This is the second week that we're spending on chapter 8. Normally, uh, Daniel Freriedman would be sort of facilitating these meetings on Tuesdays, but I'll be taking over just for the next several weeks. Uh, most likely by the end of chapter 10, we may start seeing Daniel again. Um so with that said we're going to continue with chapter 8 which you know to be to be clear this is the the big sort of concluding chapter uh regarding all things continuous state space modelwise in active inference that we're introduced to in the textbook. uh late in later chapters in part three we will see some different extensions some different ideas different sorts of um you know variations of variational free energy and the like different kinds of inference algorithms but for the time being uh this is essentially the conclusion of everything regarding continuous statebased models and active inference um so as we know we've already gone over various things from perception and learning uh attention which I will lightly refer back to uh some of the things we talked about last week or that Daniel and I discussed last Tuesday. Um I'll try not to spend too much time on simply reiterating all those points but um given that chapter 8 is you know it it's it's accumulated we've sort of accumulated all over time all these different conceptual notions and equations regarding the different kinds of mechanisms that we're employing whenever we uh design these generative models as well as the generative environments uh in which we sort of pair with the agent models during simulations run over some number of times steps. Um, so it's good to sort of recount everything that we've gone through so far given what we're sort of building up towards by the end of this chapter. Um, a quick preview is that this time I'm just going to go through some of these slides that I've been building over the course of us teaching this material. Uh and then after that we'll spend roughly maybe 15 to 20 minutes on some sort of preliminary draft code uh that could be used to recreate one of the examples in the textbook. Um we've had you know different mentions in the past of code uh for this reading group uh over the past several weeks. I know that Daniel has been working uh very strongly on this uh repository dedicated to the textbook. uh the link to that has been shared in the past. Then we also have the code page in the kod uh or rather now superhuman pages for the textbook. Um, so you would find anything regarding code there and in future we will probably be able to put this code I'll be showing you today um into this sort of um overview repository page of the code that we've been accumulating so far. Uh I know that elsewhere uh Frasier's been writing some code regarding the basian thermostat example that were given in one of the chapters. Um so over time what we'd very much like to do is sort of continue building out code and um uh it would it would be great in part because uh we actually do have different pieces uh or draft versions of code from Sanjief himself that he did use to create a lot of the figures that we see in the textbook. So it'll be important to be able to refer to those going forward. They've just not been publicly released yet. Um and [clears throat] then secondly there's a you know there's a broader sort of uh landscape regarding current uh tooling not just in active inference but in various software libraries and packages for things ranging from deep learning and neural networks to reinforcement learning convolutional neural networks. the caris package basically there is no singular unified way of sort of doing computational programming uh that that that everyone has converged upon right so there are many different ways to approach implementing these things in code um so what's important going forward is to be able to move from translating these equations that we're coming into contact with in the textbook with the actual um you know means of computing these things of of moving from functions that are mathematically defined to functions that are computationally defined working with all these different variables that we've been seeing including uh defining their update rules. So we'll come back to all that uh near the end of this session. Um for the time being I will go just a bit over things that we sort of talked about last week. Um so I will try to move a little bit quickly and try to avoid any sort of uh slideshow karaoke here. Um, nonetheless, I wrote a lot of these slides to try and be as dense and informative as possible so that you can get a a pretty full picture on a single slide. Um first in chapter 8 we're introduced to this notion of uh triple estimation is the phrase that that Nemoshi uses uh in reference to updating not just our hidden states in the terms of perception and parameters in terms of learning as we saw in chapter 3 but also updating precision itself which is something that we've largely kept fixed in the past though we've referred to it and it becomes very important whenever it comes to how we sort of uh include estimation errors whenever Whenever we compute variational free energy and whenever we do our our updates, basically our prediction errors are not simply prediction errors. They are um they are weighed based upon their respective precision. So it's like this precision weighted uh prediction errors, right? um precision is going to increase or decrease the overall impact that errors our model makes uh actually become informative information for its future updates. basically being able to take into account error uh whenever making these different kinds of updates and should those errors be disregarded and continue with inference per usual we might say that our precision is very low in those cases whereas precision is very high will upweight those prediction errors and say oh this error is very significant to my model I should update more strongly um so it's important to be able to update all of these things and also sort of situate them conceptually within the active inference frameworks So as we know a lot of active inference uh sort of core principles and notions and mechanisms that are employed in these algorithms and the like are are largely coming out of um different work with neuroiming data and and um computational neuroscience biology. So here perception, attention and learning actually all happen at their own respective time scales or at least are conceived to be operating at their own respective time scales. Of course, the way you choose to implement it computationally may differ. Of course, we are also given some primary ways that we can go about setting them at different time scales. Um, but the the the primary conception here is that perception is what allows us to track the changes in hidden states very rapidly. So, this is something that might be conceived of happen happening at the scale of milliseconds. uh it might align more or less with the degree of uh you know sort of the resolution rate of recording that we get whenever we do different kinds of uh electroinsphilography or EEG and EMG uh data collection studies uh which do are usually sampled at a very fast frequency. Um meanwhile attention which is where we're updating our precisions or doing precision estimation. This is where they kind of uh act as a sort of amplification term a gain turn uh term or or like a knob on increasing or decreasing the signal received whenever we compute an error. Uh using those metaphors that I used earlier where turning up precision effectively means uh account for this error even more strongly in my update. take that into account. There might need to be a lot of updating that needs to be done because I have a very high precision uh regarding the estimation here uh in in how I should make that update and how I should update my precision uh term and then updating the precision term then impacts the inference elsewhere because as we know a lot of these things the way we're updating them is always with respect to kind of descending a free energy functional or variational free energy or finding the recomputing variational free energy with respect to each of these different um sort of parameters or these three different things that we'll call them for the moment hidden states parameters and precisions. Um so attention gets likened to neurotransmission and modulation. uh there there's various literature in different places where uh we haven't talked about things like policies and the like and planning that'll come up much more in chapters 9 and 10 and whenever we suddenly transition to discrete state spaces where a lot of these things we're talking about chapter 8 right now might figure somewhat metaphorically in those chapters but but will not be equivalent. Um but uh the the precision the these different precision terms and amplifications can be seen as sort of like um functioning in the way that uh acetylcholine might or or dopamine and dopamineergic pathways might. Um basically this sort of re-uptake process that then amplifies or diminishes different kinds of neuronal signals being passed uh in in neuronal networks in the brain. Then finally learning which is something that may happen over the course of days. It might happen over the course of years. That's how Sanjieve claims it. Um there's a lot of learning that sort of happens during periods of sleep. So setting it on the scale of days is perhaps reasonable. Although there's no necessity to uh specifically fix it there. Uh learning of course involves updating our parameters which are kind of like these additional variables that that that inform our understanding of how hidden states change uh over time or or how they relate to observation. And so being able to act not just use a likelihood function to go from I received an observation to updating my hidden state belief, but a parameter that actually modulates that that says, oh, this not only do I uh am I able to make that kind of update, but by updating my parameters, I can actually perhaps improve my understanding of the likelihood. I can improve my understanding of how observations translate into hidden states. So we can do that same thing for um estimating the flow or the change in in states over time. So um to be a bit more brief from here on out, we're going to have prediction errors for each of these respective things. Um the the agent should be able uh as before to invert its model in order to predict what the observation should have been. Uh it should be able to infer hidden states, should be able to infer or update or learn its parameters. And then it finally it should be able to update its precisions here. Um each of them thus having their own prediction rules. Right? So each of these things is sort of modulating one another and updating one another and acting as the scaffolding whenever we take uh the the free energy of any particular one of these uh with respect to the rest. Um and then with variational free energy here, this is um you know this this 1/2 is sort of like a dampening term and the rest is essentially just recognizing that we're always taking our predict our prediction errors and multiplying them by their respective uh precisions, right? So we're always getting these precision weighted prediction errors. And so this new equation for the model is just supposed to sort of capture what it is we're including now. in this current iteration uh of how we've designed this new model. Notice that we've not included uh everything that we saw in chapter 7 yet regarding action and exogenous forces. I think the Sanjieve here was really just trying to bring everything together just as far as we can make statements about how we could update in states parameters and precisions uh in the context of generalized filtering which is sort of the path that we've been following since chapter 6 roughly. Um, >> Andrew, equation 810, that last term, >> should that subscript for the lambda be a gamma or should it be a zeta? Oh, this here. Um, I think that it would be much more Yeah, I'm I'm in agreement with you, Kobas. I think that this should be a zeta. It would make this much more coherent. Um I know that I don't think Sanji's really employed gamma except in a few cases here that's sort of provisional given a particular example. So I think you might be right that should be a zeta just as we have the respective uh precision and prediction error for theta parameters and the rest. It's a good point. So I'll make a note of that to add it to our Rada. I think it's done a lot of good for this textbook that we've been able to point these things out. Of course, there are literally hundreds of equations, so I don't blame some G for making a typo occasionally. Um, any case, yeah, so we will come back to that. Um so our first algorithm that we're given in this chapter with respect to this triple estimation problem are updating our hidden states parameters and precisions through these conceptual notions of perception, learning and attention respectively. Um I'm just going to briefly go through this but the the general notion is in step one and and these algorithms are really valuable whenever we consider implementation in code right because essentially in a lot of computer code whenever we're running these sorts of models not just in active inference specifically but uh almost anywhere I mean you can you you will find effectively algorithms and loops and different nested loops and subprocesses and the like that are all sort of triggering and running within one another. uh in order to carry out all these processes and equations that we've been seeing uh in a very hands-off way to where there's no need for someone to sit there and run and execute each individual line of code repeatedly themselves. Right? So being able to understand these algorithms is what will allow us to sort of schematically apply these things and and only having to write the equations in code once and then applying them in the algorithm so that they repeat. So step one here, we would just initialize our different uh variables, meaning that we're going to just sort of say that they exist. We're going to establish them. We're not going to run them through a simulation just yet. Nothing's getting updated yet. Um you can sort of think like if we're in a basian paradigm, of course, we're going to need to establish some priors. Uh so that whenever we do our updates, we have priors to with which to update uh per basis theorem. as we saw way back when in chapter 2. Um so what we would need to do is for the um environment side the true exter or external state as it's called in this textbook uh would be uh established for this superscript is just referring to the initial time step of zero. So effectively the very first or initial true state uh the very first or initial true observation. Observations for us are always true. they are directly observed by the agent. Um prior means all of these uh and prior variances and priors all of these are effectively just for the uh generative agent and environment. Just um here they're they're very isomorphic with one another in that um you know each of them have a straight a state transition model. Each has an observation model uh or way that observations are generated by the the environment and how the agent then receives them into its observation model. That's in reference to what I said earlier about the agent should be able to invert its model to be able to predict what y should be or the probability of observation should be given its hidden states uh and parameters. uh the other way around when trying to predict in states and and parameters. And so from there we now that we've sort of initialized everything and then also this this initial variational free energy which presumably might be very high. It might not be, but if your agents prior that you just initialized are quite distinct and very far off from the true uh environment or at least they're not very good at predicting this initial data point, this initial observation that it receives, uh variational free energy will be very high. But fortunately, we will begin our simulation where our agent can begin updating its beliefs. So what happens here given these initial priors the initial data point initial data state uh initial external state and the rest uh whenever we initialize our simulation which is going to occur over some number of time steps that we set a time step effectively meaning run all of rules 3 through 11 until for each time step until we've exhausted all of our time steps. Here we're using zero indexing which can be slightly confusing for some who aren't used to it. It's very commonly used in computation in Python and and and the rest. Um but here we're simply saying at the first or the zeroith uh time step. First we'll have what we could together rule uh steps three and four together we'll call them the generative process step or the environment step. Um that will just take into account what was this initial external state that we defined use the environment's state transition function just simply its true parameter in the environment minus x in this case x would be that initial state. Um so we would take the the uh here the parameter x= 10 10 minus whatever we set this to is what this new state will then be. It's very simple very linear sort of equation of how state transitions occur. Then after that using that updated external state we would then generate an observation using this observation generation model within the environment which would just uh essentially send that through this generating function add some Gaussian noise omega as we've seen before and then three should probably be xar not so >> uh for which sorry corbus >> uh line three should probably be x star instead of just x. >> Oh, this here for the state transition function. >> Well, in the in the main algorithm, line three on the left. >> Oh, sure. Yeah. In fact, I think that could probably be applied there as well as here for the state transition function and the generating generating function because all these X's are respective to the environment. Um, that would probably >> Yeah. Yeah. Much agreed. I think that would make this a bit more coherent as well. Um I'm glad we're able to address these things in the textbook group once again. So um so yeah just bearing in mind that this X here is X asterisk the true uh hidden state of the environment or what again we can call the external state in this context. Um and then all of the X's here would also be true because this is the environment's equation. Um after this environment step where again we we so what have we done? we've simply updated the hidden state based upon what the previous hidden state was and the parameters were for the environment. Um and then next we can use that updated hidden state to then generate our new observation and that's it uh for the generative model or what we can call the agent step um all of these uh various other things will will play out. So we we do have these direct references to the other equations in the textbook which are also stored uh in the kod. Um but what we'll effectively do is remember that we've already initialized our first v coin of variational free energy. Now what we can do is calculate the gradient with respect to our expectation on hidden states parameters and precisions. Um following each of those equations that Sanjieve was nice to point out to us. Next, we will define our system flows. Um, now that we've started to get this initial information, we can start to identify uh the the flows that we've been going uh we've been referring to. Basically, not only what do we expect the hidden state to be uh with respect to five given the free energy gradient, but also what can we expect the flow of the hidden state to be or how do they sort of transition over time. This is with respect to all those temporal derivatives and and the and the like that we went over. Not necessarily Taylor series expansions in the sense of generalized coordinates. Something that's important here is that this algorithm despite us having talked about uh action and exogenous forces and talking about generalized coordinates and measurements and the rest uh those all came up I believe in chapter 6. Um they're not here. uh this this algorithm is really just focused on specifically this triple estimation problem or just being able to perceive, learn and attend uh in in the context of a generalized filtering algorithm. Right? So so we're not doing anything with a V or A for exogenous forces or actions. We're also not necessarily looking at these embedding orders that would be involved in generalized coordinates just yet. We're simply acknowledging that we're able to have expectations about a flow and [snorts] how it changes over time. And then furthermore, this is the part that's sort of at least in the context of this algorithm. This is the part that takes into account these differing time scales uh as parts of learning and attention where we're actually going to include something like a second derivative of parameters and precisions as well. Uh frequently we might assume in a um you know in a case that we're employing this and we think the environment is relatively static or at least it's it's uh its uh parameters and whatever the equivalent of its precisions are are relatively fixed. And so we might be able to set initial expectations of these to zero which then act as a kind of dampening term. Effectively, it's saying to the agent, don't learn too quickly. Don't update your attention too quickly. Um, you know, we're we're going to expect these to be relatively stable in the environment. So, no need to to drastically update these things to learn too quickly or otherwise. This kind of this slowing, dampening uh quality that we've added uh hypothetically. Then we'll update our hidden state belief. The agent will do that. It will effectively in fact do that for all of these things. Uh the hidden states, parameters, and precisions. Again, this is just aligning with with perception, learning, and attention as we've been saying, each with their own respective equations. Here, we're going to use Uler's method for the integration given that we're also including these uh temporal derivatives. And then at the end, we'll recalculate our prediction errors. And those prediction errors, as we saw previously, um these prediction errors, barring for this being a zeta rather than a gamma, um more than likely, um each of those is going to be uh included together, the prediction errors, the respective precisions, uh and all of that's going to be uh entailed in recalculating variational free energy. Now, what do we do from there? while at the next time step we'll do it all again and again and again until we run out of time time steps. So I wanted to go into this sort of example more deeply because whenever we're thinking about implementing these things uh programmatically we're very frequently going to see this general um sort of schematic being employed. We're going to initialize various things including priors given we're using a basian framework. um we'll we'll you know computationally for some things to work or if you want to sort of track your results in some sort of structure or a list or a dictionary or otherwise you might initialize those as well as a bunch of empty vectors or matrices or otherwise that you would then uh you can think of as your your record keeping book. You sort of initialize it at the start and then over the course of running this simulation over time steps you can update uh you know the different elements in those dictionaries or lists or otherwise to store all of your results and then that at the end that's how you're able to create these nice sorts of plots and the rest that we see in the in the textbook that's effectively what Sanjieve or anyone else or you know I have done in my research and the rest um so this initialization step is ultimately very short it's more of just being able to define precisely what it is you expect to keep and what you expect to employ. And then when you run a simulation, it's it it really does I mean we could abstract away from this and say it really does revolve around this sort of loop between the environment does something, the agent does something, the environment does something, the agent does something. Uh if you've defined your code well, then by the time you get to the simulation, it's effectively just two kind of nested functions on top of each other. the one for the environment, one for the agent. We'll come back to that though. Um, I didn't want to leave off the rest of this chapter, which is of course what we should be discussing today. And so, um, it's still it's still quite I don't want to say dense, but it's it's just there's a very there's a a lot of breadth in what we're covering here because now we're also learning about not just these algorithms and expanding them and moving up to algorithm 13, which is effectively algorithm 12, except we've added action and and exogenous forces and generalized coordinates and all these other components that uh, you know, we've left off for algorithm 12. So algorithm 13 is sort of the the end all be all penultimate algorithm of of chapter 13 uh excuse me of chapter 8. Algorithm 13 is um and and that's sort of this very all-encompassing algorithm that we'll turn to in a moment. Before we get to that, what can be useful for understanding the complexity or tracing out or illustrating the complexity of these different algorithms, but also linking them to our understandings and this once again comes uh in different areas from uh references to the the development or construction of things like neural networks um going you know very hard very far back uh prior to u you know what we know as modern deep learning now and we can think of different kinds of network diagrams and uh hot field networks and and the rest. Um sort of more traditional uh modeling to just the general notion of what is a network. I mean a network is a very a very broad notion that can be employed in many areas. You can have uh different kinds of graphical or network structures that relate to you know relating people to one another where each of these dots or nodes. It could be a person and these edges or lines connecting them can be some kind of expression or variable or otherwise it states some kind of relationship between them. Uh so two people can be friends in a network whereas two other people are uh family members and some people are family and friends and so however you like it depends on how you design that and more a little bit more directly here in reference to the the textbook this becomes very useful not just because now it's very frequent that we do think of neuronal populations as forming networks where axons and dindrites and the like connect them with one another and allow for different kinds of uh information trans uh transfer or what we might call message passing. Um, it also allows us to sort of define these very broad architectures in a way that we can see how they function, you know, and so so there's a lot of uh overlap with the kinds of schemes that we might see in blueprints uh being derived from work in engineering uh in different areas, right? Uh including using uh you know precise notation uh and being able to track all the different variables that are involved. So I've just included some information on sort of how to read these graphs. Um these graphs of course can be uh seen as hierarchical or we can have different layers. We've gone over uh models in the past that have multiple layers to them. We saw that a lot chapter 5 on hierarchical predictive coding layers. There are a lot of similarities between those models in chapter 5 with what we're seeing here in that um essentially you still have these ascending prediction errors from the lowest level and then you have these descending predictions from the from the higher level and then again we can whenever we instantiate that an algorithm we we really can see that it's just recomputing or updating those errors given the new prediction estimates and then meanwhile those errors impact VFE which then impacts what the next predictions will be. And so these things are really just in a sort of um in a sort of iterative uh somewhat birectional relationship with one another where at each time step these things are always sort of in conversation with one another. one picking up where the other left off and you know given that um perception is operating at the scale of milliseconds we can say this is happening so rapidly it's almost as if it's happening in simultaneity. Um that's sort of the the conception of how something like this works at least at the at the time scale of perception. Um in any case this that's claim still holds for the algorithm in general right so you update your your expectations or your predictions those impact your errors your errors then once again come back to impact your predictions um and so the there there just there are many ways that we could draw out these kinds of schematics and some of them can be more useful than others here this is one uh where we have not just a hierarchical network but it's also including um some of the generalized coordinates So we're actually including the embedding orders here, the zeroith meaning just sort of the your your your original um primary variable and then the next embedding order will be sort of that first term or that first derivative term uh essentially the getting into a flow of the original variable and then the flow of the flow we might say after that comes much more intricate from there. We could derive these in many ways. I will largely just given the time I will focus on just one aspect which is something very interesting. Um this is in reference to equations 8.54A to 8.54D. Um these are colorcoded and they're all describing you know at a particular layer in um in a hierarchical model uh how do we recomputee uh things like the precisions and the flows and and the like. Um, I'm putting this very briefly. Uh, it [snorts] probably deserves more attention, but it's something that could have consumed uh pro probably a couple of hours. Um, it's simply to say that the way that we can rewrite some of these equations without of course breaking them, but rather just kind of reformalizing them such that they still work equivalently, but allows us to view different kind of characteristics of these equations. that the the the color coding is allowing for this way that we can sort of view these as different messages being passed and combined. Uh whenever the computations are played out in these algorithms and whenever we have these sort of negative or positive messages uh they can be likened to the way that inhibitory and excitatory messages sort of pass between neurons. This is not you know this is not to um you know this is in relation to different kinds of empirical studies and of course a genuine ideal of these kinds of algorithms would be able to take into account genuine principles in the in the brain right so being able to have models actually account for things like excitation and inhibition whenever it comes to neuronal transmission and communication. Um, so, so a lot of these algorithms are sort of developed with that in mind and being able to recognize that those kinds of dynamics play out by being able to sort of map them. That's something that allows for this sort of ability to try and establish computational substrates for neurobiological phenomena, right? Um while there are those who are working in areas of machine learning and deep learning who are more focused on you know um uh performance of models where they might be more focused on uh getting a robot to behave exactly how you want it to regardless of how much or how closely it sort of it imitates uh the the the the human brain or malian brain or otherwise. Um there are other empirical studies and those who work in places like um you know computational or or precision psychiatry uh where where we're trying to actually understand how does uh how do neuronal populations in the human brain actually work such that it can lead to different kinds of clinical treatments or outcomes. In those cases we're not so much focused on achieving top performance, right? But we are doing things like sort of defining well what would something that's more normative look like versus something that's perhaps um symptomatic or pathological or otherwise and that allows for uh guiding things like how we approach people whether it be in a in a talk therapy setting or um in a uh you know pharmacological uh intervention plan. Um again algorithm 13 is sort of the end all beall of this chapter and I'll leave off with the slides here. Um it's just to say in some ways this is an extension of uh of algorithm 12. And so we still have many of the same things occurring as before. We're once again going to initialize everything. We initialize all of our our variables, our priors uh with the true initial state here. Now we have incorporated action and exogenous forces. So we're going to define uh exogenous forces uh alongside the initial state and the data or observations. Um so this is effectively just taking many things that we've learned up to this point. We don't actually get um you know Nam Jooshi doesn't give us a clear example of an application of this. More than likely it would be something that warrants its own research article uh as opposed to being something that could fit at the end of you know a chapter in a textbook. And this is a relative I would say this is probably the most advanced algorithm we get at least as far as um continuous state space models go throughout the entire textbook right um so it it's something that would be very interesting to implement surely been implemented or experimented with in different places um it will have a lot of baggage to it in the sense that there are many computations in fact I had to fit two slides in here in order to cover the entire thing Um but it's just to say that we will have things from um you know an agent who actually commits actions uh and predicts kind of flow over those actions. We will have um many of the update rules that the that everything entailed in the triple estimation problem with with perception learning and and uh attention are all sort of baked in here as it were. Um, we're not only doing that, but we're going to have all of the different embeddings entailed in generalized coordinates. So, we'll be predicting not just a variable, but also uh uh sort of its its speed and its um or or its velocity. And we'll be predicting its acceleration all the way up to however many embedding orders you choose to employ or potentially employ them differentially for different variables. um we will have our precision updates going on as well as the precision matrices which will then kind of capture these co-variances that we find uh in in different parts of the data. Um and all of this is can be hierarchical and so that all of these are happening within these many nested loops where sort of for each layer uh we will produce our embeddings update precisions uh or rather embed our precision matrices sorry we'll calculate free energy we'll have once again an environment step where you notice this one is uh quite small right once again we're just updating the hidden state and then for generating a new observation. Uh the only distinction from algorithm 12 is that now we've introduced action and exogenous forces. Right? So those are included but otherwise everything is essentially the same. Um and then we will right we will use our our parameters and and updated hidden state in order to generate a new observation from the environment. So these environments are kept very simple uh in many respects whenever we whenever we're like demonstrating these kinds of algorithms. Uh the main idea is not to oversimplify everything but for the sake of exposition and being able to see what an agent can do and be capable of and what sorts of update rules or learning rules or otherwise they might have. Um computationally we will often make the generative process something that we the experimenters do relatively understand well. Sometimes that will involve different kinds of simplifications or sometimes that might involve making a generative process that is uh simple as pop uh simple as possible but no simpler as it were meaning that we might make you know a generative process that has some kind of nonlinearity in it because we're uh you know producing some kind of agent model that should be able to be robust to nonlinear um uh hidden states from an environment such right so being able to right algorithms. Uh and this is this is this is the case where we we we might be looking between whether it be the things I was referring to earlier with empirical work in psychiatry or it be uh sort of looking at making you know so-called superhuman uh or or artificial intelligence where we're making something that's supposed to be highly performative whatever it is. Usually the generative process or environment you define uh is going to be something that you might expect the agent to be able to handle something you'd be able to demonstrate that they can handle such that if you think there are real empirical phenomena where an environment really does work something like that we might have an agent who's able to confront that kind of environment. So um so all that is to say for the sake of a textbook uh Sanjieve has left them very simple u but just complex enough to know that there are uh the the hidden state can actually change which is why we have a state transition function and then we have of course it emits observations which it's sort of been doing the whole time since chapter 2 um and seeing how an agent is able to sort of learn those functions and whatever kind of true parameters underly those functions and then being able in real time to be able to infer what the the new hidden state would be and in cases of action being able to sort of predict what the agent should do at any given moment in time. So that's what we build up to by the end of chapter 8. Um I will say uh this is something that uh first this is code that essentially reproduces uh example 8.1 in the textbook and so it uh one one recreates a couple of the figures in the textbook most uh especially um this one uh that's used I think it's figure 8 point uh I don't want to misspeak it might be figure 8.12 12. Um and in any case it is it is simply this um is simply this algorithm 12 uh sort of in action right so um this is code that Sanjieve wrote uh that's why that figure is the way it is is the genuine source of it uh that said it's um part of the the slow release of the code dedicated to the textbook uh by Num Jooshi himself self on GitHub or otherwise is in part because uh computation is changing so quickly. There's so many different kinds of APIs and frameworks and the rest that could be used in employing these algorithms. Um just a brief statement on that. It's relates to some things that uh you know we've heard others bring up in the textbook group before. We've seen libraries like SPM like uh PMDP um uh RX infer uh many other active inference.jl. I mean all of those have their own respective programming languages in the first place let alone the distinctions and how they write their algorithms. So that there unfortunately there is no current uh you know all-in-one way to employ active inference. There are many ways to sort of employ and conceive of employing uh active inference. Although uh ideally the the general sense would be that so long as we're able to trace it back to the true maths right or our our true equations here as it were um it's there that that becomes a kind of central focal point for for all of us to look at. Of course this is still the case in deep learning uh in other areas too. This is not just a symptom of active inference. we're living in a very fast time where where you know many people are using LLM to write uh or at least assist in writing uh code. there might be different kinds of standardizations that go forward and then there will other be other people who make deliberate decisions about finding ways of sort of coming together. And so that's something with the institute is um you know we look at these different libraries and we consider making our own that might uh not just compete with other libraries but rather be able to be compatible with or even be some sort of compilation of other libraries. And that's sort of the significance of being able to contribute to to open-source libraries where people are able to sort of borrow from one another's code and and kindly credit one another and and it lead to sort of a joint uh I I hesitate to use the word venture but uh you know sort of joint uh program going forward for people to be able to uh continue producing research that is reproducible and easy to reproduce. It shouldn't be something that you need to be um you know a very detailed expert and rewrite a lot of the code yourself based on words in a paper. It should be something that can release the code. Um so hopefully we'll be able to release this um ourselves. Uh we'll of course need to talk to Sanjie and make sure uh in that I've added a few things here for clarifying details. Um in future this will be put into the fundamentals KOD under the code uh page which we've seen is here it's just under discourse on the far left under code um there's some other things including reference to uh the RX infer group which is a library I mentioned earlier the fundamentals um repo that Daniel has been working on um a demo that I wrote for chapter 2 way back when I'm hoping to release some more uh code very soon and especially what I' I'd most like to do and I'm happy if uh going forward anyone would like to discuss something like this but I'm actually working on a library right now that will be for implementing algorithm 13. Uh essentially the idea is algorithm 13 kind of contains everything that we've seen in the textbook so far. um being able to write a clear library around this kind of algorithm and allowing for different options such as you know maybe I want to use algorithm 13 but I don't want it to be hierarchical right so just treat it as having one layer and run the rest of the algorithm per usual or oh I don't want it to have uh precision and attention updating I want those to be fixed right so that's something that you there would be some option for that but essentially the idea would be to have a library that allows for a lot of multifunctionality that that can kind of coales in this way that's well understood and and sort of auditable and and reviewable through uh um this textbook. So um that that's sort of something I'm working on. I'm seeking others right now to perhaps work on that. I know that there others uh Daniel and Frasier who are working on code as well. So um I also see a question in the chat. Is Sanjief's code accessible? Yeah, unfortunately um his uh he yeah so he has some draft material online at present I believe but I'm not sure if it's um onetoone with what we see in the textbook um of course because all of this is tied to MIT uh as his publisher for his textbook the the core thing is that while we you know don't want to disappoint anyone we do need to respect certain kinds of boundaries around copyright the rest and so to the extent that I am able to share this code right here in particular in the way that I am. That's sort of this is my attempt to give a preview of the sorts of things that one might expect when when trying to implement these algorithms in a way that I'm allowed to do. So so essentially the the short answer is no. That code is not currently available. Um uh once again if we had a library for algorithm 13 that would essentially cover almost everything in the textbook leading up to this aside from the distinctions that we see in predictive coding. Um but otherwise yeah um so that is unfortunate. The book was published not too long ago and there are many textbooks that are released where it's you know unfortunately five or 6 months or so before the code ends up being released. So hopefully we'll see more updates soon. Maybe Sanjie will end up writing code that um you know effectively makes my whole idea of a library obsolete. And frankly that would be quite nice because then we would have something ready to go, right? Um that that the author himself produced. But in any case um yeah I I will aim at least to be able to release this alone and I think it's uh I I do have the ability to share it, screen share it right now. Um so yeah this is recreating uh 8.1 we don't have action we don't have generalized coordinates um but what I've done is sort of tried to include some markdown cells this is just algorithm two uh 12 reiterated it's made a little bit more concise also just making it very clear you simply initialize everything initially and then you run your multi-time step simulation over however many time steps right um this is something else that people will confront. There are many comments and and and opinions around how one should define objects in a computational space here. I've simply tried to say if you ever see you know mu_x it's going to be in reference to mu of x or the expectation of the hidden state or the agent's belief or prediction about the current hidden state. Right? So kind of having this dictionary of you know because we can't uh directly uh render these these latte here uh in the in the code itself. Um so we have to write these sort of code names for them. Some people will opt for writing more descriptive names like they might say mean_x to make it clear it's a mean and and not get tripped up in like the you know these kind of classical Greek uh symbol names. uh but other people will try to stay precisely by the book and try to write uh their variables in a way that directly is is translatable to what you'd see in an equation um as one to one as possible. Um so I mean everything is here. We essentially have our our our precision terms and priors. We have our expectations. We have um you know our generating functions that we're going to see. We have our learning rates for hidden states and parameters and and precisions all in line with the triple estimation problem. Similarly, we'll have our prediction errors. Recall that there's a fourth one because we can compute uh a prediction error for observations. We just don't have uh a learning rate for observations, right? Because the observation ultimately is something that the agent gets to know, right? It gets to see the true observation from from time step to time step. Um and what Sanjieve did here was he used a library called Torch uh in relation to PyTorch. Um he's it's it's a a major deep learning and machine learning library. Uh but most simply the the primary things he simply does is being able to use um normal distributions from this library as well as some a nice utility for computing uh Jacobians. So that's whenever we get into looking at at flows and and temporal derivatives or a jacobian would be the matrix containing a sort of multivariate matrix containing uh the temporal der derivatives of all the hidden states that we're considering or whatever variable it is at the time. Essentially it's in relation to the sorts of copulus operations that we see uh being played out. So here I've copied um that environment function again and apologies I see how much time we had and I kind of figured it would go this way because of how dense the material in chapter 8 is. Um I'm going to sort of speed through this a little bit uh is just to give a sense of how one would implement these things sort of computationally. Right? So all we've done so far really is that this has just been a lot of informative text about uh variables and the like. what this what this script is going to be dedicated to doing. Um then we're going to import libraries, meaning we're just going to bring in tools to help us accomplish this task, some code that has to do with plotting um and settings for it. Um first thing we'll do after that where we're actually defining things and and sort of producing algorithms and the like here is we will define our generative process. Um here uh Sanjieve makes a nice simplified omega function. Uh this omega as you recall we see these noise terms uh very frequently we've been using Gaussian noise. So this is a function that simply given some set of inputs that you you choose um you can just generate white noise or Gaussian noise. And so he's going to do that uh in advance. And from there he sets up you know we've seen the the delta t uh this this is an interesting way of of using bins and effectively by the end what you end up with is how many time steps and sort of how many computations occur um uh with within a time step or within a given uh time horizon. Um but but essentially I mean you could you could do this in in different kinds of ways. Um, but it's just to be able to sort of lay out the the time here, how it's going to um how it's going to function, how it's computed, how you sort of lay out the the the landscape of time. I'm sorry that was so abstract, but uh you understand the general sense that we're going to say how many time steps and that we're going to be doing these discretizations that involve deltas. um object dimensions they don't these don't actually function in this code because we're not actually using uh this is a univariat uh example right so we only have one hidden state uh we're also uh using we're not using generalized coordinates so we don't have multiple embedding uh orders or we don't have um you know we don't we don't have different uh temporal terms here we're just working with uh you know the the in states and parameters and the like um we don't have like the tail tailor series expansion implementation. Um, but nonetheless, a lot of these things will look familiar so long as we're able to map back uh what these mean with the equations. So like what was this generating function fe and g and g per this dictionary? Um yeah, FE and GE are the uh generating functions and state transition functions in the true environment, right? Um so that's a way that we can actually like map these out. So these would effectively be equivalent to what we see here. The generating function and observation generation dynamics and the state transition and state dynamics. Um all those are getting defined right here in this single code cell for the environment. Um the rest generally plays out. We have um a nice environment step that just kind of puts these things together. These look quite similar to those equations. So we have the state transition function overall. Here we're adding some noise that we generated. Um we do the same thing to generate our observation. And here we're going to be using these structures where x and y actually uh are going to have everything from every every time step. So what we do is that uh you know we say for a particular time step the hidden state the true hidden state was this and the true observation was this um for the next one and the next one the next one that's that kind of record keeping uh that I referred to earlier. um by the end uh he simply here we'll go ahead and run this um I will share this out just plea please keep a a look out for our code page and we are absolutely happy to um have other contributors here and anyone who's been working on any co any kind of code or or projects that whether it be in a tenative state or otherwise we're happy whenever uh folks share it's a way that we can sort of collectively learn from one another and sort of improve our understandings and the like. Um so so we'd be happy to move towards featuring more code than just code that you know we've written uh just you know Fraser and Daniel and I uh or others or if anyone would like to collaborate on on something going forward. Um right so um we've effectively run out of time so I'll just say for the rest um you know by the end we get to see you know what the environment looks like. We'll do very similar things. We have our agent uh model uh in the math and then from there we'll define all of its respective equations as well as these derivatives for uh being able to capture things like flows of of hidden states. Um and we will compute all of our prediction error as per usual. um we will calculate the the gradient uh and we will calculate variational free energy. These are different ways. This is the the Jacobian that we mentioned earlier and it's essentially just a way for us to have some kind of vector or matrix or otherwise that sort of captures um all these changes uh these sort of first temporal derivatives or partial derivatives. So here we're for hidden states, right? We've seen this before. It's just a partial derivative um with respect to hidden state. Uh but then we can also apply all of our learning rates that we've seen before. Um here Sanjie wrote some specific functions that do a lot of this for you in one sweep. So f this is just calculate your prediction errors and calculate the respective energies or these energy terms that get fed into um into the variational free energy computation. We also have the flows. So all this is to say that we're just taking a lot of our our math up here to put it that way. Sorry. and and to uh you know produce these overarching functions that can do a lot of this for us here. That now that we've done that the rest of this is for recordeping and uh and being able to have these like empty places where we can store everything that's going to get captured over the course of the simulation so that later we can make plots and review results that played out over time. We'll initialize other things like our uh our learning rates weren't uh set up yet. set the initial precisions. Um the agent step, this is simply going to combine many other things. Here we're using Uler's method to do all of the updates. We're going to update our precisions. We're going to update our prediction errors. We're going to update variational free energy at the end. And then the whole simulation is simply this, right? because we've nicely kind of packaged everything up such that we simply have this nice environment step function that within it refers to many other functions and definitions and similarly we have an agent step function right so that's the the one to one equivalent to being able to have these uh environment and agent steps to go back and forth right over the course of however many time steps you set. So this line of code in in Python is essentially saying for each time step in uh the range of time steps that you supply. We can review that when this is that final figure that we see uh exemplifying 8.1. So hopefully there will be more discussion of code going forward. Um already by next week we will be discussing chapter 9 and entirely switching over to discrete state spaces. I know that there are different folks who've been contributing in this group and and participating who would like to see more on discrete state spaces. So, um you know, in some ways it's a shame that we don't have more time to go over the continuous stuff. Um but that said, for anyone who is interested in it, we're always happy to bring it up again if there are ever any questions as well as in the KOD if you would like to post uh questions under discourse. You can always add a question or add a question about a particular chapter and we will take some time to to review those questions and answer them and sometimes add different kinds of notes. See here this is already one uh that we'll get into partially observable settings like um so yeah please feel free to do that. Um and once again we will see about uh sort of seeing what we can do about releasing this code. I like it very much in that one, it's it's one to one with the textbook and one of the algorithms and two uh you know it's it's very uh self-consistent in the sense that you really only need to pull in a couple of very common libraries which is why I was able to put it in this browserbased Google Collab notebook uh and be able to run it in my browser because I didn't have to install any uh you know extraneous dependencies. simply we use these these four libraries and that's it. So the rest is just Sanjie's own sort of hard hard work and in doing all the translation writing the code everything gets defined for you there. Um okay we also do not have very much time for discussion. It's just given the way that the schedules have changed. Um but next week whenever we transition to talking about discrete state spaces uh I will be sure to allow for much more time for open discussion. we'll sort of be in this new world almost of of categorical distributions and and sort of uh matrix calculus that's being carried out uh you know quite differently from how we've been using all these normal distributions and Gaussians and the like. Um instead we'll be looking more at mutually exclusive probabilistic states and um uh you know the differences between different uh specific colors and and the like uh next week. So we look forward to seeing you then. And then per usual of course there will be a session uh hosted by Frasier Patterson this Friday. Uh so uh also happy to see you then as well. Um and oh and Geiocom I will be sure to ask Frasier about that. Uh he was trying to find the right place to post his code. He wasn't sure if he should put it uh you know into a particular repository or otherwise but but I was actually going to ask him about that today. So, uh, we'll we'll make sure that we can see what we can do about getting the the code here that I've showed as well as his Vasian thermostat, uh, example put out there. We would love to, you know, it's the the textbook of course is immense. It's over half a thousand pages in length. And, uh, and then, of course, there's the entire computational side that we've unfortunately only had so much time to get into. So, that's kind of what today was about. Um, in any case, uh, thank you all so much for attending and we will, uh, see you soon.