Fundamentals of Active Inference (Chapter 8, Session 41) September 1, 2026

Greetings again everyone. Thank you for joining the active inference institutees uh textbook reading group. We're still covering fundamentals of active inference released this year uh by MIT press authored by Sanjie Namoshi. Um we will be continuing chapter 9 today. Uh and I want to say at the outset here, we are strongly considering we're basically on the verge of making a final decision now. Uh actually spending 3 weeks on chapter 9 rather than just the two as we have been for the preceding chapters. Um we're also considering spending two weeks on chapter 10. Um that of course would kind of extend uh the current projections for uh all of our sessions we'll be having. And then of course we have the entirety of part three to get through. Um it's far enough into the future that we're going to spend some more time deliberating on that. And for now just really making sure that we can try and wrap up chapters 9 followed by 10 on discrete statesbased models as sort of holistically as we can given we've only now been confronted with discrete statesbased models. So I just wanted to make you all aware of that. And then also just a couple other brief things. Uh first uh the institute will be having its uh annual symposium symposium coming up pretty soon. Um it'll be uh a really interesting time just as it has been since 2021. um all of the uh presentations that are given you know is essentially an opportunity for for researchers for students or otherwise to sort of present their own respective work uh with you know in the form of a presentation. We also accept a short pre-recorded videos that can be shown uh if someone is not necessarily able to attend or would just like to sort of share a more brief update of what they've been working on. Um people are also allowed to uh ask for having uh workshops or discussion panels. Um those are all things that can be um submitted whenever following through the links. It's just activeinference.institute/projects/simpymposium institute slash projects or even better a simple Google search for when spelled correctly active inference institute symposium 2026 will take you right there as the first result um so yes uh we of course you can also register to simply attend it is fully online through uh YouTube actually it's hosted entirely on YouTube um and then there are zoom meetings sort of going on in the background that are then shared during the YouTube streaming uh recording so that uh you know everything can happen in real time and uh we do welcome people to uh have presentations have multiple co-authors. It allows for potential discussion uh between different researchers. Um and uh we're we're always kind of tending towards this applied active inference theme where the the sense is where are people actually applying active inference or where are they applying it to whether it be some concrete use case in robotics or considerations for for clinical health um uh biology all the way down to those who are still working more deeply on the theoretical mathematics and physics behind much of it. So uh we of course will have uh a keynote session from from uh Dr. Carl Fristen as well as uh Dr. Chris Fields uh who have been kind enough to have uh keynotes talks at previous symposium uh years as well. So um yeah, hope to see you there. And then a second very brief uh update is I've been working on uh not public yet but um I've been working on this library that um I I have some collaborators uh who are going to help me pick up the pace a bit but we're making a Python library specifically for uh what we see in chapter 8 uh algorithm 13 in this textbook for active generalized filtering. uh sort of putting everything together that we've learned on continuous state space models just with respect to the the textbook itself. So being able to do things like setting your embedding order for generalized coordinates, being able to specify if you want one uh layer in your model or potentially more. Um being able to sort of clamp or hold particular variables in the model as opposed to those which are free and can be inferred over time. uh being able to apply learning to learn the parameters uh having a kind of coherent API so that people can sort of supply their own generate generating functions and definitions of the parameters into their own model. Um so yeah just to quickly share that um oh what is the submission deadline for the symposium? Um we should have that the presenter submission deadline is the 31st of October. Yeah. So so there's quite some time that that's one of the benefits of hosting this online is that there's a there's a lot less sort of um you know overhead whenever it comes to sort of planning things. Of course we don't have to worry about booking any larger rooms or things like that for hosting a physical conference. Um, nonetheless, it it is def definitely worthwhile to present sooner, uh, excuse me, apply to present sooner rather than later, if only because it gives you the opportunity to request a particular time slot. Uh, of course, you know, people uh, globally attend the symposium and so we have sessions being given at uh, almost all hours. Uh, it's very worthwhile to make sure that you can get um, a slot if you do want to present uh, in real time. So um of course we also um seeking you know for those who just want to participate once again there is no deadline for just participation which simply means you will be attending um you're given access to everything um and then uh secondly Gio I see your message yeah so I I've not made this available yet uh it will be on my GitHub uh for the time being we might uh sort of move it transfer it over to the institute at some point. Um, but yeah, it it will eventually be available for the time being on my GitHub. I have some other interesting things for anyone who wants to have a look. It's just uh the first letter of my first name followed by my last name. Uh, but yeah, all kinds of fun stuff. Uh, you know, I've and this is also actually relevant. I'm glad we're here. Um this is uh a tutorial I gave on PIMMDP uh which is exactly the kinds of things that we're doing here on chapter 9. Um this includes like an entire uh multi- aent simulation that you can run in your browser. Bear in mind that this is about 2 years old. Um but it it effectively implements many of the things that we're seeing in chapter 9 from uh you know variational free energy min minimization. Uh it also gets to some of the things in chapter 10 where we see parameter learning. Here we're just going to be learning the A matrix, but it kind of spells out some ways that we um define Oh, it also exhibits factorial depth, which is another thing we'll see in chapter 10, which is essentially the multivaried um analog to um uh what we saw in continuous state spaces whenever we have multiple hidden states that we're inferring at once or receiving multiple kinds of observations per time step and the like. So, um, yeah, and I can certainly share that. Um, let's pull it up one more time. So, yeah, I I will be frank, this repository has kind of grown uh over time. It contains many things. So, it's all sort of just historically updated in this readme. Um, a lot of this is based on a tutorial that I gave at University of Pennsylvania at their international conference uh for um computational social science two years ago. And so this was actually this all stems from uh a threehour tutorial I gave. Um here I'll I'll drop the the link in the chat here to that. Um but again you can look me up on GitHub apache. Um but yeah so so it even has these slides that I presented. Um there's effectively [laughter] uh 89 uh slides of material here. Uh no there's an appendage. So 81 uh you know and this is sort of this is me trying to keep everyone updated not just on how you can produce POMDP models in uh PIMDP uh but also moving up to sort of a framework for doing multi- aent modeling um and so this is like a partic I I essentially just took a classic uh already existing uh multi- aent simulation uh that that comes out computational social science and is more in the realm of what we call agent-based modeling which I would say that active inference is fully compatible with that. Uh the phrase agent-based modeling first intuitively just makes sense here and then secondly that phrase is expressly used in various pieces of the literature. It follows essentially all of the same principles. And so it kind of starts out with what agent-based modeling is, but then gets into uh things like how do we construct our agents in terms of their matrices. It gets into uh you know a lot of the material that we saw in the 2022 textbook. Uh this is you can see why I'm so partial to to making all of these uh uh tutorial slides and the rest. I've spent a lot of time doing it in the past. Um so yeah uh we can just see you know how sort of the ontology of how these things are all described like we're not always seeing Sanjieve sticking to uh the the the kind of traditional terminology of a lot of these things like whenever we we haven't I don't think we've seen the phrase outcome modality yet or maybe we have uh by the end of chapter 9 uh or into chapter 10 probably would be more relevant when we talk about factorial depth There are hidden state factors which comprise uh each factor comprises multiple discrete hidden states that like you could have a weather hidden state factor which contains hidden states raining and not raining as the two mutually exclusive hidden states within that factor. Um just many many sort of things like that. Um and then kind of showing like how are each of these matrices constructed. Now, all of that said and and to bring us even closer to the the topic for today, um you know, I'm I'm going to get into those things again, but sticking a little bit more closely to the equations and figures from the textbook that we're allowed to share um on online in this way with with the public. So, um so now I'll get into chapter nine. Um oh, it one more brief bit. Um, I also presented this tutorial uh sort of a very short redux of what I did at University of Pennsylvania uh with the institute not too long after I did the talk at UPEN. So that is available. It's active inference model stream 15 Andrew Pache active agents an active inference approach to agent-based modeling in the social sciences. uh so that that can be found quite quickly, but it's essentially just on the institute's YouTube channel, which is of course where we have uh all of our other videos that we've had in the past, including those symposium uh videos that I mentioned earlier. So, um yeah, feel free to look at that. Unfortunately, it was it was a bit rushed relative to the the sort of longer 3-hour version. Uh but, uh hopefully it kind of gets at some things. I kind of get a bit tongue-tied on some of the the the way that some things are sort of defined. Um, so hopefully chapter nine here with the the the our textbook reading group right now is kind of going to be more comprehensive relative to what I shared before. Um, but I still definitely recommend for those who are because I know that there are people in this group specifically who are interested in the discrete state case uh of of modeling. And so uh yeah, I would suggest paying a lot of attention specifically to chapters 9 and 10 in this textbook and then feel free to have a look at that uh tutorial link that I shared in the past. Um also worthwhile once again to you know look at the um the new documentation for PIMDP. You do not only have to do PI use PIMDP to do to do discrete state space active inference as you all know but they just have these kind of nice um uh like a gallery of different notebooks that they've put together that's sort of things that you can view in your browser. Um you know they PIMDP has kind of taken its own direction. It still unfortunately is missing a couple of more general functions that sort of should be there. I believe they're scoped in after I talked to one of the developers. Of course, these things take time. They've made a lot of advances in other areas. Uh I would say PIMDP is very well suited right now to be the top contender versus SPM for doing discrete state space active inference potentially even faster at doing it in some cases. So if those uh those who are concerned with things like scaling, you know, doing many uh multi- aent models and the like uh very worthwhile to look at PIMDP because they've very nicely implemented what's called jacks uh into the sort of API and framework for doing active inference. It essentially just means that you're better you're you're more equipped to being able to do things like GPU related acceleration and the like to to be able to run these things uh much faster than what you could with like a baseline Python and NumPy. Um and then nonetheless I I do have a fork of PIMDP that does have some of those functions that I referred I I was kind of sort of referring to. Um they're especially relevant in chapters um 10 whenever we get into preference learning and habit learning um as well as uh uh updating expected policy precision gamma which comes up in planning. So these are all just sort of essential things that have been in SPM for you know over a decade. Uh whereas they're still sort of missing from PIMDP because I think PIMDP has you know you only have so much time whenever you're contributing to these repositories and and they've had their own particular priorities for where they're going with it. Um but yeah, my my fork of it has uh a lot of these additional functions for uh just monitoring, being able to actually look at expected utility and information gain and all these other things that we see later in chapter 9. I think I've spent too much time on on all of this stuff. So, I'm going to just focus on on our uh textbook if we uh don't mind bearing with the in absurd number of files on my desktop on this computer. So, um so yeah, chapter nine. So, we recall that uh at least as far as I made it with you all last week. Um what we looked at were a lot of analoges between the continuous state space case and the discrete state space case. Um, put very quickly, we're still working with the equivalence of of um, you know, a posterior distribution, a prior distribution, uh, model evidence, and a uh, likelihood distribution. And these this is the way that we're able to relate states to observations. We've been seeing this since effectively chapters 1 and two in the textbook. Um but now we're uh conceiving of those as categorical distributions in these discrete models where we have discrete states that are mutually exclusive. And so we're no longer using these normal distributions where a value is necessarily greater than or less than another value. Rather now we're just thinking of you know um uh reigning versus not raining. Those are necessarily mutually exclusive. there are two different sort of competing so to speak hidden states uh within say a single hidden state factor. We can make some kind of uh posterior inference about what uh the probabilities of it raining versus not raining right now are given some observations that we receive uh and and then and then using some form of bases theorem. uh but of course you as we'll see we'll start getting into variational free energy given the intractability of always trying to directly apply basian exact inference uh in this way. So, so recall that uh we're using D to represent this vector your prior beliefs about hidden states or sometimes called prior over initial hidden states just to really make the point that actually this is only used at the very first time step of a simulation. This is these are your sort of ultimate priors that you began with because as soon as we move into the next time step we will shift to using something else. um that being our B matrix I'll get to in a moment. Um but we also recall for the A matrix the likelihood we're similarly relating uh observations in this categorical probability distribution observations conditioned on states just as we did in the continuous case. Just now swap out Y for O and X for S. We're effectively talking about the same thing. Can use the exact same terminology. We're just talking about different distributions and ways of calculating them. Um given that we're we're we've switched to continuous values to discrete mutually exclusive states. Um so so we have this kind of matrix where each row is going to be your observations and each column are going to be your states. Um this this is a very row major perspective on how to construct these things. I just mentioned that because if anyone's using mat lab you'll notice that many default options in mat lab are actually going to be column major meaning many operations that you carry out are going to be focused on doing different things based starting with the column followed by the row whereas in Python and many other languages uh things will be row major and so um just worth noting that um the B matrix is what we we implement in order to allow for a sense of temporality in our models. It allows for the idea uh you know sort of baking into the model itself. And this is equation 9.16 of bases theorem again but now we've switched out our priors with our B matrix. Right? This is effectively once we once we leave that first time step um you see in 9.18A this is the this is sort of the the inference rule we we use at the first time step but then 9.18b is what we use at every single subsequent time step because we're no longer looking at initial states once we move past the first time step right we've already updated our posterior and then can use that posterior as a new prior in order to make our next update. And then that allows for using the B matrix which is essentially just more granular. It says not only do we have a sense of where hidden states are at that we can sort of plug into our B matrix which then itself we can plug into the broader inference equation. But it also carries given now that we made a prediction about the hidden state from that first time step. Can we use that and have a sense of temporality, a sense in the model of how hidden states change over time? Some assumption that they can in fact change over time uh into the model itself to use that as additional information for making yet another prediction. Right? So a model does not have to have a B uh as it were. or it could just be a a you know the D and the A and then you're kind of just in this more classical sense of a basian uh uh basian setup. uh you can think of that as you know the same way that we do a sort of classification model in in more standard machine learning or deep learning methods right you just given some observations what is the the hidden state uh given the result of a COVID test what is the probability that the person who took the test actually has COVID right then you end up with all these sorts of things like positive and negative um you excuse me, true and false positive and negative rates and all the rest. Um, that's the kind of domain that we're working within whenever we look at a static model. But whenever we move to this dynamic model where sure we started with the D, but now we've moved on to using the B and implementing this temporality, it's there. Now we we can move to saying like, oh, this is a model that expects that the the environment could change, right? Um so this is this is much you're not fixed with some assumption that the hidden state will always stay the same in the static case. Um so I I think that I've made that relatively clear. And then similarly with that the matrix uh construction as we saw for A and D where um columns uh and and rows are distinct. Here we're looking at uh the the rows the way that Sanjieve has depicted them here is states will be for uh by row and then states at the next column uh at the next time step will be represented by columns. Um it again it's a bit tricky because it's going to depend on if you're you're focused on the sort of row or column major operations. uh it's worth sort of just bearing with this um as we continue and just because essentially for chapter nine um as long as you have a nice library like SPM in mat lab or pydp or some other such thing um it you know your inference algorithms are already sort of predefined for you a lot of the things we'll see in the textbook are already defined for you so what does that mean that means the most essential thing when setting up your own simulation will be precisely to know how to set up your A and your B and your D and the rest uh whenever we extend beyond what we're looking at presently in these earlier sections of chapter 9. So I just want to iterate that because I've um you know I've been communicating with different people and I I think it's you know brilliant the kinds of ideas that people come up with but if you're not able to just kind of understand the basics of of these inputs that you have to supply for your own simulation where you yourself are probably not concerned with making a you know a hyper simplified uh simulation where you were just trying to predict if it's raining or not based on if the observation was wet or not. like you're probably working in something much more uh interesting frankly. Uh in that case yes you will want to know how to uh properly define dimensionalize and all the rest all of these different vectors and and matrices that we're seeing. So um this is some draft code that uh Sanjieve supplied us with. So, I just want to bear in mind like this is um I've made some modifications to it uh just to to try and make it something more kind of compressed and condensed to be able to show on a single slide. But um this is what we're doing whenever we talk about how we're using the A and the B and the D and some one hot vector of observations uh to do inference. We're given algorithm 14 which kind of get tells us how to do this, right? So given those three parameters, given that we're going to run a simulation over capital T time steps and given some observation or set of observations, O hat, I think we're using O hat to denote not just one observation but rather uh an entire series of observations. Um also quickly a question is SVO here Q of S um in in a sense right uh I would say that the the the SO here is going to be referring to Oh yeah this is going to be Yes. You're exactly right. So I see exactly what you're talking about now. Yes, that's right. it'll be Q of S at zero time step. Yeah. So I would I would write it like you know something like this. Um yeah and thank you for that question because it's that is important here. Uh and and it will follow from you know it's effectively we're just following equation 9.18A in this step. So I'll work this from sort of top to bottom. We're just looking at a model. First of all, there's no environment. Recall the previous algorithms we've been seeing. Sometimes there's like an environment or generative process step and then there's also an agent or generative model step. Here, we're not concerned with that. There's a very concise uh simulation where we're just going to assume that the agent is in some hypothetical environment where it's always wet. Uh so we're just going to create a whole series of observations in advance. that's a in this line here observation sequence where it's just always going to be wet. So that one hot vector simply means that we're going to have a vector of all of our potential observations. The 1.0 is going to denote the probability of wet. You know one not wet is zero. So what does that mean? It means we're giving the the agent uh an observation that's been encoded into this nice vector that it can use for this equation where the 1.0 know for wet means it's a it's as if it's an observation that's um I take this loosely but it's it's we're fully certain that the observation is wet and it's definitely not the other mutually exclusive observation not wet right um so so so from top to bottom first we're going to say we want to run a simulation for 10 time steps so t capital t equals 10 uh we're going to and I brought this up in a previous coding example we're going to initialize. This is just a big empty uh matrix that we're just going to be able to store the history of all the hidden state updates. I don't include a plot here to show the results of this simulation cuz that wasn't quite the point. But you'd be able to kind of take this S, not only of course take this S and use it for inference below, but you could also nicely, you know, make a plot of all your your um results as they've been stored. Right? So we'll vary frequency, not just one, but probably many different kind of containers that we make initialize at the beginning of a simulation. Next, we'll create that observation sequence. This is saying all of the observations will simply be wet as opposed to not wet. Uh and we'll make 10 of them. So this is just kind of Python uh syntax for saying that for each of the time steps over the range capital T. So if you took 10 made it a range from you know uh down from 0 to 10. So we're going to have 10 time steps. So it's just going to create 10 uh sequential observations for the agent. And you could kind of pop in you know whatever you like if you want to take you know rewrite this exact code try different things. Uh notice also that we're actually supplying a whole distribution to the agent as observations. So you could technically have um you know a distribution over the observations in the sense of you know maybe there's some other layer beneath that is creating these observations for the agent where there's inference there that happens that then makes some prediction about what the observation should be and so you won't necessarily have a one hot coded um vector that's something that I think gets overlooked a lot because in many simulations people will do this kind of strong one hot vector encoding big sort of thing. But we are these models are equipped for handling taking in probability distributions too, right? So that gets very interesting whenever we look at like hierarchical modeling where maybe a lower layer has to make a prediction about a hidden state and then that hidden state prediction gets sent up as a as a hypothetical observation for the next layer above for additional planning decision making. you know sort of a maybe a higher higher order um um you know functioning or otherwise. So um here we can see the a likelihood matrix which we know is just uh the probability of observations condition on hidden states. Again we have this kind of dynamic where the O is u uh is represented by each row right? The first row is the wet observation. Second row is the not wet observation. The columns are then the hidden states. Uh this is like a a weather hidden state factor containing is it raining or is it not raining. So our agent is going to you know observe is it wet or is it not wet and then infer is it raining is it not raining and these are these values essentially uh represent the beliefs our agent has about how the observations relate to the hidden state. So the probability of it being raining whenever they observe wet if we ignore everything else if we ignore the priors if we ignore uh the agents beliefs about hidden state how the hidden state itself could change over time if we're only looking at the the likelihood matrix the agent would say oh it's a 08 probability that I I see wet and therefore it must be raining right um for D of course is very simple this is just saying the agent has a strong prior probability that it's raining. Uh very low probability that it's not raining, right? So, so these elements are just equivalently indexed to the columns here um in the A and then B. Here we're going to have each column represent uh the hidden state at the previous time step. So, was it raining before? uh and then not raining uh if it if it was not raining before and then the next time step or we'll say current time step. So if you recall the way that we sort of denote all of this is that we can see what is the probability of the hidden state at the current time step condition on the previous time step that that's what the B is sort of capturing there. So if you're able to encode that using the right kind of row and column terminology then you're you're off to a great step. Then uh we'll use equation 9.18A which we recall again that's the only one that we're using that actually uses the d for the prior because we there is no st minus one right there is no uh inference or or or prior or otherwise about what it was the last time step whenever we're still at the first time step that be making reference to a non-existent time step. Right? um we actually can have that kind of logic, but but even then it's we're still going to start with this. Um we'll get into post-diction and prediction uh shortly, but um so after we've run that first step, we now have uh as someone asked in the chat, yes, this would be our Q of S or our our well not quite because we're still in exact basian inference here, right? Like we're not seeing anything about variational free energy. So we could actually say that this is our exact posterior at the the first time step. So let me reframe that further. But yes whenever you know in the the immediate subsequent sections after this yes we'll start getting into Q ofS in the case of using variational basian inference. Um so anyway we'll we'll we'll run that as our first step and then after that it's as simple as kind of plugging the rest of the observations in. Um this actually should be slightly modified as well. Um this we should be skipping the zeroith time step. Pi Python uses zero indexing. So this this line here is correct for using equation 9.18A. But for the next one, uh there should be some logic in here to skip that, right? Because we've already received the first observation. we should be on to the next ones up and through to the end of the simulation after that which is exactly what this s you know t greater than zero denotes in 9.18b it's like for every time step after the first time step right so we'll um and then also uh two other crucial things first uh the softmax function we're using this is simply ensuring that because we're taking logs uh as well as um you know doing different kinds of transpositions. You'll notice you know in the previous uh slides which of course are just you know a lot of this is just figures taken out of the book um we see that you know t this kind of capital T I think it's a backslash top or something like that in latte format um but you're you're just transposing so here uh this is just saying like you know Sanjieve has given us a nice readable row representing the priors but really with the t he means that we should be viewing this as column. Um similarly for let's see yeah for A this is exactly how it should be. Um you know B is not transposed either. So yeah that again it brings us back to the whole column versus row major thing and paying attention to which is which. Um once you have figured out your particular computational method for PIMDP or SPM or whatever once you've decided on that you just stick with that right you don't have to um it's not as um confusing or or or detailed to talk about because they will always be the same within that library which is again why it's it's quite a feat to write these things by hand and if you were to write them by hand you should really choose uh a nomenclature and stick with it uh otherwise you're going to have many problems. going forward after that, right? Um so um after all of that um you know this agent would effectively be making exact basian posterior uh inferences given that you know sequence of of 10 observations you know 10 time steps of inference uh getting the wet observation I did I didn't include results here but we can imagine if they're seeing it's always wet and they're already very biased to assuming it's wet because it's in their priors as Well, um they're probably by the end going to very much think that it's uh it's raining, right? Um the B actually is a little less certain, which is interesting. This means that the agent it thinks that it will go from raining to raining again with uh a probability of 6. So, not terribly high. The B might actually limit the agent from sort of overinferring that that it's going to keep raining. Um but but the rest of it points to the agent will probably strongly infer that that it's raining. Um and if we included learning which we'll see in chapter 10 then in the case of learning the A matrix, D matrix and B matrix all themselves could be updated over time as our parameters. Just as we saw in chapter 3 for the continuous states based models, how we were updating our parameters theta for our continuous models over time. So even these parameter values themselves can can change and we'll get to that in chapter 10. Um so section 9.2.1 I wanted to isolate this. I'm going to move through this a little bit more quickly. The the whole point here was just to say uh given this figure where uh Sanji gives us 9.7 um very frequently you'll see published um simulations and models and the rest whether it be an SPM or otherwise but especially in SPM where we'll make the generative model and the generative process isomorphic. So more or less what we mean is structurally equivalent you know so so if the matrices and the agent are roughly of this shape they'll be the same shape from the process right and then we know through uh model inversion that not only can um you know given some predict g excuse me given some observations an agent can infer what the hidden state should be we could also reverse that say given some hidden states the agent could sort of infer what the observation should be. So um a nice property of using these isomorphic uh you know model and process together in the same simulation is that you could see how well your model performs relative to the true process in sort of a onetoone way. um in matters of perception like this where we're not focused on action, we're not focused on there's nothing regarding exogenous forces or preferences that we saw before in the in in chapters like six and seven uh and 8. Uh instead this is just perception only. Um, we might think we, you know, we might be able to say that an agent who performs well in this simulation is an agent who, you know, through updating and perhaps learning by the end their parameters end up roughly matching exactly the parameters of the process. We saw that in the continuous cases too, right? Sanjieve would frequently use examples where you could see how um you know through learning an agent's parameter started to look very similar to the true processes parameter. Um there is absolutely no requirement though to do this. there's no requirement to make things isomeorphic and in reality in concrete use cases um I'll put it very simply um you know in in a in a true use case uh you you yourself don't know the true process right like you're not creating that you're you you have whether it be in some um you know some some health related work or trying to do some kind of inference to to to analyze neuroiming data or you know a robot who's trying to navigate an unknown environment that you yourself did not create. It's a real physical robot and real physical space that you don't have every bit of control of a self-driving car. You know, we wouldn't be doing any of this modeling if we already knew the environment. If we already fully knew the environment, we'd probably not be using variational methods. uh aside maybe from some of the the ways that they um you know make things less computationally expensive but it's just all this is to say we're we're trying to build models that can actually kind of learn uh the environment around them. So this is this whole isomorphic approach. It's it's in one sense fantastic for testing a model, but it's utterly unrealistic in in reality if we don't know the generative process. The experimenttor doesn't know it. The model itself doesn't know it. Uh we use the model to infer it. Right? So that's that's the goal. But um so I just want to make a strong statement about that. It's not really well spelled out in other places. Um, but you can see whenever we do that, you can do things like, oh, well, we created the process. So, we're able to track what the true state was. We're able to track what the the the agent inferred and um, you know, it's it's it's quite uh a trick to plot things from discrete state space settings given, you know, things technically aren't, you know, snowy to cloudy to rainy to sunny. like this isn't a proper y-axis in the sense of we're going up and down continuous values, right? They're just discreet. So, um you know, you do have to be kind of clever how you plot things, but this is pretty well done, right? Because you can see where the the inference matched up correctly with the the true state and where it didn't. Uh it looks like this agent is a little more privy to inferring that it's raining when it actually is snowy. I didn't show the matrices for this. That's all things that you'll find in the textbook. Um but it's for example 9.2. But um yeah, so um an entire next section observation and belief time indexing which is section 9.2.2. Uh I see that of course we're running out of time. I'm I'm glad that we we it seems we'll be uh spending a third week on chapter 9 because there's so much more to get into. Um but uh at least for this section all this is to say we're going to introduce um one more thing with respect to time. Um previously we've been using small t to represent a particular time step out of all capital t time steps. So you know in the this coding example we had a capital t of 10. So t equals 2 could mean uh you know it's the second time step out of 10 time steps. Um that that that's where what we're kind of alluding to here with observation indexing. Um you don't have to use that phrase but it's useful here just to denote like this could be the time scale at which an agent actually receives observations. They could receive it at time step one time step two and so on. Um, we can kind of think of it as global time, you know, in the same way that we think about um there's a difference between 2:00 p.m. uh you know, saying it's 2 p.m. or it's 400 p.m. versus saying 5 seconds ago or 5 seconds from now, right? The the former case of like 2 p.m. or 4 p.m. That's a sort of global time that that we can kind of recognize, right? That's sort of our simulation time. Um this meanwhile this belief indexing is more of the idea of like you know one hour from now or one hour ago. It's very relative or I've written it as relative to the agent. I've taken some partial liberties here because there were some aspects of how Sanjie wrote this that um I'm not sure if I'm fully in alignment with how he taught them. And I I just want to make that very transparent here that I I don't want, you know, if if I misinterpreted anything. Um, you know, it's it's worthwhile to continue reading the textbook, ask a question the discourse, uh, you know, and Frasier will have a session on, you know, each Friday as well, and we'll maybe see his take on it, which is given I've most of my previous work, uh, has been in the discrete realm here. I I I suppose I have a few opinions. Um and uh so it's whenever we allow for this idea of having relativity for the agent, we're just saying that you know we we've been looking at these these marov models. You know, a hidden marov model is is what we've been seeing so far with the A, B, and D. We're not talking about action yet. We're not talking about preferences yet. Those are all things we've seen in POMDP models, but so far we've seen HMMs or hidden markup models. All of them have these different marovian assumptions baked into how the models are designed. Uh that means that you know everything is technically relative to the agent from the agent standpoint, right? It it doesn't have a sense of time step for and global time. We we haven't kind of given it that. We haven't encoded that sort of into the model. What we do have is something like an inference window from where you're at currently where you assume that everything is temporally sort of uncorrelated or independent. All hidden states are independent of one another. Um and so everything is going to be very relevant uh relative to what's currently in your model at the current moment. Recall how uh another example of this is like whenever we did exact inference, we started with the D at the first time step, but we leave it off and switch to the B with the the temporal state transition matrix B at using our previous prediction in the previous time step. Um we use that at all other time steps. So this this D is sort of you know it's not for all intents and purposes gone so to speak, but it is in a sense sort of gone from our model. Um the only way that we would use it again is if you sort of deployed this agent over multiple simulations in which case you once again would start with the D at the beginning of each simulation and that that becomes relevant in things like behavioral trials or if you have a model that you're going to do learning online learning in you know one day and then you know the next day you do it again starting a new simulation you might do that and that allows the agent to kind of have a sense of like I'm starting a fresh I have I'm going to use my D and um yeah so um so everything is just sort of uh rel you know um related to the current moment even with the B matrix right like that's that logic is here this isn't saying at time step 4 the hidden state was this and therefore the agent will use the B to predict what the hidden state is at time step five this is always saying what is the hidden state now relative to what it was one time step ago right? Whether you're at whether you're at time step five or you're at time step 10,00 does not matter. That same logic applies. Um so that this kind of notation is what allows us to do that. And then whenever you sort of extend it, you can actually have within your agent the capacity for making predictions uh beyond or prior to current moment. pred uh predictions would just mean you could have an agent who has this inference horizon h of of three sort of internal time steps where we'll call them towo rather than lowercase t that that that is a very consistent nomenclature that's been used for quite some time. Um but that means the agent can always have it can at each time step sort of update its beliefs not just about what the hidden state is now uh at at toao but it can update its beliefs about what uh the the hidden state will be at to + one and then you know it'll continue relatively uh in that way at each iteration. it'll always have some belief that it has about what the next time step will be regardless of what the current global time step is. So it's kind of these two time scales that are operating. It's just they're it's not that one is faster and one is slower. It's that one is sort of global and the other is relative. And then the the the little minute liberties that I took where uh he gives us this figure 9.9. uh I I didn't I sort of see the value of what he did by he's trying to show us here like if time if global time step 16 little t is 16 um I guess he's trying to say that uh towel this is sort of the 16th time step that the agent is confronting but this is um this is not how I would do this this I would I would say that tao is always zero in the present um that's what you'll find in a lot of code. Uh that's that's what you'll find baked into the logic of how the B is constructed. Uh I I think it kind of defeats the purpose of even having two of these if you're going to write them as if they were always of the same value. But I I I can see how he included towel for the future one. So this this first uh image because I should have more expressly said what it shows. This is just showing like an agent over time. ignore the A and the B and the D because that would just kind of over enumber the the graph, but just assume that they're there. You know, this is just our agent operating over time. You know, if the present is global time step 16, then the agent could still make predictions about the future. um because it has this inference window where it's going to maintain a tow + one and tow plus 2 and plus 3 uh you know over time. Pardon me, I put these slides together a little bit hastily so I wasn't able to kind of copy in uh you know the towel symbol over and over and and make this nicer looking. I also just wanted to make it very plain that like you know these were added which is kind of why uh you know this sort of crude uh dark blue coloring. Um but the so I just added these to uh symbols here and same thing for down here. So so this was prediction where the agent is able to maintain predictions about the future. Post diction is actually the agent's ability to revise its beliefs about what happened in the past in light of new information which is a very you know a thing that we can intuit it quite well. We, you know, as people just on the dayto-day, we very frequently, you know, see something happen. We don't know exactly what happened. We're very uh unsure of it and then we get some further information or context or learn something afterward that then allows us to better understand that phenomena that we didn't quite understand the the first time around. Uh you know, all the time. Uh so so that this is sort of um you know as far as a hidden markoff model goes and the implementation goes this is sort of the equivalent of that. Um another thing that's important to note is that because the agent obviously hasn't seen uh future observations, right? It's not coming into contact with them. Uh it's not able to to carry out a full inference. uh you know it's not able to carry out the equivalent of equation 9.18b uh to or let alone the the first time step version either to make predictions about the the future in the same way because those observations are unknown. What we do there is that we still use something like the same equation except for the O which would be our observation um we're just going to make it a null set. Um I I do like how Sanji described this and the notation he uses. He's just making the point that this is actually like an empty vector that would contain one hot encoding of obser discrete observations. Um but because we don't know it, it's empty. uh if it is empty uh whenever we you know multiply that by log a this entire term cancels out that leaves us with the b or at least log of b you know thrown through a soft max um so that simply means that we're doing u we're just using the b to make predictions cuz sure we don't actually know observations that's canceled out if we don't know observations then we can't use a likelihood that relates observations and stays that cancels out we're left with our B. But our B does within it contain, you know, a sense of time. You could technically quote unquote roll out this B over time and say, well, if if it's currently raining, uh, you know, and or excuse me, if it's currently raining, what what is the probability uh if it's currently raining, what is the probability that at the next time step it will be raining still or not raining? I mean, you you could just keep doing that multiplication over and over. uh hypothetically into the indefinite future if you're if you set your inference horizon to be you know exceedingly long. Uh or if you did some sort of extension where you had like a um a dynamic inference horizon or something like that which would be much more complex to to discuss but could be done and you could do that going back into time and then going back into time might be worthwhile in the sense that you at least do have true observations that you witness. So you would be able to uh with with post diction uh kind of look back and and and and do a fuller computation of that. You could also uh on the flip side of this you could uh determine what the agents expected observations would be because you have an a matrix. So you could do additional logic where you have an agent who does a roll out or maybe procedurally does some kind of roll out and then using that roll out can then predict what the observation should be because you have an A matrix uh that that would be able to tell you um you know given you made a you predicted a hidden state for the next time step. You could use your A then to then use the likelihood to figure out what the observation should be at that time step given your prediction. Of course, you can see how this snowball uh as you know you start building more and more inferences whose very inputs themselves are inferences, right? So, we we we should expect that the further the agent makes predictions into the future with all these things, the less likely it is to less likely in a figure at least in a general sense to be accurate with its predictions. So this isn't to say that that the agent would be doing something like perfect you know exact uh high accuracy inference. Uh you know if you set h to 10 and the agent you know built up uh you know predictions on predictions on predictions on predictions all the way up to the 10th prediction for both states using a roll out and observations all these being dependent on each other. Right? So, so it there's a lot of sort of computational depth involved, but it so um it's just important to be able to recognize the agent can have this sort of sliding window relative to itself that goes into the future as well as into the past. PIMDP has some very straightforward ways of defining whenever you first make your agent, what should the inference horizon be um and and and the like. So um this is the last slide I I gave. Uh not the best one to speed through, but given we're about out of time, uh this is in one way can be read is sort of a a an alternative take on chapter 2 whenever we saw gradient descent on free variational free energy in the continuous state space case. Uh if you recall what we did was that we used this gradient of of free energy with respect to hidden states partial derivative. We could use that to update our belief about hidden states and then use that updated belief about hidden states uh to update what is the variational free energy now and we could do that in a series of iterations this kind of you know gradient descent iterations within a single time step. So you could have a single time step that sure here you could have a you know in the discrete case you could have a single time step where you only receive one observations but using that observation you could that singular one you could do 16 iterations of gradient descent using that. So you could kind of make that initial inference but then after that followed by even further ones um using the by following the gradient we're just able to make this much more uh sort of computationally cheap while allowing enough iterations for a sort of improvement over time and then uh Sanjie does include and I think we've seen this in previous algorithms but this convergence rule you technically don't have to include this there are there are posit positives and and and and and you know pros and cons to including a convergence check at this kind of you know such a granular scale. Uh if if you if you what this is saying is if the change in free energy as you're carrying out these gradient descent iterations within a time step say you set it to you know always do 16 iterations per time step. Um then what happens if you start doing what happens if you you really nicely minimize variational free energy by say the fifth iteration. But then after that the change in free energy keeps staying the same or worse starts going back up. You know like we we've kind of overshot uh a local minima or something and now we start moving back up and actually our inference is getting worse by the end. This convergence check is a way of sort of hard coding into your model. Oh, if variational free energies change is uh not less than some uh you know he uses this symbol. It's like a math cal t in latte. Um if you you could set this to be maybe a negative number. For example, you could say it variational free energy needs to be continuing to go down. So if it's not less than zero or if it's not less than some other value then um then stop doing those iterations and just move on to the next time step. Um in that case what you could do is just sort of fill you know once you do hit that convergence you could fill the rest of the iterations with that posterior update you reached. Um and then this plot in the lower left corner I just included it is it is from the textbook but I I wanted to show it in the sense that you can see there are only five global time steps in this simulation. So how is it possible that actually these lines which represent the probability of each discrete hidden state in this model you know this could be raining versus not raining. Uh how is it that they're able to slope and change within a single time step? It's because here this is doing uh you know iterative updates per time step. So so what we're seeing is you know these values are you know uh more granularly shifting. And so um this relates back to sort of a lot of the the more theoretical or empirical work that's been done on trying to say that um you know again we have something like a nested a nesting of time scales where the the rate in this kind of setup is as if we're saying the rate at which an agent is making their observations in global time um you know is actually slower and and and I'm this is another thing I think the textbook switches where it's it says faster rather than slower, slower rather than faster. It kind of gets these things um backward in the sentence where I saw it. So, I just want to say the it's as if uh the the iterative updates or belief updates are happening much more rapidly, meaning at a faster time scale. more things happen uh you know within uh this this time scale. Whenever we compare it to global time or global time it took um you know we get one new observation for every 16 gradient descent uh you know based updates flow updates um if if we say that there's 16 for each one. So um this tends to align with the the notion of um of sort of the scale at which a lot of neuroiming data works whenever whenever uh tracking uh EEG or otherwise at like a fine scale you know within the realm of milliseconds or or hundreds of milliseconds um or so um and uh what's really interesting that this textbook doesn't get to at all is that you can actually whenever you do things like fitting this to neuroiming data and being able to see like oh it's quite interesting how these things start to sort of emulate what it looks like whenever we look at neuroiming data that is whenever the neuroim imaging data is you know itself being generated incredibly rapidly by the brain during a behavioral task which I'm not saying the behavioral task is slow but I'm saying relative to the speed at which sort of the the brain is is is kind of producing these measurable uh quantities in the [clears throat] form of uh EEG sensor uh measurements. Um you know there's so many of those versus you know what actually happens in the behavioral task. And this gets related to things like actually being able to compute things like uh local field potentials over time and being able to potentially uh see an alignment between neural imaging data and how the brain uh is sort of doing what we theoretically at least with an active inference are saying uh is uh you know minimizing prediction error. So um it's um that that's all a lot of fun stuff. I would strongly suggest looking at uh more work specifically by Carl Fristen uh Ryan Smith and and Thomas Parr various others who've more directly looked at the neurobiology uh because while it might seem like discrete state spaces are less aminable to the kind of granularity that we see in continuous state space models, you know, neuroiming data itself is almost almost always continuous or at least the way we we work with it. Um but uh nonetheless the discrete state space case actually there's a lot of been a lot of work on the neurobiology uh sort of substrates for this. So um it it's quite interesting. I strongly suggest uh having a look at that. So um so this this is the scenario where um you know if we if we if we took this same code here but then we modified it a bit to actually carry out something like gradient flow on variational free energy um so that we could we could do more variational basian inference rather than exact inference. it's there that you know we're considering excuse me considering our our our Q of S rather than just um an exact posterior of P S condition on O right so so there I I sped through this a bit because one it's of course very dense we saw in chapter 2 that we were given gradient descent very quickly that that's a that's a topic that that definitely warrants its own entire course uh but uh at minimum I could say that if you kind of flip back and forth between the the algorithms we see at the end of chapter 2 and then look at this algorithm, you'll see uh a broad variety of similarities between the continuous case and the discrete case. And given that the textbook was sort of written the way that it is, that we did a ton of continuous state space work leading up to this, um, once again, I think it it makes for a bit more of a coherent learning experience, especially if you're someone who just wants a broader sense of the continuous and discrete cases to be able to relate them back and forth to one another. See that we're still working with priors, likelihoods, state transitions or flows. Um uh you know we can apply gradient descent related algorithms here to to iteratively update uh in a in a much computationally cheaper way that may or may not better align with with the actual uh functions carried out in the brain or in a way that might be measurable or or able to be modeled. So, um, some things that of course are still in chapter 9 because this was only up to a condensed version of 9.3, but we still have sections 9.45 leading to the summary and conclusion. Um, is that we'll next week get into planning and action. So we're gonna it's it's there's going to be quite a bit but it'll it'll be all the remaining core elements uh that that we'll still be able to relate back to the continuous state space setting. Like if you recall in the continuous state space setting whenever we included action um there was this kind of forward [snorts] model that we used that related to to the derivatives of of y with respect to a and the like. So what I mean to say is with action there was this kind of um not quite uh appendage but there was this sort of additional uh equation that we're using that does its own updates next to the primary model that we've been working with for doing hidden state inference. That's what kind of allows for action and yet they they sort of work together over time. It was like the the the bump agent that we saw I think in chapter 7. Um I've gone a bit too yeah this here um this agent who uses action to sort of counteract an exogenous force and that agent sort of maintains its own uh model of action that is conceptually within the environment rather than within the agent. Um right so we have uh updates to a uh itself and really the way we do that is in a variational manner is using our sensory observations to better inform that it allows the agent to use action to uh uh try and impact the environment such that the environment elicits new observations perhaps those that the agent would rather see uh in the way that I use my hunger my my observation of of is my stomach growling or not yes or no to infer if I'm hungry or not uh you know I I I will then eat uh as my action to change the hidden state of the environment my body's hunger level or or caloric intake uh currently to lead it to elicit oh it's no longer growling my stomach is no longer growling like I'm I'm good uh I could I don't know the hidden state without using observations That's why action and observations are so uh uh so strongly needing to be sort of coupled together and related to one another in this way. Um and allowing for a kind of proper inference to sort of get what you want so to speak. Um we'll see that kind of logic with uh the models in the later part of chapter 9 next week. So that will go over we'll no longer be looking at just hidden Marov models. uh they will instead be called partially observable marov decision-making processes. Um it you you can already tell like hidden marov model partially observable marov decision-m processes. It's like we we have hidden partially observable both of them can be employed in the same kind of environments. It's just an hmm will just hypothetically just sit there, right? It's just perception only. It doesn't carry out any action. So it could take in observations but it can't do anything about them. It can just predict uh you know hidden states. It can predict what the next hidden states will be given the inference horizon or it can update through p postiction its beliefs about what they were before based on new observations. Um and and then on the the flip side a POMDP model will be able to do essentially all of those same things. It's just that it will also be able to to infer policies. It'll be able to infer what kinds of actions it should take. And that'll be sort of the discrete analog of of this of of uh having a you know in the continuous case having a gradient flow of action and being able to choose which actions to take. This partially observable Markov decision making process. We'll be able to make decisions about what actions to to take. So it's just going to be an extension of HMM's. Nothing is dramatically different. There are just a couple of things that are slightly dramatically new, including preferences and what we'll get into what's called expected free energy, which is something that is distinct. We haven't seen yet, but it'll have some analoges. Um, so I see that we're past time. So, so I would like to thank everyone for for attending and feel free to join on Friday as well for Frraasier's f Frasier Patterson's take on uh week two of chapter 9 and then same time next week for this session. Um, we'll go through hopefully chapter 9 just one more time and we'll get into the POMDP models and then for chapter 10 I would very much like to to deliver to you all uh some kind of at least partial or mini tutorial on working with POMDPS. Um, I'm not doing that beyond right now beyond just showing some sort of starter code as it were for constructing the matrices because we we really need to kind of already have everything together in order to properly do a full demonstration. Otherwise, you're just going to be getting these partial pieces and it'll be unclear of how to fit them together right away in order to do any kind of like, you know, genuinely interesting experiment that isn't just a hypersimplified like subset of what could be a full experiment. So, yeah. Um, all right. So, thank you all so much and uh we'll see you at the next one.