Fundamentals of Active Inference (Chapter 10, Session 45) September 15, 2026

Right. Greetings everyone. Thank you for joining the active inference institute textbook reading group. We are still reading Sanjie Nam Jooshi's uh compendium of uh active inference uh to date the fundamentals of active inference released in March of 2026 this year and we are just now starting chapter 10. Um before I proceed further, I just wanted to share we have a uh one of our uh board members and and participants in the textbook group, Aliana, uh has released a uh you know some series of YouTube videos related to uh AI uh machine learning and now very recently uh uh a video on the notation used in active inference over the past decade or so. And uh I I would uh I not quite watched it myself yet, but I would strongly recommend checking out different kinds of um uh resources like this in with respect to the fact that for those who have been reading the the textbook up to this point, who have already read previous literature, they might notice variations on the notation that are used in the mathematical equations. um very worthwhile to get a sense of how that notation has been used over the years especially because unfortunately you know at at different times there have been attempts to standardize notation. Uh other times um you know you you have folks coming in uh from other uh domains who will apply their own respective notation like there there are many uh folks who do reinforcement learning um uh in engineering settings where it's more often that X will be your observation and Y will be your hidden state or the other way around um and and and B will denote your likelihood and A will denote your state transitions and all the rest. So, um, yeah, I do I do recommend that. And then, uh, maybe going forward I'll find some of the the institute's own materials on some of these things. We've had the ontology project in the past. We've also had um, if I'm able to type this in uh, quickly, we'll see. Um, yeah, that's got it. Uh there's been a what uh Daniel Freriedman has called the general um notation notation or GNN uh project. This is um hosted on our repo uh publicly with the active inference institute but it's essentially an attempt to be able to sort of consolidate a lot of uh previous literature and and uh computer code and and programming u that that's related to or associated to active inference and adjacent uh models um to to be able to sort of be able to translate between different kinds of notation uh that have been used in the past And uh it's it's still a rather fascinating concept. I've contributed to it uh at at different points in time. Um but it's still something that I think has a uh a lot of um you know it's kind of fertile ground for for growing in the direction of not just being able to sort of standardize the notation but actually having a sort of Rosetta stone so to speak on translating between previous and current notation. So that way we don't simply uh you know eliminate and forget about previous notation and that helps us maintain a kind of coherency over over the years. So um that that is something that is is worth um looking into and uh that he he uh wrote a nice uh write up uh preprint on on this uh with uh Yakob Smeckl who's done a lot of work in active inference as well. So, um, worth checking out. Um, and in any case, uh, we will now get to chapter 10 because I know we're, uh, already fit 15 minutes, uh, into the the the clock for the session. Um, so, uh, to to sort of bring it back here, um, as we know, the past three weeks we have spent on chapter 9. Uh, we we we very deeply, very quickly got into discrete statebased models. Um and we uh were discussing hidden markoff models and POMDP models where POMDPS are simply hidden markoff models but with the capacity for action and decision making whereas hidden markoff models are purely perceptionbased. So they'll receive observations, they'll update their beliefs but they won't actually sort of do anything from there so to speak, right? They they don't have any policies or actions available to themselves. So we're uh going to be more precisely focusing upon POMDP models. Um you can recognize that if you have a POMPP and you uh only supply it with one possible action, you can consider that to be virtually equivalent to a hidden markoff model. So for anyone who's still thinking about the difference between those two things, you can think about well if you only ever have one action, then you effectively have have no action, right? There's nothing you're actually doing. um you know that there's no modulation, there's no consideration over different potential actions or policies, right? Um because there's only one possible action. So it's as if there is no action. So that that might be a helpful intuitive point for some folks. Um in any case, uh chapter 10 will simply extend much of what we learned in chapter 9. uh wi with chapter nine I would have liked to in the later weeks uh at least the final week give some kind of direct coding example um but but I didn't because chapter 9 and chapter 10 I would argue are very much of a piece um that is uh we could have shown some code for chapter 9 and I actually did uh during the first week for anyone who wants to look at that but that was just sort of getting into some sort of basics uh with respect to kind of doing things like constructing your A and B and and B uh matrices/ vectors. Um so there's a ve very simplistic sort of code for those who are just wondering about like what what is a a very minute uh you know single algorithm look like whenever we define these different parameters that we've been talking about. Um but from there any further coding of examples would have brought up many questions and many of those questions actually could be uh answered by just moving on to reading chapter 10. So hopefully we'll get to more coding examples and I know that uh Frasier has been putting together different kinds of of code examples as well and I believe he's already put those into the fundamentals uh superhuman page here. Um let's see. Yeah, so he's already consolidated yeah some nice code examples for chapter 9. So those could be very worthwhile to have a look at. I believe he's doing something similar to me where he's trying to give examples that directly uh draw from the examples given in the textbook just to keep this more coherent. So, um, so chapter 10. Um, apologies in advance for some of the the slideshow karaoke that I'm about to do, but, uh, I I don't usually prefer that, but, uh, you know, as you know that I I write, uh, these slides to be rather holistic. I'll drop the link to them in the chat once again. And um yeah, so I I I put a lot of text on these just so that um folks who are learning asynchronously have uh something very plentiful to sort of chew on. Um especially whenever you don't yet have the textbook your yourself because we are allowed to share the equations and the figures. We're just not allowed to, you know, share the raw text. You'll have to purchase the textbook for yourself in order to gain that. So um so chapter 10 uh the the extensions or expansion it kind of provides from chapter 9 that is section 10.1 uh we'll get into parameter learning which we'll recall um each of these new things that it that it it provides to us um have their own continuous state space model analoges in previous chapters of this textbook. So again, for those who are reading from the very beginning of the textbook up to this point, uh, who have already spent all of the time with all of the continuous state space model aspects that we were looking at with, you know, um, active generalized filtering and predictive coding and the like. Um, all of these uh, concepts that we see in chapter 10 have their own sort of analoges more or less in in those previous chapters. So just for for coherency, I wanted to provide a sense of that. So in section 10.1 again we look at parameter learning. Well we already covered much of that in chapter 3. So intuitively or conceptually you can kind of carry that with you. We'll still be using the symbol theta to represent sort of a collection of all of our parameters. So in chapter nine, the main parameters we've been working with are our likelihoods or a uh state transitions or b uh priors on initial states or d and then and then some other things that we saw glimpses of were uh are preferences or priors on observations c which gets used in uh decision-m and uh and expected free energy uh computation and then um e which is our our habits which we didn't see too much of in in chapter 9, but Sanjief gets into it in chapter 10 here. It becomes much more relevant whenever we start looking at further ways that agents can can make decisions and and what kind of uh algorithms can be played out in in support of that. Um so so parameter learning for POMDPS we'll also see um you know we saw a lot on uh precisions uh you know these kinds of uh scalar values that act as these kind of gain terms in our in our computations. We saw precisions uh sensory precisions or precisions on in states uh and their relationships to to prediction errors. So um we have different kinds of precisions in the POMDP models that we're looking at here too. Um and and that'll be very interesting with respect to decisionm and I uh well I'll get to that. Uh in section 10.3 um this starts talking about what uh Sanjieve labels as factorial depth. Actually on that note uh I will also share this um uh I feel like this is rather canonical because Nooshi co-authored this um but there's been a lot of recent developments in the notion of structure learning. Um and so um that this this is a very sort of pertinent uh area uh uh in active inference uh literature and research and and many non-active inference areas as well just more general uh reinforcement learning. There's been considerations over structure learning for LLMs and and and the way that LLMs can be put into different kinds of agentic workflows. So I would also recommend this. It provides very nice take on these things like factorial depth, hierarchal depth. We get each of these in chapter 10. Um this notion of generalized depth which essentially just relates to the generalized coordinates of motion that we've already seen in this textbook. Uh I think starting with uh chapter six and then temporal depth which kind of just relates to the way that we've been using towel. um that is uh kind of extending an agent's sort of inference horizon or how far ho how how far into the future or into the past it can sort of look within this kind of marobian setting um so uh worthwhile and I've done a lot of work in these areas and this year was working with um uh Lance Dosta on uh the uh renormalization generative models which also might be of interest to others But I'll now uh uh I'll get back to our our programmed textbook conversation here. Just want to >> do you mind to drop us that link, please? >> Uh yeah, absolutely. Which link? Uh Kovus? >> Uh the one you just had up. >> Yeah, the building blocks. >> Yeah. >> Yeah, for sure. Thanks. We'll just that. It's having fun playing games with me. Here we are. >> Yeah, I I I like this paper in that um you know, similarly to where we started where I was talking about uh the need for you know, if not a universal notation uh in active inference or at least a way of translating between uh different forms of notation. Um this paper also tries to provide a more universal language for talking about structure learning which um unsurprisingly relates to things that don't have to do directly with structure learning right like we're like I said you know um chapter 10 here we don't talk about structure learning but hierarchical depth and factorial depth directly have to do with the structure of your model whether you're going to be updating the structure itself self or not is its own question that would be what's involved in structured learning. Um but I I personally see this this paper is a is a great and of course it's discussing all the things we're talking about right in Markov models and it'll get into um POMDP models and then it talks about a lot of these more interesting continuous statebased models that um we are not covered in the textbook. So um yeah very much worth getting a little familiar with the the terminology going on in this paper. I very much enjoyed it and view it as sort of maybe where things should go next um to to put it briefly. So um so uh with all that said given that we'll have three weeks for chapter 10 um we will mainly be focusing on the first section which is just 10.1. I'm going to kind of put uh 10.3 and 10.4 together. In the in the final week we will get to 10.2 2 on things related to precision uh next week and then of course there's anything that Frraasier will cover during the Friday sessions. So for 10.1 um I'm essentially going to try and cover everything that we can for now regarding parameter learning which again the chapter 3 was sort of the continuous model analog of this for for continuous models. We're just now looking at discrete models. So, um, so again, we're just as before, we're going to sort of collect all of our parameters into this theta term. Um, it's not actually necessarily a totally essential thing to do, but it's it's allows for a more compact way of writing the notation here. So what we've done is that whereas previously we could write our full generative model in this way equation 9.49 where we had you know uh our generative model is composed of you know the the way that it's able to understand observations states and policies. In the case of a POMDP, we've now simply added theta, right? And so now that means that we're going to treat our parameters as as rather than fixed parameters that we never update, now they are now they are in fact learnable uh so to speak. And so now we have that that means we have to do what we did in chapter 3, right? We have to provide a prior on on parameters because we're still in this basian framework where whenever we're doing basian inference whether that's for hidden states or updating our parameters or otherwise we need some kind of prior in order to reach uh posterior after collecting some kind of evidence. So now we've supplied this uh prior on parameters which we we u factorized this which we've seen a lot of factorization it relates relates to the the notion of mean field approximation just being able to treat all of these variables as independent of one another which allows for a lot of nice qualities whenever we do inference and here variational inference. So, we'll actually have a prior on our uh A, our B, and our D. Um, and that's seen here in figure 10.2 just to quickly show that this is the same exact figure that we saw for a POMDP in chapter 9. It's just now pointing out that we have these these small B uh priors on our B and small D and small A. So all these will be the the the the priors that that are used for updating our parameters. And then what we saw previously was that whenever we used priors on parameters for continuous models, we always had to sort of shape those priors. We had to say, well, what what if if you have a prior and you're initializing a model where where nothing's been learned yet, nothing's been observed yet, um then the question is, well, what should that prior look like? what is our initial sort of guess for that and furthermore how should it be sort of encoded like do does it follow a particular kind of distribution. Um and so with the continuous models we we modeled all of our priors as normal distributions which of course are these continuous kind of distributions that have a mean uh around which most the mass may be concentrated and then a variance which which is essentially how spread out is that distribution. Now, we know that we cannot do precisely that in the discrete state space setting. We're looking at mutually exclusive states. Um, being able to flip a coin and receiving a heads or a tails is two mutually exclusive, you know, sort of observations that we could receive when we flip a coin. Um, you can't say that heads is greater than tails or less than tails. We're not on this kind of continuous plane uh able to to compare values in that way. So we can't use a normal distribution. Um we're actually going to do for the discrete state space setting. I I know that I've packed a lot of information into this this single slide, but actually um the discrete state space setting is much simpler uh arguably I I would argue for for learning your parameters because what we can do is that we can supply these priors that are structured exactly like the the parameter itself. So we can have um what instead of a normal distribution it would be called a deerishlay distribution in this case. And so if you're if your d uh parameter for example is just you know um if it's structured as having sort of two slots in a single vector you know like maybe an agent uh uh has a prior belief uh that it's raining with 0.5 and that it's not raining with 0.5. um then you could have a prior on that D that is structured in the same way. It will have two slots, one of them indexed to uh raining and the other one not raining. So that they look virtually equivalent structurally um except that the deers distribution is not itself a categorical probability distribution that is it does not have to sum to one. Um so how do we use that? Well, this dearishly distribution that is structured just like our priors uh excuse me the the dearishly distribution prior which are structured just like our parameters like how this this D could be uh structured equivalently um to to the dearishlay distribution on D. Um you can start adding counts of observations. you can start adding counts of um um you know what the agent has inferred in the past. So these dearish laid priors they just kind of accumulate. So I've labeled this I've titled subtitled this is accumulating experiences that the POMDP can start sort of collecting counts of different relevant events uh for each of its parameters. And so we have these very similar looking rules for each of our parameters. At first, uh, Sanjieve only gives us learning the A and the B and the D. Um, and so we don't have a C or an E. And so actually you could you could use these same rules for updating a hidden markoff model, not just a PMDP because a a hidden markup model will have all these same parameters. But what you can see here is that we'll we'll just kind of uh t take this prior on our a for example which we know relates observations and states and so if you have some parameter a that relates observations and states over the course of a simulation an agent will repeatedly have these instances where it receives an observation and then infers hidden states. So you can take that particular observation it receives at that time step and then the followup uh inference over hidden states from that time step and just add them to this prior and so so your your your prior this dearishly distribution it just starts accumulating more and more experience as we as we go along. So you could you could use these uh learning rules uh at every single time step of your simulation. So if you have 50 time steps, you could you'd be you'd be inferring uh hidden states 50 times and you'd be learning your parameters, you know, 50 times at the end of every time step. Um but the way that Sanjieve introduces this is actually a little bit differently. Um I do I do like that he he does it in this way which is uh he he introduces this in the context of trialbased learning. And a little bit of context on that. Um, it for those who who have come to to active inference through things like computational psychiatry and have interests in in behavioral science and analysis and doing different kinds of um behavioral experiments, they'll be, you know, all too aware. Uh, whe whether you're doing an experiment with um, you know, a human being or it'd be like a mouse in a T-mase example or otherwise. Um usually we carry out these empirical tasks um through trials you know so you you know each trial the mouse is trying to sort of solve the teammates so that it can get a reward and avoid uh you know an aversive stimulus or if you do this with a person who's playing some kind of um card task right then each trial will involve you know trying to solve what is the correct card um before moving on to the next trial where we sort of reset the environment and the the agent once again or the person human once again has to solve for the correct card. So in that case um the way that Sanjieve uh introduces learning is that we only learn at the end of the trial where each trial is composed of some set number of time steps. So in the in the T-mas example that we've seen uh in the past, you know, there there are three time steps uh and and so um you know, you don't learn at each one of those time steps. The the agent will in will receive observations per time step in the trial and they'll infer hidden states per time step in the trial. And they'll they'll they'll do decision making and and and um you know expected free energy minimization that we saw in chapter 9 uh uh during each time step of each trial. But then the learning the update of the parameters themselves will only come whenever the the trial ends um just before we commence the next trial. Um and this sort of speaks to the notion of of uh differentiation or having multiple time scales which we've already seen earlier in this book and it's a very common place you know kind of ubiquitous notion in computational neuroscience and elsewhere that the the brain effectively operates at these different time scales. So learning itself usually I is operating at a slower time scale than than fast real time inference. And so this kind of mirrors that logic where within a trial you have fast real-time inference where you know you have these different time steps in the trial but then the slower time scale is uh learning happens at only after the end of um all three of those time steps right it happens at the end of the trial together. Um so that that's what's kind of captured this notion of of learning at a different in in in a learning by trial at the end of each trial. Um I've also included so so what we do is that we we add these uh events related to the parameter like we'll we'll just focus on the D for example here is perhaps the simplest that we're given because it's it's not it's it's just your prior on hidden states. So it's it's we're not looking at we're not we don't have to include any logic about how hidden states change over time which is our B matrix or state transitions and it it's not including anything about how states actually relate to observations. That's our likelihood in the A. Um we're only looking at uh the agents prior on uh on on hidden states or prior on initial hidden states. um which is what it becomes more relevant to think of it as priors on initial states when we think about trials right when you beginning you begin a new trial you've not received any observations yet it's there that we sort of rely on this D uh this initial uh prior uh so we will we will you know whenever a trial ends for the D we'll just add the the agents um uh inference about hidden states at the very end of that trial, you know, and and that that's kind of updating like, oh, before I start the next trial, I've learned that actually if I were to be more biased about what the hidden state is in this environment, uh, you know, I have an inference for that. I I've used evidence from the past and experience that I've accumulated in the past to be able at the next trial to have a better what what the agent thinks to be a better guess about what the D should be a better guess about what those initial hidden states should be when we do that. Um so so say we started this agent originally had a D that was um you know we could normalize this. It would be a 0.5 and a.5 like do you think it's raining? Do you think it's not raining 5.5 um this is unnormalized but we would end up normalizing this. Um then then the agent by the end of a trial infers oh actually there's a much much higher probability that it's raining rather than not raining. we would just add this inference into that prior and now our new prior again it's not normalized. Um it doesn't have to be because it's it's technically a deerishlay distribution rather than a standard categorical probability distribution. So that these values can actually just get that you can imagine they will just increase more and more and more depending on what the agent infers. um that's that's what will uh determine which of these indices starts growing. So if the agent repeatedly infers that it's raining then uh by the end of each trial then you can imagine this this slot regarding its prior beliefs about raining will will become you know massive whereas not raining will stay roughly the same. There's nothing to to add there or or maybe a very small value like a 0.1 you know. So, so that's what's happening. We're just we're just accumulating more inferences from the past in order to inform the present or the next time step. And whenever we do that, then we just nor renormalize it and that's our new d vector. So really um you can consider uh taking this prior uh you can consider it itself to be kind of accumulation of of counts of past experiences that Asia has had. When we renormalize them then the new D was really just going to be the respective proportions of the events that it's witnessed in the past. Right? So, so roughly, you know, 88% of its past inferences uh have have have leaned towards um raining as opposed to not raining vice versa for this this other index. So there's also um you know uh this special rule for for whenever we're updating our B matrix uh or tensor whenever we're using a POMDP where the the factor is controllable in which case we we have to account for not just what was the previous state and the next state which we could add but we have to also pay attention to which policy the agent chose. So it's the agent saw you know that um in the last time step it wasn't hungry but in this new time step uh it it it is hungry and the agent chose to not eat food then we'll add a count to the tensor that says when you are not hungry but then you are hungry and you chose not to eat food as a way to control that then add that to your experience and then you can imagine the agent would start to learn oh if I don't eat uh even when I'm not hungry, I might start to become hungry, right? Um kind of an odd way to put that such a simple illustration, but but u there are more intuitive ways of understanding these things. So um so those are all the distinct rules that we get for learning the A and the B and the D. And um it's very much worth playing around with with code um for doing these uh these things. And there is, you know, plentiful code for working with POMDP models and HM models in SPM and in PIMDP, which uh I'll get into in a moment. Not the code itself, but just more explanations of it. Um, and yeah, let's move on. So, from there, here are some things that aren't discussed in chapter 10. Now, I I did uh you know, we we've acknowledged this textbook is is a is a wonderful compendium of of previous uh knowledge leading up to this point in in research on active inference. Um but there are some things that I think were a little uh deprioritized in this chapter uh or or or or not quite given enough attention. So, I just want to make sure that they're they're known. And there are some things that that were ignored alto together, which which is fine. It's still very plentiful amount of information but I just want to give some more context here. Um so this is just sort of still on parameter learning but I just added a second slide. Uh first we saw on the last slide that we can update our A and B and D uh parameters here. We can also update our C and our E. Um not everyone wishes to wishes to do that necessarily. I mean it's it's entirely irrelevant whenever you're working with just hidden markoff models because you'll remember that the C is only involved in decision making and so is the E right so the C is your prior on observation your your prior on um expected observations observations you'll see in the future and that's only used when making decisions um otherwise we we we consider all observations to already be known right um that is all things I've seen in the past in the present are already known to me. I've already seen them. Uh there's no reason to have a prior belief about them. Um however, it becomes relevant when decision-m because then I think well what will I do? Uh and and what do I want to happen? Uh that's where our C starts to come in. That that's where we have these preferences on observations that that can be used. If I if I if I eat food, will my stomach will I start to observe that my stomach is now being quiet rather than growling at me? Um that sort of thing. And then E would be our our habits. That's where we have a prior on uh policies, right? And so that's a kind of um that's a way of, you know, things you've done in the past repeatedly. We start to build them up as habits. So there's some really nice intuitive aspects of these parameters that can kind of get lost when we're just looking at the symbols over and over and trying to figure out what these, you know, equations mean and how the symbols fit together. But um to to think of C as preferences and E as habits, you you can actually learn them over time. And fortunately, they would follow very similar learning rules to all of those that we are given by Sanjieve. Uh for the A and the B and the D. um we're effectively just adding more counts, right? Uh for C, it's incredibly simple because usually the observations you receive are these one hop vectors. So um did you see uh rain or did you see no rain? Um if you saw rain, then that's there's one more count of rain to add into your your prior preferences and expectation that it'll rain, right? Um, and this relates to the notion of self-evidencing that agent self-evidence. It's that whenever you um whenever you make decisions here we're calculating expected free energy. That's one of the core aspects of decision-m with these models. But just to say it more plainly, if you're making decisions, there are more things that you've expected in the past um then um then you might act to kind of evidence those. you you will act to continue your expectation that those things happen. So so in that way whether you really kind of quote unquote prefer it to be raining or not, you will still if you if you expect that you're going to see rain all of the time, you might make different kinds of decisions that incorporate an expectation that it's going to be raining, right? So So while it's nice to call sea preferences, it's ultimately just your expectations about about observations. um more more generally rather than saying you necessarily want them or not. uh for context I've done some uh some work uh in computational psychiatry with respect to um those who uh uh suffer from PTSD where one perspective of sort of modeling the dynamics of PTSD is that and various other disorders is that as someone becomes used to the kinds of symptoms that they experience they then proceed to expect those symptoms going forward. Right? So someone who regularly starts experiencing or observing themselves having hypervigilance might in the future when decision-m oh should I go to this social event yes or no they might say well I should remember that I tend to feel hypervigilant which which kind of comes with a variety of things that don't feel so good especially whenever I'm in a social event. So, so then you can start incorporating that because you you could imagine hypervigilance. It's like a you know this this accumulation of previous experiences of of feeling that um you can imagine that that the index in your C vector for hypervigilance go way up and then that will start to impact your decision-m where you it's as if you're kind of assuming that you'll be hypervigilant and that might very well lead you to choose not to go to the social event, right? It's not that you want to be hypervigilant. It's that you've experienced it. You've observed that in yourself so much in the past that it starts to bias your decision-m because you're expecting that you will be uh hypervigilant. And then E, I guess in that same context, you know, if you um if if if an agent sort of has has a policy to withdraw from a social event as opposed to attending a social event, if if the agent repeatedly in the past has avoided or withdrawn from from a social event, um they might be more likely in future to do that. the the the e your prior on preferences. That's that's essentially saying that you have this prior on what you should do regardless of any other context. Saying that regardless of what you've observed, of what you've um of what you think is going on, this is what you do. I mean, one of the easiest things to think of is someone who has some kind of sub substance use issue with you. Think of u cigarettes, right? It's like if you have a strong habit for cigarettes. We'll just ignore the kind of chemical addiction aspects because with with with with smoking um it's not just that you can develop a chemical addiction but you can also develop a kind of habitual action of smoking, right? So, you know, if if someone's stressed out, go have it. And you have a high habit for smoking cigarettes, maybe you go smoke a cigarette. Um did you just finish lunch? Well, not sure what to do. you could just go ahead and have a cigarette, right? That that's sort of what habits will do. They'll they'll just say, "Well, regardless of what I've observed and regardless of my decision- making, well, here's something that I've done a lot." So, it's kind of like a contextfree uh prior on decision making or contextfree prior on on on what to do, what kinds of actions to take. So it it becomes relevant to be able to treat these as learnable or these kind of free parameters that that can be inferred or changed over time. Um just as it was relevant with the A and the B and the D. So I just really wanted to make a point that we can learn those as well. Um secondly, this is jumping ahead a little bit, but uh um section 10.3 will talk about factorial depth. uh factorial depth for all intents and purposes. We we'll get to it in a couple weeks, but um to to to look at the continuous analog for that. Um if you recall in chapters 2 through 8 you know just before we got into all this discrete stuff um we're looking at continuous models and Sanjieve would usually give us a kind of simpler univariate example where there's one kind of observation per time step and one hidden state that our agent infers per time step. right there. It wasn't that you um it wasn't that the agent could could see things and feel things. It could only see only see one thing at a time. So, it was a very simplified uh you know examples that he would give us for for the sake of illustration and he even gave us these univariat algorithms to to demonstrate that whereas he would use the word multivariate in cases where you might receive two kinds of observations at the same time. So an agent who can who can see uh you know who can who can see and feel things at once, right? Like I see that it's raining and I can feel with my hands that that that you know a surface is wet or something like that. So you could receive multiple observations at one time. And you could infer different kinds of hidden states at the same time. What is the weather like and am I hungry? Like those are two things that you can infer sort of sort of at at the same time, right? There's two different sets of beliefs that won't necessarily contradict each other. They're their own they are their own factors in in in your generative model so to speak, right? They they have their own thing going on and they can be inferred from these different sets of observations. So here with factorial depth um we can have not just one a matrix that relates you know um one one observation modality to to hidden states but you can have an A matrix for what you see versus your beliefs and then you can have another A matrix for what you feel versus your beliefs and you can have one for what you smell versus your beliefs and so you can have these different A matrices. um altogether and they they all get um utilized during variational free inference uh excuse me variational free energy uh and and the process of inference uh about states and policies. So it's just kind of saying that we can have these multiple factors is we can we can be receiving multiple things at once. We can be believing multiple things at once and we can be even doing thing multiple things at once. You could have multiple controllable hidden state factors where essentially the a model who can walk and talk at the same time rather than only being able to do one and the other. Right? So there's this kind of simultaneity and and being able to have these multiple factors. So that was a quite a mouthful but I want to include selective learning just to make the point that one of these um you know one of these relationships can be learned while the other remains static. um that can that can be essential in an agent who, you know, maybe maybe you want them to be able to learn their preferences where they um you know, they can start to expect different kinds of conditions um from from the weather. Um you know, maybe that's something that is learnable, but uh you don't want them to modulate or learn their expectations about their blood oxygen flow in their body, right? You might want that to stay fixed. We might be able to we might want an agent who, you know, for all intents and purposes can maintain its homeostasis by having certain preferences being fixed while others are are free and allowed to modulate. Right? So that that's sort of a choice uh of of the experimentter. An experimentter can make all of the parameters fixed. They can make particular parameters fixed. They can make uh particular factors within a par parameter fixed but not others. uh and the rest. So, so those are dynamics that can be there. I just wanted to be clear that you don't have to universally make everything free form and and learnable. Um there may be some things that necessarily need to stay fixed so that the agent can continue to persist in its environment. um you know it's we don't want an agent who learns to disprefer caloric intake which will probably lead it to to dying because it'll it'll assume that it doesn't need food uh during its decision making right so th those are different kinds of things to take account of uh whenever you're whenever you're constructing agents and models and simulations the like um another thing um I'm I I put the equations here so for those who are wanting to implement things like this um you know I I'll just keep this shorter. You can have learning rates. We saw learning rates for learning. We saw learning rates for inference and the like in these previous chapters um uh for the continuous models. We we can actually apply that um for our POMDP models and our HMM models during parameter learning. Um this is actually this is something I was I I'll be frank. I was a little bit disappointed that that um Sanji didn't touch on so much. Maybe it's just something I miss. So, so I won't accuse too quickly. Um, but learning rates on parameters, I mean, this is actually a very important notion. So, it was just slightly surprising to me that we saw that for continuous models previously, but I I didn't quite see that here in the discrete case. Um, this is important especially with respect and I've had conversations with other researchers about this. whenever you construct a POMPP model you know maybe it learns every trial and the like as we've been talking about like you know one way of making these models useful especially for empirical work and not just you know creating uh AI agents who who do things um you know for you with some kind of um where we want some kind of super performance but rather in cases where we want to fit our active inference models to to real uh you know human generated data or or or mammal, you know, mouse generated data or otherwise in in the case of observational, you know, experiments and and and being able to sort of map them and and fit them to models. um so that we can sus out if we can produce a relatively accurate model of a of a real person which then becomes incredibly relevant whenever we're thinking about doing computational psychiatry or something along those lines or any kind of computational work that is used for you know you know uh trying to accurately map uh parameters to to inference and decision-m in real people. um then suddenly this learning rate matter becomes really important because we don't always know exactly how quickly uh a person learns. So if if if we just went with what what I saw in this chapter of you know ignoring learning rates and this is just saying um you know everyone learns the same rate and every single thing they could learn is learned at the same rate. uh when in reality learning rates um and this is from talking with other researchers who who have you know um who are psychiatrists uh or otherw who who have worked with computational psychiatrists they keep finding many of the the the individual participants in various behavioral studies um whenever doing this kind of modeling and trying to fit the model they find that many people have their own respective learning rates on average even the learning rates themselves could modulate over time but but you'd at least want to be able to have the option of changing what that is or finding a way to fit that for an individual person. So, so I just I know that there are others who who uh come to active inference with with um empirical uh model fitting in mind whether it be for psychiatry, for zoologology in any case trying to model some kind of real world empirical circumstance. And so being able to acknowledge like oh there can be different learning rates um is really important otherwise you're going to be stuck with all of your models will always have the same learning rate and it's not going to account for for this kind of dynamism that we find in in empirical reality uh between different individuals. So I just kind of supplied that. Fortunately it's very easy to implement them. um you're kind of just taking the update you would have made like this is the equivalent for equation 10.4 for C that we saw before. Um you notice I I did nothing whenever I changed the notation here other than I added a small kappa learning rate, right? The c the learning rates are exactly as they were in the continuous setting that we saw before. Your learning rate is really just a scalar value, a float value. So if if your learning rate is uh and and it can not it cannot be less than um zero, right? So, so if your learning rate was zero, that means you'll never learn anything, right? That it'll always take uh what you would have added to your durishlay counts and and effectively just turn it into a zero. And so there's no update made. Um if it was a one, then you would you would be one one just adding in the event um to your dearish slay uh counts. Um but if it was less than one, then you would actually learn a little bit slower than what you're actually experiencing. if it was above a one, uh, then you'd be learning a lot faster than what you're actually experiencing. Um, so I I actually dropped a a link to a very interesting paper that perhaps not everyone who who's watching this would would find it interesting, but because it's very much uh revolves around computational psychiatry once again. Um but it kind of looks at like what what what kind of dynamics do we see in people who have these different sort of learning rates and how they sort of comprehend and learn from their experience right um and what happens when the learning rate changes and in this case it actually looks at stress disorders like PTSD again this is a little bit self- serving this my own research background um but it looks at things like you know it seems to be the case that those who have experienced some kind of you know, acute uh traumatic event such that they uh are eventually diagnosed with with post-traumatic stress disorder. It seems that there was some kind of something that's happened along the way that actually boosted their learning rate. That is even though you only went through the singular traumatic event. Um the the you computationally you might say it it's as if the learning rate was so high that that singular experience went on to dramatically impact your model. um going forward it dramatically in impacts your your decision- making and it doesn't seem that the the learning rate was you know uh a relatively you know simple or standard learning rate that would say oh well I had that one experience but I've had all these other experiences so that one that one experience shouldn't dominate my experience right but but but whenever we think about this sort of neuro neurobiologically it's very clear that someone who experiences that singular um um experience uh it ultimately goes on to to impact their their their mental generative model very strongly, right? We even see different kinds of changes and in um um you know neurom modulation and and and different regions during you know um functional connectivity studies whenever we do neuroiming of people who who who uh suffer from these kinds of disorders. So um yeah, so I'll I'll stop with the the whole bit about learning rates, but it's uh it's quite interesting. And then finally, marginal message passing. I may bring this up again in future. I I think uh I've spent a little too much time already talking about all these other things. Um marginal message passing gets its own short section later in this textbook. Um and I just it's a bit downplayed. So I didn't mean for this slide to be sort of a critique of the text of Sanjie's textbook because we we've a great effort that he's put into this you know over over half of thousand um pages you know over 500 pages here but um and it's a true collection of of of uh many concepts and mathematical equations that we've seen in active inference over time and and other fields prior to active inference. Um but marginal message passing is a is a kind of altered way or an altered algorithm for how we can do variational free energy um uh minimization uh during inference and um there there's a nice paper that was written by Fristen and Thomas Parr and others um in 2019 that compares the standard mean field based variational free energy algorithm that we've seen in the textbooks So far it compares it with back propagation which many of you who do things like deep learning and and have have constructed large language models yourself or otherwise will have are all too aware of back propagation. Um empirically there have been many findings that the brain doesn't actually seem to be doing back propagation. And furthermore, it can be very inefficient um in particular settings. And we can imagine, you know, with active inference, we're very much trying to focus on not just sort of empirically and neurobiologically how how how neuronal dynamics kind of function, how that can be sort of um you know, we can construct these comput computational substrates for those processes and these models that we're making. Um, but we're also really attending to um, you know, why why why does the brain only need roughly 200 calories a day to do the dramatic number of computations that it does every day? Why does it need such little energy uh, relative to what seems to be going on and it's it's able to accomplish so much, right? So, um, that that this where all this kind of variational inference stuff kind of comes from empirically at least. um that's why we start to to care. We're trying to model real neuronal dynamics in some way, shape or form in a reasonable manner. So um so marginal message passing is just sort of another way of doing this and it's it's it's a slightly more involved than standard variational free energy minimization. Um but it's it's much more efficient and and more realistic so to speak empirically than than back propagation. And uh more importantly, it's it's very frequently used in previous research that uses SPM. Um and it's also, you know, there's been some really nice support for doing marginal message passing. I wouldn't say comprehensive, but nearly comprehensive support for doing that in PIMDP. And so we we have these two core tools for doing discrete active inference that, you know, have prioritized marginal message passing. So it's a bit silly to me. you know, I just be frank that that it was so truncated in this textbook. Um, but of course, this is just one of many things. So, I just wanted to include a little bit of context on that. Uh, in previous simulations I've written, I've used marginal message passing. Um, that this paper makes a very good point about maybe why we should be doing that. Uh, actually, so um, you know, to each their own and and and very worthwhile to read about these things before assuming one is better than another. I'm just one researcher of many. So just sharing my thoughts on that. Um so we'll we will we'll wrap up there. Um I think that within the next couple weeks uh Frasier and or I will have more example code uh to to share and then we can actually spend some time going through it. Um, if if I if I feel confident enough, then maybe I'll just get through the rest of chapter 10 next week rather than spacing it out so much. Uh, especially because I realized I already gave a very relatively thorough redux on what factorial depth is already during this session. Um, whereas I thought I would get to it in two weeks. So um hopefully we'll we'll get into some code and um and we'll we'll get into uh what this is all about regarding uh using precisions in our discrete models uh and how that plays into decision-m and um and we'll also spend some time talking about hierarchical models which are are very interesting and I think arguably um you know probably something that someone should prioritize whenever they're doing this kind of modeling with respect back to uh empirical work, you don't always have to make a hierarchical model. Um there is value in producing simpler models. Um they tend to be a little bit more explainable or at least you can explain them more quickly and interpret them more quickly than you could a hierarchical model. Um but nonetheless um you know the brain does seem to operate at these this this kind of uh hierarchical manner where you have these different kind of almost think of different subsystems uh interacting with one another. And we've already seen how we're we're doing things like um you know p descending predictions and ascending prediction errors over the course of a hierarchy. And all of these models are kind of implicitly hierarchical and that have the our sensory receptors down here that sort of get passed up during perception to to these these sort of um inner beliefs uh and inferences that an agent makes. And then you can kind of have um this even higher order decision- making that modulates those beliefs from there. Um but nonetheless we'll we'll actually construct different layers that are something like this uh on top of one another where the higher orders kind of modulate the entire level below and so on just as we saw uh in in hierarchical predictive coding in chapter 5 and hierarchical generalized filtering in chapter 8 and the like. Um so I think we'll we'll leave it there. A quick reminder um for next Monday. Uh so I guess this is a good time for me to say this but there will be a tutorial session taught by an uh who's who's been one of the more recent core developers of the PIMDP library. Um the link is here. You'll find it in the slides. It is clickable but he'll he'll be um talking about how to implement PIMDP given the most recent updates. It'll be a nice what seems to be twohour talk. So I'm quite excited. I think they'll be a really nice resource for us to have going forward. Um, and uh, I'll probably share the link to that again uh, in future. Uh, and then another quick reference to Ryan Smith's previous tutorial which um, is going over POMTP models as we have been. It's a little bit more of a classic tutorial in the field. And um, and then whenever we finish chapter 10, I'll we'll sort of bring it all back together. We'll look at like kind of what a full active perception loop looks like for these discrete state space models just as we had similar algorithms for the continuous state space models. And if we have time, I would like to talk a little bit about how many of these precisions and parameters that we're looking at do have at least uh puditively um you know some theoretical or evidence-based uh support uh for uh kind of relating these parameters and their precisions to neurotransmitters. So I think anyone who is more interested in empirical work would find that very interesting. Um for example, next week we'll get into precision on policies. Um frequently denoted by this this term gamma. Um this this this is actually rather well established. So many people are doing outside of active inference and inside of active inference been doing research on dopamine. It's very interesting neurotransmitter. people most uh uh frequently uh you know if they come across things about neurotransmission one of the core ways I think they get into it by through social media or or googling around or otherwise they quickly see this word dopamine and want to know what that's all about but it it gets very much related to decision-m and gets uh related to reward processing uh other neurotransmitters do as well so it's it's not quite enough to say it's only dopamine that does this uh but but it's something that's very interesting and and we we'll actually be able to see how dopamine this supposed dopamine can be modulated within our our POMDP model. It'll relate very much to the agent's notion of of confidence and its own decision making and kind of go goal directed decisionm as opposed to simply relying upon habits when it doesn't know what to do in reference to the the E I was talking about earlier these kind of contextfree habits. So um yeah, many thanks for for sharing those articles once again. Um so I think we'll we'll leave it there and and uh we'll we'll probably have more time for for discussion and the like um whenever we approach the end of chapter 10. We might just do an extra week or something along those lines to to have more time for everyone to be able to discuss this because I know we've been given a lot on these discrete state space models all at one time. So, um, thank you so much everyone. We'll we'll we'll stop there and we'll we'll see you at the next one. Thank you.