Fundamentals of Active Inference (Chapter 9, Session 44) September 11, 2026

All right. Hello everyone. So it's uh 11th of September 2026. We're here in active inference uh the reading grip on Sanjiv Banoshi's new book on active inference. We are concluding chapter nine today. Uh so that's the first chapter in part two of the book where we're going into the discrete state space formalism of active inference and we've been walking through what that means kind of getting on uh the same footing with respect to the sort of new mathematical uh machinery we've been introduced to with with the uh you know ma matrix and vector-based um categorical distributions and things like that. We've done all of that now and now we're really uh doing active inference proper with with these kinds of models. Um, so today we're gonna basically loosely we've got 9.5 to go through. That's the biggest section in in the whole chapter and it's the the most meaty and then 9.6 and 9.7. I've prepared another code notebook to go through. Uh, it ex it instantiates three of the examples. I think it'll be good to to have a look at that. It's available now on the page. So I can if I share my screen I'll make sure to share my entire screen this time rather than uh just separately. So, you should be seeing the whole thing. I'll just hide you guys for a sec. There we go. Come back to the page. Whoops. Just pop out here. Okay. Here's a page for chapter 9. Um, I've also just while I'm at it, if we come down to the uh the chapter map, we have all of chapter 9 is is here. Now, in terms of the notes, so my notes for 9.5 and 9.6 and 9.7, they're there. Um as I say chapter n well n section 9.5 is quite long. So this the notes are are quite long as well. The examples so the the code examples from last week that I instantiated they're there as well as a code notebook and the new ones are here as well. But I suppose the most um the best sort of way to navigate to the code examples is you come across here into the to the menu. You scroll down until you see this little code thing. Click there. Now, this is kind of where we're hosting the the collab stuff as well as the actual repo. So, this is kind of the the collab notebook side of things. You can just go into each each chapter here for chapter 9. Click on this here. This will open it up. There's the example from last week. Um, what I'll do is I've got this this one for this week where we look at examples 9.6, 9.7, and 9.8. And there's a few extra things to play around with. What I would recommend is you you open it. Well, I'll open it in a second. You go to file and save a copy in drive first and that will actually save a copy to your Google Drive and then you can play around with that to your heart's content. Uh you you can just directly run the example too. There's there's you can there's no need to do that. But if you did want to play around with that, that would be the way to go. Eventually, I think we'll have these things in the actual git uh GitHub repo. a bit more of a standard way to do things and share. But you definitely can just, you know, plug and play. You can just open this up immediately. Let things load a bit. And you can come down here and you can just immediately click the play button. Uh or I think it's control enter and that will start running code and you can just run stuff um and and see what's happening. So there's no need to download anything or anything like that. Sometimes the first cell takes a bit of time. There we go. It's done. Uh and that's how you can interface with with the examples. So, I'll stop this just for a second leave and we'll come back. The if I go to the page again for chapter 9, the only other thing as well is that the figures for all of chapter 9 are there. There's only a couple really to um to show, but they're all that's complete now. So, all of the all of the figures we'd like to see are there. So I think I will get stuck into 9.5. Uh and we will begin talking about effectively what we're going to be talking about today is planning and um action selection by means of planning and the expected free energy. And these are kind of like the the crown jewel aspects of active inference in some sense is this ability to do planning uh to consider rollouts and to to evaluate the expected free energy and on the basis of that select actions. that's kind of quintessentially active inference like a lot, you know, I tend to think at least. So that's that's very very exciting. To get there though, let's go to just I got my notes on 9.5 here. Hopefully everyone can see this. Um, you know, last time we we looked at um we have we have a notion of a hidden markoff model uh which is to say a discrete state space model and with this there state transitions across time. Uh and that's great. We left off talking about how we want to transition this hidden markoff model into a partially observable markoff decision process. Say that 10 times fast. What that means is basically we've got the ability to choose between state transitions. Okay, we we we can actually do something. We can issue an action and that will change the that will influence the state transitions uh that we have. So I've shown this figure before but if we go here what we what that now means is that so we've got our observation likelihood matrix our A matrix we have our B matrix which tells us the probability of ending up in a subsequent state conditioned on the current state now conditioned on the current state and action. Okay, so really what we end up with is this kind of B tensor thing where we've got a matrix or a B matrix for for each potential action we could take and the running example throughout this section for all of uh the remainder of the chapter is this kind of grid world environment this here. So really really simple it's just a 3x3 grid uh and you can be in one of the states or not and if you're in a state you can get a particular observation and of course that observation is given by the the likelihood matrix the A matrix. So, that's the setup. That's sort of where we left off. Um, and we looked at notions of how to uh score uh or sorry, how to move around in this in this situation here using policies, which is to say sequences of actions. Uh, so the idea is that we're going to have to select an action. We're going to have to select a policy that allows us to influence state transitions in the manner that we would like the state transitions to be influenced uh to be aligned with our prior preferences. We're going to get to that in in just a second. And we just assumed that we had good policies. But now we're going to actually explain how do we get how do we find good policies? Basically, that's the hard part. Um so that's what we're going to have a look at just now. In order to get there, I'll come back up to my notes in 9.5. We're basically going to have to slightly alter the generative model in some sense. So again, we're starting out bases rule. That's the north star. We have our observation likelihood a matrix. We have our state prior belief, our D vector. And we know that the troublesome uh model evidence is is there as well. Of course, the generative model is is just the factorization of the well, it is precisely the um uh the state prior and the observation likelihood. I've got a typo here. That should be uh P of P of S, sorry. Um, yes. Ah, no, sorry. Rearranging. If we look at so 9.54A, if we just if we ignore the the central term, if we multiply both sides by P of O, then the generative model is equal to the state uh the the conditional distribution on states given observations times the model evidence times the probability of an observation. Okay, that's one that's the first sort of uh step we're going to be taking in trying to think about how do we get the ability to have preferred observations encoded in our model. Okay, this is step number one. So the idea is okay, let's replace that ordinary marginal distribution over observations with one that's a little bit biased in some respect. And the idea is that this biased one, this is going to encode, it's going to uh simply be a distribution over the observations that we would prefer to observe basically. Um, and we're going to notate that it's a bit hard to see. In the book, we notate that with a little kind of brief symbol, this kind of upside down cresant thing that is saying that yes, this is a this is a marginal distribution over observations, but it's specifically a a preference distribution. It's it's saying these are the observations I would prefer to observe. It's not just a regular old marginal. Uh and basically that consists in adding a bias to our model. The idea is that the model is not just kind of honestly reporting what the observation marginalss would be. We have preferences now. So if we can substitute that in to we can just directly substitute that in here. Um and that will turn our model P given P and S into a bias generative model. And that's effectively where preferences kind of come in to our our model. Uh now we're not quite done, are we? Because we still have the troublesome true posterior distribution here, the the the distribution of states given observations. And we know that we can't contri we we can't calculate that. Um but of course we've looked at everything from chapter 4 uh in the continuous states base formulas. We know that we introduce a variational posterior here, an approximate posterior over states. And that's what we uh optimize by means of variation or for energy minimization. So okay well that allows us then to say our bias generative model is equal to this preference distribution over observations uh times the approximate posterior over states. And we know that that sort of works because the approximate posterior over states is approximately the true posterior. Um so this is now how we're going to talk about our generative model. basically uh specifically it's a future joint uh density over preferred future outcomes uh and predicted future states. Okay, so these are what outcomes I would like to observe and these are the states that I'm predicting I am in. And then obviously with a notion about the observation likelihood matrix I can I can think about predicted observations as well. So this there's a whole bunch of um you know ways of there's a lot of uh kind of representation of this in terms of the model. Uh so now now we're talking about we are talking about trajectories or sequences of observations, sequences of states and now we've also got actions in the mix and we're going to have to talk about sequences of actions. Uh they're also part of our generative model. the way that actions are talked about or the way that sequences of actions are are termed is is as a policy. So we have the way that Sanjie has chosen to notate this in in the book and he he says in a footnote here he's always going to show in the generative model a policy which is to say pi and that's going to just be a sequence of actions. So this pi thing here is just like a sequence of states or sequence of observations but it's a sequence of actions now. Okay. And it's just not showing uh all of the actions there. Uh this if we actually look at what this looks like now, we can have a look at on the next page, figure 9.2, this thing. This is what I showed uh last week. This is now a partially observable Markov decision process. So we have, if we ignore the top half here, we've got our sort of hidden markoff model, if you will, of states transitioning across time, states giving rise to observations. But now the state transitions are not merely the consequence of the previous state. They're also influenced by the policy the action that I'm taking on in in the current state and the the current action. So this policy thing, this is the thing that's going to allow us to and we're going to select policies. There's we have a choice about what policies we're going to enact, what sequences of actions we'd like to enact in the future. And that's going to determine our state transitions effectively. And then the question becomes how do we select good policies? That's that's the question and that really is the all of the remainder of the chapter is about that question. How do we select good policies effectively? Uh basically to spoil the game what we're going to see is that there's another quantity a little bit like the VF. It's a different free energy called the expected free energy notated with G. And by minimizing that we're going to get policies and our preferences are going to also influence the EF is going to influence selection of policies. Policies are going to select state transitions and then we can actually play the game of active inference. So that is where we are going. Uh I just I skipped over a few things here in the notes. So okay. Okay, so we've seen how to uh augment our generative model so as to be able to encode preferences over observations. And I must stress sometimes in the literature you'll see um preferences uh encoded over states, but it's overwhelmingly the case that people tend to express preferences over observations. And there's interesting questions there about whether one is better than the other in certain circumstances. Usually preferences are are defined over observations. That's that's very very common. Okay. the and because we're working with a discrete state space model here, we're going to encode our preferences as a vector basically um like like this here. So we have preferences for certain observations basically. So this is kind of like an observation vector but now it's saying I would like that index that uh that index and my observation vector and you know so we've got a we've got b now and d now we have c we have an additional vector that's part of the generative model now and this is quite literally just our our preference vector over observations in in most of the examples in the chapter the um but this is a different point now the the observation likelihood mapping is just the identity matrix So it's just a table or matrix with ones on the diagonal and zeros everywhere else. And that basically means that there's no uncertainty at all about what state I'm in given an observation because if I observe something that tells me unambiguously what's happening. So that's a that's a simplified I just wanted to note that that's a bit of a simplification and in some of the code examples I I remove that that um requirement. So there's some discussion I I talk about sort of the differences or between logits and and actual probabilities and things like that. You can read that in your own time if you want. So the whole generative model uh just coming back again we have a joint distribution over sequences of observations, sequences of states and now sequences of actions or policies. That's that's what we call sequences of actions. Just want to note really briefly for those who are familiar with reinforcement learning, this is a different notion of policy than is used in reinforcement learning. Um, reinforcement learning typically talks about a state to action or an action to state mapping as like a function. That's not what we're talking about here. We're just we just have a sequence of of actions and that's what we're calling a policy. So don't don't get those two things confused. Just a few things to note as well. You'll see that this is all all of the time indices here are now towels. Okay, they are um we're now maintaining a generative model over sequences of things into the future basically up to some time horizon big t okay so tow equals zero and uh Andrew had some some good discussion about you know choices around notation for that in his session do go and watch watch that for for a bit more of an in-depth thing there we covered that we covered the difference between toao and and t t and that sort of thing um very nice so that's like what do we have fundamentally we have our our observation likelihood mapping A that's our A matrix. We have our B tensor. Now this is the state transition function. Uh it's a function of both the well previous state and and and action. Um and then we have prior over so we've got our our D. Okay. So we have D a prior over hidden states. Where do we initially start out? A B and we also have a prior over policies now sequences of action. So that's really the only sort of new thing that's that's come in here and this is going to be sort of the form of the the general model that we're going to use going forward. >> Just a quick question um fra uh equation 949 to be consistent with um Sanjie's layout earlier. Shouldn't that uh capital T be changed to a capital H? >> Uh uh very possibly actually. Yes. Uh because he sometimes uses H for horizon. If I just find 49 here in the in the book itself. Yes, I suppose. So, uh so I mean he does use capital T. Um I think he the notation slips sometimes between you know saying all right I've got some simulation that's running for some amount of time. Let's call that big T. Sometimes he also says consider big T to be the time horizon. Sometimes he uses H. So, I suppose probably it would be best to use capital H here um if we're being precise, but as long as we're consistent, it doesn't matter. But, uh >> yeah, probably probably would be best. So, >> just want to say >> yeah, that's a good question. And uh with respect to the term on the left hand side cuz it the way this is written this is I think it's supposed to be referring to the entire trajectory over the course of the simulation is if we're essentially where POMDPS come from is that we're we're trying to like kind of quote unquote solve PMDPS including like over entire beyond the model own time horizon. um we're looking at like a global time in which case we would use capital t um so I think here we would be using the capital t because that's the way that the the sort of problem to be solved is set up in this particular equation but >> that's a good point it probably it probably is the way to go I'm just having a look in the appendix because of course all of those uh so like appendix C or rather appendix B is all the sort of uh indexing notations Uh, I he might talk about that. I think I can find it briefly. There's a lot of indices and so on here. Um, yeah, I might not be able to find it in time. Yeah, I mean, yeah. So, I I think I probably I you've you've convinced me uh Andrew, it's probably because this is like the entire simulation time, you know, and within there maybe, you know, the entire time of the simulation is 100, but I'm only ever looking ahead five time steps at any given time. That would be like H capital H within that sort of window as we slide that along. So yeah. All right. Um let's go now to so we've seen the structure of the model. The thing is we want to be able to select these policies now and how do we do that? That's that's the crucial part. So uh and well the actual posterior of interest is the posterior over states and policies given observations. Now this is the thing we would like to be able to find. Uh basically so you know typically we've only had the posterior of states given observations. Now we have the policies as well. Those are things that we're trying to find. Have we we have beliefs over them in other words and the only thing that we observe are the only thing that we're allowed to observe are the observations. That's that's the the evidence that we get. So what is introduced is uh effectively 9.51. uh we need to have a variational we're going to we're going to be doing variational free engine minimization. So we're going to have a variational posterior. Now we're just going to introduce Q of S and pi and we're going to say well look let's assume that this factorizes as the you know the probability of my my sequence of states given my sequence of actions and a prior variational posterior over uh policies over sequences of actions. That's the new sort of variational posterior we're going to use. So really that looks like this across time. So these are sequences here. We need a we need a a prior belief about policies and we need our conditional belief about states given policies and that's going to approximately give us the the posterior that we want in this case over uh states and actions. That's what we need. So how do we get those two things basically? Uh so there's basically two kinds of variational beliefs. the in order to get there I just want to pause a second and come over to the notebook itself because this is where things are actually going to start kicking off. So the code let's go back to here chapter 9 I'll open it up in collab because that's what everyone will actually see. So I recommend you yourself go file and then save a copy and drive this one here and then you can play around with your heart's content. The first example here is actually 9.6 from uh the previous section where uh we're just being able we're just moving around in grid world. Um I just want to show you that because we're going to be using that for everything else that we do in in this session. So maybe I'll make things a little bit larger. Uh the setup is we've got a 3x3 good world. You can make it as big as you like. Our indices going to be zero based index. So we have state zero, state one, state two. That's how our things are indexed. And we have our four actions that we can take. 0 one two three up down left right um and we know that we have our B tensor now which is this 9 by9 by4 block of of numbers that's literally this thing here uh quite literally that okay uh coming back coming back and the observation likelihood is just going to be the identity matrix here just for simplicity so I don't want to I don't want to explain the code but basically I can construct this I make this a bit smaller uh with some some helpers here. So we can construct our our A matrix, our B our B tensor um to encode this this little environment here. So if I I can show you what that looks like. The coloring here is is just showing you the different states. Okay, so we've got our 3x3 grid world. Um and I've just colored things according to states. So and I've got some information about how to read uh stuff here to be consistent with the book as well. So, where we left off last session is how do we actually roll out? You know, I've got a policy. Where does that take me? Basically, that's kind of where we left. And now, now we're going to how do we select policies? So, we we start out in the central square. Let's just pretend we start out in number four here. Uh, and we've got a policy to go up and left. So, this is a sequence of length two of actions zero and one, zero and one or up and left. Whatever the whatever the encoding is, doesn't matter. Well, zero and two. There we go. That's that's up on left. Where does that take me? What happens when when I when I do this? Basically is the question we can ask. Um, yes. So, and we'll get to the question about how do we select policies? So, what I need to do is roll out. So, the verbiage here is I've got a policy and I want to roll it out um to predict what happens as a consequence of that policy essentially. So, I've got a belief, I've got a policy, I've got an observation likelihood and I've got a a B tensor. I can then predict what happens as a consequence of applying that policy in my model. So that's what this little function here does. I don't want to go through it in too much detail. So it's very simple. It's only two length policy. So the most probable state after each time uh well actually each belief time is four one and zero. So there's two steps I'm making here. You know if I go up my predicted state is one. If I go left my predicted state is zero. Very nice. So that's the same information here. There's a bit of information about how to read those as well. It gets a bit tricky with um all the all the indexing. It's kind of crazy. So this is what this looks like. I've got my policy and I roll it out. Um I and this in this case the observation likelihood mapping is unambiguous. So I go I I I say I want to go up and left. I actually do end up going up and left. So and yeah, just around indices. So I guess like little t would be an index in total time. Um so if I'm at here then this is now and then this is one time step into the future and this is the other time step into the future. So typically we use little t for stuff that's happening now or total time and we use tower for future beliefs. So all right that's where we we left off. I also have a little bit of an extension here where we look at uncertain likelihood mappings. But now we want to think how do we how do we actually get to decide these policies? That's the the the fundamental thing. So let's come back here and this is this is the center of the the entire chapter and in some sense it's the center of all of active imperence. So I would recommend you pay attention. So let's come here now. Basically the so and okay so coming back again we need these two beliefs. We need our prior belief over policies and we need our our state belief conditioned on policies. Those are the two things we need. So VF that gives us beliefs about current time and and even previous time but we need prospective stuff. We need to look into the future roll out uh policies score them in some sense and then choose a policy to enact now basically that's what we need. So and again we've got tow greater than uh the current time. So VF is used to infer you know the hidden causes of observations that have been received. We've done that to to death. We've done that a lot. We're going to now introduce another quantity called EF expected free energy and that is used to score possible futures whose observations have not yet been received. So if I come back to this figure here in the actual pom dp if if I imagine this is uh imagine we're currently here at this time step technically all of these observations here they should not be shaded in gray because that denotes that we're observing that particular uh thing. Uh the thing is observations in the future haven't happened yet. So we actually are going to be uncertain about these future observations. back back. Now, there's an enormous amount of and I'll talk about this at the end of the session. There's enormous amount of discussion to be had about where the expected free energy comes from. But for now, really all we need to think about, all we need to regard it as is a way of scoring potential policies that we're going to roll out in order to be able to select an action now to perform that's going to in some sense in the long run be adaptive. Okay. And there's it's going to take a while to get used to that, but um it's it's it is a bit of a a doozy. So the expected free energy, it's basically a score on policies in some sense. So uh it's it's not an observed data loss function in the same sense as the VF. It's an information theoretic score assigned to a policy's predicted future consequences. And he points to the appendix, I think, uh C10 if you want to go through a bit more of the information theory there. So basically we've got pi we've got a potential policy that's going to predict future state transitions much like I just showed you in the the code and we're going to be able to then work out the EF the expected free energy of that policy. So there's lots of decompositions of EF. The one we're going to focus on is called the RO decomposition. The risk and ambiguity or expected ambiguity decomposition. There's many other decompositions um much like there is for VF. I would recommend uh in addition to reading this section to actually reading the the original book as well. Section 2.7 in here uh goes through to the end of the chapter goes through EF what it is where it comes from. It's only about eight uh pages 2.7 um I would strongly recommend this as as compliment and Andrew has done many many sessions on on that book. So all right we'll start from from here. What have we got? We basically got two terms. We have this thing called risk and we have ambiguity and these are predicable of policies. So what are risk and what is ambiguity? Uh well we know what the KL divergence is. So the idea is that risk is going to be the the divergence between the predicted observation likelihood given policies. So you know some length of time step into the future what are the observations I'm going to get if I follow a policy. It's the difference between that and my pref preferred observations basically. So it's like well what is the difference between what I would prefer and what I predict in some sense in a loose sense that's going to be called risk basically and then the other term in the EF is this thing called ambiguity or rather it's expected ambiguity and the idea here is that the I I can predict what state I'm going to be in in the future conditioned on a policy much like I can for predicting what observation I'm going to be in given a policy and even if I know what state I'm in. The ambiguity is is asking how uncertain am I about the observations I would get in that state. Basically, so the classic example is, you know, if I flip a coin, okay, what's the probability I get heads? I observe heads conditioned on the the weather being sudden. Well, that would be maximally ambiguous because there's no relationship between uh that state and the observation that I get basically. So colloally, you know, this this expected ambiguity thing. It's a measure of how uncertain I am about the observations I would get if I'm in a particular state. And we've already talked about sort of relationships between uncertainty and entropy. As you can see, it is in entropy. So we've got this EF, we've got this function that scores policies. It's main that the fundamental terms are the risk and the ambiguity. Um both terms are non- negative on standard definitions. And the idea is I have to predict what's I have to predict policies and I have to rank them with with the EF effectively. So I go in here in terms of I explain effectively what I just did in terms of where the risk comes from maybe some intuitions about that. Uh the same for the for the ambiguity as well. The really cool thing is well let's uh let's there there's one little bit here and then in terms of theory and then we can actually get to the examples. Uh the cool thing that this gives you that is very very excellent in the sort of active inference formalism is that we now have a way to kind of balance there's there's a canonical problem that happens when you're an agent. Basically exploitation uh versus exploration. So if you're an agent and of course active inference as many people think is a theory of agency. A lot of the time you end up in this dilemma here where you you've got only so many resources, so much time, so much um energy. Uh do you go and do something now to act on stuff that you know is probably going to be good for you or do you try and seek out new and better states, new and better things? It's it's very unclear how to how to how to do this, how to satisfy this trade-off. It's common trade-off that happens for basically all agents. The idea is that the EF is going to give us in some sense a principled way of balancing this trade-off and you can kind of see it in the structure of the EF itself. So you might analogize risk to uh like how this is in some sense a notion of reward. So my my prior preferences you can kind of think about them as encoding a certain kind of reward. So that's a sort of pragmatic valuable state of affairs I would like to be in and I would my my risk is saying I would like to minimize the difference between my predicted observations and my preferred observations. That's the sort of pragmatic side of things. And ambiguity is I would like to resolve my uncertainty about how observations are generated from from hidden states basically. And indeed there are decompositions. Another very common decomposition is uh pragmatic value and information gain. Very similar kind of kind of thing here. So like really generally this difference is the difference between pragmatic and epistemic value. uh over here there's lots of papers on uh epistemic value and active inference. Uh there's actually too many really to to share active inference and epistemic value. The idea is uh let's come back to the example that uh we well suffice it to say that the idea is that there are different perspectives from which we can look at this general dichotomy that the EF gives us and the dichotomy is largely between how do I satisfy my my my goals and how do how do I resolve uncertainty about the states or the observations and so on. That's sort of what the EF gives us. And it gives us, if we can minimize it, it gives us a sort of natural way of being able to dance the or you know, balance between the imperative of of pragmatic and epistemic behavior. So that's in some sense why it's very valuable for us as as cognitive modelers and things like that. So all right, so we've done the risk, we've done the ambiguity, there's there's some mathematical yoga to be had there. Basically, this can all be expressed. Of course, we're in the we're in the realm of uh categorical distributions now. So, we need to express this in terms of the language of matrices and vectors and that kind of thing, linear algebra, and indeed we can do that. So, the there are expressions that he gives for uh the matrix form of the RO risk and ambiguity version of the EF. Um there's a little bit of bookkeeping that needs to be uh looked at here in terms of the indices. So the idea is we we need the EF for a certain policy and there's going to be so many policies who knows how many. Okay. And then of course for each policy there's going to be a point in time in which that action belongs to that policy. So we're going to use P to index policies and TOAO to index time in a policy. So that actually looks like this. We've seen this before. So if we've got a bunch of policies in this case four policies are made they are sequences of actions going across columns like this. So you know this would be tower this would be sort of now one step from now another step from now and so on. So we're going to use P to index policies and tower to index time in a policy. Righty righty come back here. So we've almost got to the point where we can score policies and we can choose an action because really at the end of the day we need to choose an action to perform now right now. uh and we haven't quite got to the end of that story just yet. So we know we like which is to say that we haven't yet told you how to get your prior over policies just yet. That's the the crucial thing that you actually need to do all these calculations. So take it for granted that we can represent the EF risk and ambiguity form in matrix vector notation. That's very good. So we can actually evaluate stuff. The idea is that this this is going to be the EF for one policy, one time point in one policy. Okay? So if I had five actions in a policy, I need to do this for each uh each time point in that policy. That's that's what's going to happen very basically. So there's I I do go into sort of great pains talking about the indices here. It gets a bit hard to to to go through all of that. Don't get too worked up about it. But the idea is that we've got the EF. It's evaluated per time step. Basically, what we need is the EF for the entire policy. And that's just going to be the sum of the EFs across the entire planning horizon that we happen to care about. So this is where the H comes in that Kobus was talking about before. I've got some policy. It's H timesteps long. Maybe five, maybe 10. Don't know. It's kind of up to you. So that's, you know, tow indexing T+1 up to T plus H. And I need to sum the EFs across those time indices. Basically the expressions for those EFs are given above that gives me the EF for a particular policy essentially. Now how do I so that's that that is a a score about a policy in terms of you know for the whole horizon length. How do I turn that into a prior on policies? I need a prior about which policies I want to enact effectively. And really at the end of the day, I need to know what action to pursue right now. So I'm going to get that from my prior over over policies. And the idea is you get it from a softmax of the negative of the EF. So what what does that mean? Basically very similarly to how the EF or how the VF minimizing VF makes is better. that gives a better approximation to my approximate posterior minimizing the EFE expected for energy is going to be better. It's going to give me uh better prior on policies uh out into the future. So what does that mean? Let's say that I have policy one and policy 2 and the EF for policy one is less than the EF for policy 2. That means etc parabus all things being equal policy one is better than policy 2. How do I turn that into a probability? Well, if I take uh if I take the negative of this and I put it through the softmax function, that will give me a normalized uh value that I can use as a probability effectively. So that's saying that if I do that, that's saying that the probability that I choose policy one is greater than the probability that I choose policy 2 because the EF of one was less than that of two. And so I want I want my posterior I want my prior on um policies to be in proportion to the negative expected free energy. I want to choose policies that all all things being equal have less uh nef than than the other ones. That's kind of where this comes from. There's more to be said about that, but that that kind of ties the bow and everything that we need in order to be to be able to do rollouts that now extend our model by looking at stuff that's happening in the future effectively. So last little bit, I I will come back. We're kind of running short on time. That brings us to example 9.7. So reward and information seeking. Uh there's not a lot to see effectively, but we're we're still operating within the grid world scenario. So part three expected for engine machinery. So I go through here and I sort of talk I talk about all the stuff that we just talked about. I then have functions to instantiate things like the entropy, the likelihood of observations, things like this, my prior preferences. Uh and then I have a function to package up the EF in terms of the linear algebraic expressions that we just saw. So if I run that, that will give me everything I need. So example 9.7A, really simple. We're just going to imagine we got two policies. Okay. And we're going to look one time step ahead in the future. So really really simple. Um if we imagine my predicted hidden state beliefs under the two policies are as follows. So I have one hidden state predicted prior that's very very uniform totally uniform and I have another one that's very skewed. Um and I have a particular likelihood matrix. And let's assume that my prior preferences there's only there's only two possible observations. I I want the the zeroth or the first uh observation. Um all right. What happens now? Basically, so I I I maybe need to come back and change the name of some of these things. Basically, I've got my A matrix. I have my two uh my my two prior beliefs over policies. I'm just assuming I have them at the moment. I have my reward, my prior preference vector, my C vector. I can go through and say well the predicted observations were such like for the first policy the predicted observations were such like for the other policy what was the risk for the first and what was the risk for the second so you can see the risk for the first policy 6.33 something nets much smaller than the risk for the other policy the idea is that's saying that all things being equal I would like uh my first policy is that right my first one Yeah, basically there's much less risk associated with me taking the first policy than there is with the second one. So set parabus I would like to take it. That's just risk though. So and I can kind of look at that visualize that. So I can plot the risk associated with those two policies. Um now I can do the same thing for information seeking and and ambiguity basically. So again the same likelihood matrix. Um oh no sorry we we changed the likelihood model. So over here up here this likelihood model is relatively precise. It's close to an identity matrix kind of you know very very strong diagonal entries relatively weak off diagonal entries. So that's giving me relatively precise information about states given observations. If I now have a much more kind of skewed observation likelihood matrix there's now more ambiguity about the uh the observations given the states basically. If we run this here, the we can evaluate the ambiguity associated with the two possible policies. Uh, and now they're they're relatively similar basically. So there's there's much less um well there's more ambiguity associated with these two policies in this particular example because of the the observation likelihood mapping. So policy one predicts you know very slightly lower ambiguity observations. So if you were just evaluating things on the basis of ambiguity, that's the one you would take. So again, we can we can visualize that this one here. Uh yeah, these are two separate demonstrations in terms of just risk. We we just looked at risk individually. We just looked at ambiguity individually. But now really what we want to do is we want to put them both together and look at the EF and total essentially. So we want to be able to take we want to be able to construct prior over policies in proportion to the soft max of the negative of the expected free energy and that's what we do here. So I have I have a a way of enumerating policies you know kind of rolling out in some not rolling out precisely I have a way of evaluating my policies uh and then I can well I can score them with the EF and get my my action marginalss. So that brings us to well slightly before I do that I'll come back and I'll share what we just looked at for example 9.7 which is this here. So this is figure 9.21 to one. The idea is that this is showing you visually, okay, well, we've got just a belief over hidden states, belief over observations. And in this case, um the separate the the kind of relative spread of this belief is the entropy. So, this one over here has a has a lower or sorry, a higher entropy because it's more spread out inverted commas than this one over here. That's hopefully what I what I just uh showed you in the code. Now, are there any questions at this point? Because we're almost at the end. The last thing we need is the notion of action selection. We've kind of built up everything that we need so far to do that. I'll just stop sharing, but are there any questions thus far? I see the chat's been quite active. Very good. Yeah, I could say a lot here, but I want to make sure that there's time for you to get to that that final piece. Um but yeah that a lot of this um you know a lot of a distinction in active inference that I think is worth noting is that uh in comparison to a lot of other methods that have been employed in like RL agents there's this question of exploration versus exploitation which uh Sanjief gets into and um the one of the issues is how do you have agents who have some sort of natural drive to explore and not just exploit. Um it's been a classic issue that agents who only focus upon what we're calling pragmatic value here or reducing risk in terms of what we're seeing in the textbook. Basically agents who would be purely reward driven. Um that those agents tend to sort of move as quickly as possible to reward. Uh we know that biological organisms don't usually work that way. uh especially in cases where um you whenever you're only reward driven, you're not taking into account the uncertainty within your model. And so you can have agents who sort of just head towards what they think is the the best strategy without consideration over what it's unsure about. So you can have agents who kind of uh stick to very suboptimal um solutions. And so whenever we think about sort of the way we phrase things in gradient descent where we're trying to uh reach some kind of ideally global minima here uh rather than sort of suboptimal uh min you know minima or getting caught in a local minima uh by being able to have some principled means for exploration uh in addition to reward seeking at least to agents who typically learn more about their environment while also paying mind to rewards. So there's this sort of famous claim that active inference quote unquote naturally balances exploration and exploitation here naturally balancing risk and ambiguity. Um that that sentence itself has been uh interpreted in a variety of ways but just in reference to a question in the a chat um what the a the benefit of of active inference here is that we're we're having agents who are considering both reward and seeking more information. So you could have policies that are evaluated to have very low risk and very low ambiguity. In which case that's that's great. That means you'll have a lot of learning opportunities while still being able to pursue reward at one time. But then you can have a situation where one of those terms is high, the other is low. Vice versa, you could have both of them being very high and so low reward and no learning. That'll be a very poor policy. So it's just u and then the ambiguity is is is is it's great in the sense of you know it's directly referencing the uncertainty within the agents model um whereas a lot of other RL methods I always pick on this one called epsilon greedy but um you know there have been many advancements since then but the idea of epsilon greedy is you have purely reward driven agents except you add on this little epsilon value say it's a 05 saying like 5% of the time just do something random. Don't do what you think the best rewarddriven policy is. And that's not taking into account like why do something random? Like is doing something random good? If you want to, you know, I don't know, uh if you want to accomplish a goal, uh you don't just suddenly do something random to to try and get out of a a local minimum, right? Like you you will do things that maybe relate to the uncertainty in your model. If you want to learn how to swim, uh you won't just hold your breath forever, right? That is a random thing to see if that helps you to swim, right? You you'll do things that relate to what you already know, but also the remaining ambiguities. Do I move my arm this way or that way in order to tread water, right? So, that's that's sort of the trick there. And so, uh it's it is a strong benefit that that active inference kind of brings to the table. Anyway, I've been I've been going on for some time. Please raise for your >> No, thank thank you, Andrew. There is an enormous amount to say here. So it's it's going to be tricky to to to limit ourselves with respect to everything that we could say. Basically what we're going to try and do is we're going to get to now this is example 9.7. We're going to do exactly EF minimization in order to select actions for one-step look ahead. So our plan is just going to be one step into the future basically. And if we imagine basically we've got 0 1 2 3 4 5 six right we're going to be in this current state down here at number six. And let's just imagine that we have a preference to be at state number seven basically. So there's one, two, three, kind of four actions that are available to me here. Of course, if I go down here, that's not going to do anything. If I go left here, that's not going to do anything either. Uh really the only state, and if I go up, that's just not going to bring me to state seven. If I go right, that's the only one of these policies out of the four that's going to get me to my um my desired state, which means that uh I can evaluate the EF with respect to these potential policies and they're all just length one plans, which is to say they are just actions. Uh and then I can build up my prior over policies, which is going to be really simple. In this case, all of the probability mass is going to be on move, right? Okay, so in code, what does that look like? Part seven, example 9.8a, onestep grid world. That's that's what we have here. I start out by saying let's let's start in in state six. My goal is state seven. Um I have a prior belief on where I am now. This is actually subsuming the D vector here. Um and I have my prior belief about the goals. So these are just one hot encoded vectors basically. And my policies I enumerate all of the policies of horizon length one. So that's really just actions. And how many actions am I choosing? In this case, I've got n actions defined as four. There's four possible ones here. I have my function to evaluate policies that steps through and and does the EF calculation with respect to all of those policies. Basically um and then I can say for each policy uh for each record and for each probability print all that out. If I do that this is actually what you were seeing in that figure just now. So action up the expected state I would be in is three because if I go here I go to 0 one two three. All right. and the the ambiguity the the um likelihood matrix is is is totally uh identical here. So I'm always going to be where I expect to be in this case. You see that the risk associated with everything except for go right is 36.84 knats. Okay. So and the ambiguity is zero for everything because there's no uncertainty with respect to the observation likelihood. It's just an identity matrix. And the the risk associated with going right is zero because that gets me to where I want to go. I want to go to to state number seven. This one here. Bada bing, bada boom. So if I transition to there, there's no risk. I I get exactly what I I predict that I would get exactly what I want to get. So there's no risk associated there at all. Perfect. I'm under. And then on the basis of those um valuations for I guess ambiguity and risk, I can build up my prior over policies. Uh so and it's just a categorical distribution over actions. All right. And that is exactly just a one everything probability mass all of the probability mass on go right and that's what you're seeing here. So that's like the simplest possible case to show how it is that um my prior over policies can be built up from evaluations of EF. Okay, this is just a onestep look ahead. There's only four actions. It's a tiny state space. Very very simple. But that's hopefully uh that if you kind of think about this, play around with it in your own time, that can uh get you to the the intuition that you need. This culminates then in algorithm 18. We're almost done. I know we're we're out of time. So what what I just did in the code is I walked through I should have shown you this before. I walked through equations 9.6A and 9.6b. So this is really the this is the the crux. This is the end of the the the road. I need my I need a poly from what action I'm going to take right now basically and for our purposes that's going to be the sum over all policies given the predictive prior of uh actions given a policy and my prior belief on policies I'm waiting my prior belief on policies by this predictive prior on on actions given policies so this thing I get through EF minimization that's what we just uh just saw And there's there's a lot of mathematical machinery to be looked at there in terms of how you can encode that with chronic delta distributions and things like this. So that gets us then so this I actually do walk through example 9.7. We don't have to look at that. That culminates in there there's more material here section 9.5.2. So I go through all of those steps I just showed you in in the in the code. So that's written out here if you're unsure about how to follow that really we want to do this for the general case now. So for multi-step look aheads that kind of thing and that's that's algorithm 18 that is and thank you Andrew that is in the um the equations if I go down here uh here we go the actual algorithms are there as well so algorithm 18 that's the final algorithm for for general active inference effectively although it's expressed in terms of exact Asian inference in some some respects. So the final example and then I'll stop talking is the the really general example example 9.8 and that looks like this. We go down here. So we're going to assume now that we're not going to just do single time step look aheads. We're going to do multi- time step look aheads. So this situation here where now we're looking at the policies of length two. So we're starting out sort of further away from our desired goal. We start out in the middle and we have a desire to go down here in six. So this is you know state four. we want to be in state six. There's now 16 policies I have to enumerate. Okay. And I have to sort of score them. Um if we come across to the code, I'll show you really briefly. Probably best to go through in your own time, unfortunately. Um yes, we got our pretty prior policies for that case. So for this case now, for the two-step grid world, there's 16 policies as I as I just showed you. Um we do exactly the same thing we did before really. We need to just say, okay, well, where do I start? What's my horizon? I'm only looking ahead two time steps now. policies of length two. I have a prior belief on policy sorry on on states. Um I need to enumerate all my policies. Okay, I need to score all my policies with EF. Uh and then we can print out what happens here. So there's more there's more policies to enumerate the 16 of them. And in exactly the same fashion, um this is now just the we have the EF itself. Uh it's bit hard to read. Sorry. And then we have the pred the predictive prior over policies that I build up from the EF. So you can see that there's only two actions that have any probability mass and they're shared. So if I show this for each uh policy here, this is my predictive prior over policies basically. So down and left has a probability of 0.5 and left and down has a probability of 0.5 as well. And that corresponds to these two squares here. So only these two squares are the squares that are going to get me to my desired goal. So I want to put all of my probability maps over them if I can. I'm able to do that because observation likelihood mapping that mapping is is um non non-ambiguous here. So that takes us to the end of the examples. There's actually a bit more in the notebook. I got you know I talk about how to get from policy probabilities to action distributions. That's what you need in order to select an action. It's all very well and good to have a plan out into the future, but that has to that uh belief over that has to transition into what action am I going to select right now basically. So there's there's a lot to be looked at there. I do recommend you go through that. Um and then I talk about sort of receding horizon active inference in general. So that's the general the general loop basically that we're we've just looked at. So do go have a look through here. Uh very nice. So if I start out I have this little plot here. Imagine I want to be in seven. I start out in five. You can see that you know we have grid squares states on on the y- axis. We have the time on the x-axis here. I start out somewhere and eventually pretty rapidly I I I uh end up where I would like to be basically. So and there's some optional extensions in here as well about noisy noisy states. So that is the end of the chapter. That is the beginning of expected free energy minimization and action selection thereby. If there are any questions now would be a great time but uh I think we'll have to sort of leave it there and then chapter 10 is going to get into extensions on this. So learning in this framework um and that kind of thing. So very very much along the lines of what we've seen before. So questions questions are there any? Let me have a look. The chat's been pretty active. Nice. So, I'd encourage you to go through and have a look at that notebook. Um, try and follow along if you can. Uh, I think that's the best way to to learn this stuff. We left a lot out, Andrew and I, uh, here. So, there's there's a lot more to say. We're going to be saying more of that, as I said, in chapter 10. So, >> yeah, I think all this is going to repeatedly come up uh, whenever we get to chapter 10 because it's just all of these things are sort of of a piece and uh, Sanjie is still doing his typical giving us things in a very peacemeal fashion. so that we can build up. So I think I think chapters 9 and 10 together is essential will essentially be enough information for people to kind of know how to uh make POMDP models and HMMs um in a pretty holistic manner sort of sort of up to date with where those things are at uh aside from any more clever extensions that some of which will show up in the later chapters. But uh yeah uh looking forward to bringing it all back together whenever we get to chapter 10. >> Nice. Nice. All right. I might end the recording here guys, but uh we'll be we'll be back next week with um the first session for chapter 10 sort of generalizing everything we've looked at. Again, we'll spend three weeks on chapter 10 as well just because it is so long and it's it bears it bears uh study. So yeah. All right. Thank you very much. I shall end the recording. Anyone who wants to stick around uh can. Goodbye.