Fundamentals of Active Inference (Chapter 9, Session 43) September 8, 2026

Great. Hi everyone. Thanks for joining us. This is the active inference institutees textbook reading group. We're still reading fundamentals of active inference by Sanjie Namoshi released earlier this year. Uh we've decided to spend an additional week on chapter 9. Uh and so we're still looking at sort of this first chapter that's explicitly focused on discrete state space models. Um we've already covered things like dynamics and hidden markoff models and are now moving towards uh uh places for action in specifically partially observable Markoff decision making processes. Uh and those kinds of models have been very almost ubiquitously employed in sort of discrete state space uh active inference. And uh prior to me starting to record just now, we were talking a little bit about um the computational psychiatry course being taught at uh in Zurich that's going on right now. Unfortunately, I think ticket sales already ended, but uh I was going to share some nice resources that relate to it, including this paper from 4 years ago, but it's still quite tried and true, I would say, as far as getting started on learning how to employ POMDP models. Um, and and so this was written by, you know, Ryan Smith, uh, as well as Christopher White Carl Fristen. And so Ryan's been giving these tutorials for quite some number of years now. Um and uh so this paper's tailored around the mat lab SPM implementation but uh there's some nice sort of uh these pencil and paper exercises and solutions. Um we're supplied with you know of course per usual many diagrams but includes various examples. Um we get into message passing and mean field approximation. many of these concepts that we've come into contact with from reading Sanjie's textbook. But then we get further breakdowns of all this sort of uh different variables they're being passed things like expected free energy which we'll be getting into a little bit more today. Um and just you know direct examples of how you construct like an a uh likelihood matrix or tensor um state transition tensors and and all the rest. So um I might come back to that a little bit later. And for now um so with chapter 9 uh of course we've switched into this discrete state space um sort of paradigm and I a couple weeks ago started with trying to relate continuous models to discrete models given that we spent the entirety of this reading group and the entirety of the textbook prior to chapter 9 on continuous spaces so that maybe it's not such an abrupt shift to start looking at the discrete statesbased models since they they do of course have um you know very particular aspects that we will not find in continuous models. Um but nonetheless we see various similarities including having likelihood uh these sort of likelihood and observation generating functions uh in each of the kinds of models. So we just need some way of relating observations with hidden states. And then similarly we had these sorts of uh state transitions or state flows uh within our continuous models. Similarly we will have a an a discrete analog of state transition models in our hmms and pomdps. uh essentially the idea of how do states change over time and that gives a sense of temporality in the model as opposed to just staying static and just you know only receiving one observation inferring a presumably static state moving on instead the model has its own sense of how states themselves can change over time. Then uh as we moved further we got into how we can actually supply our agent with actions and so we started looking at how do you um sort of embed that within the models that we're looking at. So that involved um kind of constructing these three-dimensional and really we can move up to four-dimensional and the multi this sort of multiffactor case um you know so Sanjie has been giving us these examples where we just have what we could call one observation modality and one hidden state factor meaning that you know uh one observation modality could be what the agent sees and it could see that it's raining or it could see that it's not raining. Um and then uh a hidden state such as you know is the weather good or is the weather bad but it's just a single factor right um we could have multiple where the agent you know sees um you know with its eyes but then it also feels different things like is you know is the air wet and can feel it through its skin. So now we had suddenly have multiple modalities you know and then we have to kind of dimensionalize that in our structures. So, we're still just focusing on this single modality, single factor case to keep it as simple as possible and no simpler. Um, so moving on, it's a a rather involved slide. Um, because it's chapter 9, it it in many ways really is almost an entire, you know, full recipe for discrete state space models with active inference. Um whereas the previous chapters I mean we chapter two we already jumped to gradient descent and then chapter three we went to learning and chapter four um I [snorts] think is where we got into variational inference chapter five was predictive coding six introduced action more action in seven the whole thing in chapter 8 so it was really spread out whereas chapter 9 really kind of get just gives us a lot at once and that's why we decided to stick with three weeks uh for chapter 9. So, it's worth really spending time uh learning how to construct, you know, your A and your B and your C and your D and your E. Um, whenever you put together these models, and once again, I'll just kind of hearken back to Ryan Smith's tutorial here is just one of various examples uh of how we could learn how to do that. Of course, we should be able to do that from reading chapter 9 in this textbook as well, but it's it's worth getting kind of multiple perspectives on these things. They frequently boil down to doing the same thing. Uh it's just the amount of interconnected dependencies. It's worthwhile getting different researchers and experimenters sort of perspectives on on how they they go about constructing these things and in order to sort of build um a more full intuition over time. So um so where uh we left off in my sessions last week was um we had talked about the A the B and the D. So so our likelihood uh matrix that allows the the agent to to relate. Um and here I'll kind of point at equation 9.49 here. This generative model we had our observation model. we have this nice subscript a to remind us like you could essentially replace this whole term with just a a large capital bolt a. Um that's the the agents a or likelihood model, right? Just the the the probability of each individual observation condition on hidden states. Um and then we had the state transition model which was the what is going to be the next hidden state given the previous hidden state at the previous time step. That's what allows for that kind of um time step oriented temporality in the model. Uh and then we started to um include action which we frequently denote with pi. More specifically, pi represents policies. Um policies are sequences of actions that we play out over time. And so these these models the the POMDP model specifically as opposed to the hidden markoff model where the hidden markoff model is just it can have state transitions but has no capacity for action. Right? So in that case we wouldn't have this this pi being included as an additional conditional variable. um but now we've included it and that's what that that kind of additional um structure to the B is sort of all about where we end up with a distinct transition model dependent on each individual action which is why we're now calling it a a tensor. is sort of a a three-dimensional object containing probability distributions dependent on well what what action did the agent choose? If it if it if it follows up then it believes the state transitions will be different versus if it shows down. And and then we recognize that that supplies the agent with actually a sense that it can control the environment that it has some kind of agency here in how it impacts the environment and how the environmental states change uh using its own actions. So, um, with all that said, uh, the reason why, uh, we have this entire section 9.5 in the rest of the chapter and why it continues so much is that we still haven't talked about, well, how does an agent choose to, uh, you know, act and take a particular action versus another? Why does it choose one policy versus another? And that's what I'm attempting to sort of condense in this this slide here. Um first uh to to hearken back a little bit um example 7.3 uh where we had the continuous uh model there there was this example where you have an agent in in a in a like a one-dimensional environment. So you can imagine an agent who's just sort of standing in place and it can either move up or down. Um it was in this environment where there's this this kind of Gaussian bump as Sanjie calls it. You can think of it as just an agent, you know, standing on the the bank of an ocean sort of on the on a coast and and a wave comes in and the agent doesn't want to be lifted off the ground by the by the wave. And so what does it do? It needs to sort of counteract that wave um by by using action. Assume the agent is somehow pressing its weight down or something along those lines. Um, in order for the agent to have that preference of not being pulled up, we we we we supplied it with this what we call the preference prior where we actually took V, the exogenous force that is the wave itself. Whenever we put that in the model, we we clamped it and we kept it set exactly at zero. Uh all that's essentially saying is by clamping it rather than inferring it as a free variable or a free parameter, the agent, it's as if the agent sort of wants to stay in a a specific space where it wants that exogenous force to sort of be zero. Um and so all that is to say is that by having something fixed in the model, we're effectively biasing this model. And then whenever it it engages in action, it actually ends up choosing an action that counteracts that force. So I won't spend too much time on that, but it's just to make the point that we're clamping some we're keeping something fixed in the model that gives it this sort of persistent bias. Um, and that's what sort of guided action in the continuous case as far as we're looking at these uh generalized filtering models for in the continuous case. But now we're in a discrete world. So how do how do how do we sort of implement that in a model? Um well up to this point we've had priors for hidden states. Our D uh vector it would be a matrix in the case of multiple factors. Um and then of course we've had the A and the B already to kind of explain the different dynamics. But we haven't ever supplied a prior on observations, right? Um and and for good reason. the these models, as we've seen throughout the textbook, observations are usually treated as a known variable, right? We we just assume that the agent directly receives the observation. There's nothing unknown about that. Um it treats it as something that's already sort of determined and not something that needs to be inferred. So why would you need to supply some kind of prior on observations as if we'd have to do some kind of basine inference about the observations? They're they're known. But here um what we can do is that we can supply the agent with a preference on observations that we fix actually and we don't have to fix it but that that's sort of the sub one of the subjects of of chapter 10 that you could learn preferences over time and so the the C could become a parameter that gets learned as well but for now [snorts] um we we'll keep it fixed. Uh and so whenever we do this that that means we're giving the agent another parameter that allows it to have a kind of bias towards particular observations. You'll frequently see in the literature sometimes we see the word observation. Other times we see the word outcome. Uh outcome is usually referring to observations. It's a bit more of a um sort of a contextual uh nuance that uh when we say that the obser the the agent uh sort of prefers certain observations, it's like it it prefers certain outcomes outcomes of of what usually outcomes of its actions and the the process of the action perception loop playing out. Um so they they kind of effectively become sort of the same thing, observations and outcomes. So for those who are familiar with things like reinforcement learning, we know that agents usually have some sense of, you know, reward or otherwise. Um here we're sort of doing something like that where the agent uh sort of wants particular observations as opposed to others. And that's that's what we're doing whenever we supply this vector uh of observations um where where each value sort of denotes a different preference level. Um you'll frequently see and even Ryan Smith and others uh do this frequently people supply these unnormalized values. They're just relative uh logit uh or logit values. Um, the idea is simply to be more intuitive. Like whatever observation this index refers to, it's just a higher number. Some people might use something very extreme like they might say 100 for a particular observation versus a zero or even a negative value for another observation just to really mark the difference. But of course uh we do end up uh normalizing or rather soft maxing uh this vector which does normalize it to a regular probability distribution that sums to one. Um so you know some experimenters like to be more clever when they write this but just be aware that it's still going to get normalized. So, so just as a personal preference, I prefer actually supplying the particular probabilities. Um, at least I know exactly what they are going into the model rather than doing this and then, you know, applying a softmax function and not actually looking at the final result, right? Um, I like knowing what's going on in the model. So, I like applying this in advance and knowing what it's going to be. Um in any case, so we've given the agent now with this C uh vector or matrix um uh these preferences over what observations it it sort of wants. The book starts using this this little brev symbol to refer to the future. um whenever we look at the C and how it's constructed um because the agent is is there's a bit of a conceptual shift versus how I referred to reinforcement learning earlier here remember that agents are um in active inference conceived to be self-evidencing and so it's not simply saying that the agent wants these observations it's saying that the agent uh expects these observations and we're talking about sort of the future Um to put it more simply whenever the agent expects these particular observations in the future uh that means that we can start to employ those when with respect to free energy minimization so that whenever the agent starts to choose policies uh and evaluate policies, it can do so knowing that there are particular observations it it quote unquote prefers or rather expects uh versus observations that it does not uh prefer. or expect and that's sort of what supplies that reward piece and we'll get uh further into that very shortly. So um the way that we will incorporate the seed is that um so Sanjie gives us you know previously he's given us different forms of variational free energy and so there was like a C form and a G form and uh a D form and all the rest. um for now he just gives us one form for what we're calling expected free energy and so expected free energy um some people have referred to it as something like the the free energy of the future. Um there have been different ways of trying to sort of um explain it uh in in in sort of clever idioms just you know as a as a point of reference or provide intuition. But what we're doing here is that the agent is going to evaluate the policies it has available to itself through two different terms which themselves end up being used to uh compute this this overall G or expected free energy value for each policy and more specifically for each policy at each time step because recall a policy is a sequence of time steps uh or a sequence of action s each one of them index to a time step and this is over a policy horizon. So if you have an agent who has a policy horizon of two then that means each of its policies will be looking forward uh two time steps into the future relative to the current moment. So each of its policies will be something like move up and then left and then another policy will be move up and then right and and so on right for for each potential combination of actions. Um or you can limit the number of uh policies that the agent has available if if there's you know it'll depend on what kind of experiment you're running but you can essentially restrict the agent to particular policies. In any case, it's still going to carry out the same kind of computation for each policy it has available for each time step. And then um because we're giving it this this prior preference distribution, the C, it's able to sort of distinguish which observations it expects more versus others. And so that is what allows for computing something like a free energy around it. Because if you recall for variational free energy, which variational free energy is only looking at the past and the present because that's going with known observations, observ observations that have actually already been observed, right? So the agent is able to sort of um sort of minimize the complexity of its model while maximizing the accuracy of its model. Those are two core terms that we see in uh certain ways that we look at variational free energy. um you know it it's it's trying to minimize variational free energy by reducing complexity and increasing accuracy. So with expected free energy we're instead thinking about observations that the agent has not received yet. Instead we're thinking of observations that it would prefer or expect to see always in the future. Um and so the two terms that we get in this RO form of expected free energy as Sanjieve uh calls it instead of complexity and accuracy this form gives us risk and ambiguity. And so um similarly to variational free energy which we we consider to be a minimization problem. We we view it as a minimizing of uncertainty or more specifically variational free energy is a minimizing of a a proxy of uncertainty or surprisal. Uh that being uh variational free energy is a sort of variational uh easier more tractable quantity to compute than direct surprisal. Here we're going to be trying to minimize uh risk and we'll be trying to minimize ambiguity. um risk. You can I I'll try to give a little bit of a an intuition for these things. And again, I left a link to the slides in the in the chat for those who want to look at it because I I tried to be very thorough in these slides so they can kind of act as a standalone thing, especially for those who don't actually uh own the textbook just yet. Um but whenever we try to minimize risk what we're saying is we'll take the the Kobach Lebler divergence which we've seen references to that previously in the continuous case um uh uh this DKL or Kobach lier divergence is a way of measuring a sort of difference between two distributions and so this is saying what is the this is computing what in in risk what is the difference between uh the observations that the agent expects given a policy. So it's sort of like what do I expect to see if I do such and such policy that I'm evaluating uh versus or or how much it diverges from this brev P. So what is that call that's that's exactly our C. So a U Sanjieve just tried to leave out the the C here as it were. you just focused on um trying to make it as clear with the rest of the notation, but I left a reminder in the top right corner. This is just equivalent to our C or preference distribution. So, it's really saying um you know, this is kind of in in a sentence saying will I see what I want to see given this policy, right? Like what do I expect to see if I do this policy versus what do I actually want to see? So you can imagine a policy that the agent has sort of inferred like oh this policy will lead to things I want to see. Uh the divergence of that with what the agent wants to see will be very little potentially zero. Right? So the agent will end up going with probably policies that lead it towards observations it wants to see that this is exactly where our C parameter sort of gets embedded into the model. It's not in variational free energy because again that's referring to the past and the present. That's only observations that are known and have actually been observed or are currently being observed. Whereas here we're looking at future observations in the f if I do this policy if I carry out this policy will I end up seeing observations that I want to see. Um and I'll give a little bit more intuition on on that shortly. We'll move on to the next slide which I uh hope will be less um just sort of overwhelming. Uh and then here ambiguity this is this is different. We don't see the C being employed but we do see um our uh the equivalent of our A matrix right that's observations cont conditioned on in states you know it's equivalent to our observation model up here. And so what this is doing is um especially given that we're taking the expectation where we're actually weighing things out based on our model and our variational uh distribution here condition on policies and then we're taking the entropy. All this is saying is, you know, if I have this a matrix that helps me perceive the world where it gives me a sense of, oh, if I see this observation, it let leads me to infer that probably this is the current hidden state of the environment as opposed to a different one. Um, if we take the entropy of that, um, what that's doing is kind of measuring how precise your your likelihood matrix is. So um this gets into the notion of many to one mappings, many to many mappings, one to many mappings, one to one mappings, these different ways um of of looking at likelihoods uh that that that relates to relatively standard machine learning and and reinforcement learning and the rest. It's basically how informative is my my A matrix. uh right like if if my A matrix says well this observation could be could be related to this hidden state or another hidden state each of them get a probability of 0.5 that means that this observation is maximally uninformative of the environment right like if I if I hear a particular sound and I have no idea what it means right it's it's a it's a very uninformative observation there would be very high entropy very high uncertainty around that that mapping How does that one observation map to any of the hidden states? Right? As opposed to a different kind of mapping where you might say, oh, this the the agent through its a matrix is very certain that this one observation with a full probability of one matches another a particular hidden state out of the many hidden states of the environment. In that case, a probability of one, it would be very low entropy. It would be full certainty that this observation is very informative. it tells me exactly what the hidden state of the environment is. Right? So, it's kind of taking the agent's ability to distinguish observations with respect to hidden states or observations conditioned on hidden states um into account and then using that entropy as a measure. And because we're trying to minimize that means what we're trying to do is we're trying to minimize um the the ambiguity around the observations that the the agent is seeking. So the this latter term essentially has has nothing to do with the C matrix. It's just this risk uh term that has to do with the seat. Ambiguity is really just referring to the agents sort of how um let's say how uh uh you know how how much fitness sort of does the the A matrix have? I don't want to misuse that word, but essentially how how informative are observations u based on the agents beliefs about how they relate to hidden states. Um we'll we'll turn back to that again, but it gets related to to this notion of um you know if the agent is actually able to sort of figure out well how uncertain am I about how the observations I receive relate to the hidden states of the environment and then include that in its capacity for action. It takes a direct account of uncertainty with respect to its actual model. like there's there's there's nothing random here um that's going on. It's it's actually considering the uncertainty within its model and then from there using that as one of the the major factors that it incorporates into how it chooses what to do. Um so that's what this expected free energy is sort of all about has to do with what kind of observation should I be seeking? What observations do I know I want to find? Um and so that your ambiguity and your risk in a certain way um while acknowledging what you do and don't know. Um mathematically we we essentially do that. There's a different variation that that follows more proper matrix notation, but I'll just say that this equation 9.53A um is essentially the same as the RO form of expected free energy. It's just kind of using the notation that we've seen elsewhere in this chapter to remind us that we're really just working with matrix operations with dotproducts between matrices and vectors and doing log transforms um at particular times and and all the rest. Um and then there's there's this other sort of variation uh but it's essentially the same equation again which is outcome prediction error. I won't spend too much time on that but it's a sort of an interesting notion. In any case, what we do is that we we apply this function uh to compute the expected free energy for each policy for each time step of the policy. Right? So this is a rather large space of computation that we're doing. Uh we will end up with uh potentially a very large matrix of uh expected free energy values whenever we score this for each time step of each policy. Um but then um does that tell us what to do next in any sort of coherent way? Like why use policies as opposed to just considering single timestep actions? And so what we can actually do to allow for this capacity for planning where the agent maybe starts to uh commit to a particular sequence of actions commit to a particular policy is that we would then sum those expected free energy values for overall time steps per policy. And so now we simply end up with the negative uh we simply end up with the expected free energy uh for each policy. And then that way the agent kind of has a sense of which policy to follow. You know, maybe there's maybe there's high expected free energy for policy number one in the next time step, but then much lower at the following time step after that, right? In the sense of allowing an agent to plan so that you know perhaps it it sort of has to bear with the cost of the next time step but in order to achieve something better in the time step after that. Right? So we can distinguish those sorts of things. Um and then this is one form of policy inference because now now that we've we've computed all this expected free energy and the like you know uh we still end up with a final posterior distribution over policy. So now the agent actually okay great I've computed all these things regarding expected free energy. I've I've tried to minimize risk I've tried to minimize ambiguity which one do I go with? Well, we're just going to apply this soft max to those expected free energy values per policy. Um, and then that gives us our posterior. Um, I will give a quick note that this is not the only way to compute your policy posterior. Um, that's not to say that they're dramatically different ways. You'd still do exactly this. you'd still be using expected free energy, but there are other variations that would take negative expected free energy and negative variational free energy um per policy and uh uh a log E, which we haven't gotten into E yet. we will in chapter 10, but that's just giving a prior on um uh a uh they're called habits, but sort of a a prior on policies that itself can be learned over time and built up such that an agent suddenly has something like habits. That comes up more in the psychiatry literature and cognitive science literature but less with respect to sort of the robotics uh in in you know trying to de develop a more oriented model that will be applied to you know some some sort of performative high performance focused use case but um yeah so I just to point that out um there are different ways of computing a policy posterior and then you know variational policy poster. This is just saying we're we're still working with the categorical distributions that we have been with these models. So just as our A is a categorical distribution, our D or B all the rest so is so is the the final posterior uh over policies. So we'll this is another take on all of the earlier information. Um but figure 9.20 20. We'll we'll just focus on that first because it looks very similar to all of these graphical models that we've seen up to this point. We have the D and the A and the B, but now we've simply added policies and then said, well, so long as we commit to this um this sorry, this previous equation, you know, we're just using G to compute policies. Then that means we're able to draw these direct lines from G to policies. And then something else that we have to incorporate into G is the C whenever we're computing uh risk down here. And uh quick apologies. I normally have a second monitor here. So I'm kind of doing a lot of clicking back and forth. But so so all this is to say we're able to now include into our discrete state space model this Marovian model um policies being informed by expected free energy that we minimize and then part of how we do that is incorporating these preferences. Um there are many sort of um you know the the nomenclature from here gets very expansive whenever we talk about expected free energy in part because we're specifically talking about an agent's behavior. We're talking about how it makes decisions and and what kinds of behaviors ensue from those decisions. And so we'll find you know purely theoretical published papers uh you know by by those who work in in psychology or or economics or otherwise that start to relate all these sorts of concepts with pre-existing concepts from from other sort of um um you know research domains. And so we'll see people talking about risk in relation to economic risk and we'll see you know ambiguity being related to things like um cognitive dissonance and psychology and the like. So it can become very very broad. Um I've tried to only include um some kind of um some kind of core terminology that active inference refers to frequently when talking about these things. Um and and it sort of boils down to well if we look at this this RO form of expected oops sorry uh of expected free energy then we're just looking at these two terms that we already talked about risk and ambiguity. I just tried to make it clear this is your C term being injected sort of into the risk. Um this relates to uh in reinforcement learning this sort of classic tradeoff known as the the the exploration exploitation tradeoff. Um this is this has been sort of a trick for a very long time. I mean this is still an ongoing conversation in uh in various fields right which is um for those who are kind of speaking more in RL or ML terms um if what it if you have an agent who is only seeking reward um then how do you make an agent who seeks reward and doesn't um you know have a high chance of getting stuck in a in a local uh minima or maxima so to speak that is for an agent who only seeks reward. Um, where is any kind of drive to learn about the world or to to learn potentially alternative more optimal strategies? You know, so if you if you have an agent who's already sort of convinced, so to speak, of the right strategy, it's convinced that that it will lead to reward, it only has a drive for reward, then it sort of, you know, it allows for a lot of nicer properties in a way. If you're thinking about performance where it kind of just skips a lot of things and goes straight for the reward, no other concerns necessary. Um that might be great depending on your use case, but more often than not uh especially for for unknown circumstances and especially whenever we do things like trying to do like actual um you know with relation to to cognition and psychiatry and the rest trying to study behavior empirically, it's it's repeatedly seen that that you know people even mice are not purely reward driven, right? They they they they show varieties of behaviors. um if if all of us sort of had the same computational phenotype and were all reward driven, then we'd presumably all act uh in exactly the same way, right? There there would be no further exploration, no differentiation, no diversity. So here um it's uh we're we're able to look at these two terms uh sort of as exploitation with relation to risk with relation to you know how do I get what I want right like will this policy lead to observations that I want if it does then you know you know bonus points to that policy I should probably choose that policy we could just stop there right we don't necessarily have to have this ambiguity term at all for scoring ing our policies for for distinguishing and choosing which policy to go with. It's whenever we include this ambiguity term that that our paradigm changes a bit because we're giving the agent an additional sort of drive that's being guided by um sort of will these observations that I'm seeking like lead to a lot of ambiguity? Will they actually be informative? Will I gain any kind of information from choosing this particular policy? uh or or will things remain ambiguous? So, so the idea here is that by minimizing ambiguity, we're referring back to our model and saying like, oh no, we should also incorporate the fact that not all policies I choose are going to lead to predictable outcomes, let alone pol outcomes that I want, but it might not be very, you know, the the the agent might not be very secure in its ability to make these kinds of decisions in the first place, right? So, by incorporating ambiguity, we're kind of limiting the agent's quick drive to seek reward, including taking shortcuts and all the like. This is kind of saying, well, if you know, the agent needs to seek more information. And so, it needs to to to use the information it has and also acknowledges the the information it doesn't have and then incorporates that into uh policy selection and inference. So um whenever we we sort of do that that's that's quite distinct from how a lot of other fields have viewed exploration. Um I I frequently do this I will pick on a particular approach. I I kind of put it in caricature just a little bit just to really make a point but it really does work this way. But I'm I'm just I'm kind of picking an easy example for the sake of illustration of of what I mean by this whole active inference kind of sort of naturally balancing uh reward seeking and information seeking uh drives and behavior um which is uh reinforcement learning. One one method that has been done or or employed in in RL agents to prevent them from being purely reward seeking um you know essentially to try and prevent them from getting stuck at a local minima is um you you can have them still be rewarding but you simply supply um it the the the technique is called epsilon greedy. So the idea is that you supply this little value uh denoted by epsilon. You know it could be a 0.1 a 05. Um but it's kind of like saying okay be reward. You're telling the agent be reward seeking but with a probability of epsilon a very tiny probability just do something random. Don't do what would be reward seeking. And uh that is very easy to pick on if I just say it that way because clearly it's like well that's not really exploration right that's just you know sometimes don't do what you think is best is what you're telling the agent to do but it's a genuine techniques that that's been used because it's not clear how to actually um you know how do you give uh a model an agent uh any kind of principled motivation to seek information especially in a way that doesn't uh you know completely completely deter it from reward seeking, right? Um, if you have an agent just simply do something random, well, I you know, not to uh be dark sort of, but I mean, you could have an agent who just suddenly decides to to die, right? You you could have it just suddenly decide to do something so random that that that that it kind of defeats the purpose even of um you know, this issue of getting stuck in a local minima. Is it worth exploring if you know that it's going to pull you very far away from any kind of minima uh and make things worse? Right? So, so here we're not only giving like a sense of exploration um you know that's going to be taken into account in the model. I mean it's it's this sort of double bit double benefit of first the agent is using both risk and ambiguity, right? So it it won't necessar even if it minimizes ambiguity for one policy it doesn't mean it's going to ignore reward seeking that it's especially important if we're trying to sort of model empirical phenomena I mean it's great if we you know see a child at play who's exploring but they still need to breathe right they still need to breathe they still need to eat food they still need to do other things that lead to preferred outcomes such as staying alive having healthy caloric intake having enough blood oxygenation in their body by breathing. Right? So, it's not to say that just because you minimize ambiguity, it means we're going to lose anything ever related to reward where reward can be something essential. It can it doesn't just have to be a monetary or quantitative uh gain. It can also just be how does a being persist through maintaining its homeostasis, right? That that's also related to reward seeking. Um so with ambiguity though um we we're we're combining that with risk right so that we we don't lose track of both of these important things and then uh and then sort of secondly ambiguity is there specifically as it is not to say do something random in fact it's almost the opposite of doing something random this is saying given your current model and the degree of ent entropy in the likelihood mappings that is the kind of ambiguity in your model and your ability ability to perceive the world. Um, it gives you more explicitly a way of of kind of determining uh, you know, where is where is more certainty to be found. Where is my model currently lacking? If I account for that, that should be part of what helps me to account for how I could learn more going forward, right? What do I not know? What needs to be more clearly uh, disambiguated in the mappings and my beliefs between observations and states? So that's sort of this this sort of ambiguity term is is all about. Um so we're sort of just combining those two terms. It's that that's sort of the distinctness of active inference. It's part of what drew me to it as a as a researcher that I found very interesting. And then you know with uh that in tandem with different kinds of um empirical work that's been carried out. It's like how do we figure out not just you know how we perceive the world but how do we act within it. It's like, well, we we do learn and we do explore and we do those things, but exploration doesn't usually involve something that's purely random, right? Like someone wants to see something that they haven't seen before, but it doesn't mean that they're necessarily going to go to the bottom of the ocean without any oxygen, right? That that, you know, the the exploration that happens usually is related to um you know, the rest of our understanding of the world. Uh and so so here exploration is still in direct reference to one's own model and still accounts for important things regarding reward seeking whether they be trying to seek greater monetary gains or if they have to do with taking care of our ourselves and our our families and friends and society and all the rest. So so um yeah that's essentially kind of what we're looking at here with the end of chapter 9 is just how does action play out? I mean we we saw in chapter uh 7 that whenever we incorporated action there was sort of this this extra thing going on uh right that that that action actually uh used sensory observations and this like action model to um you know it makes reference to the original model but it still has its own sort of operations occurring. This is sort of the discrete analoges that just as we had variational free energy for perception with known observations that is we're able to compute uncertainty about the past where we have known observations because we already saw them and the present because we're currently seeing those observations here. Um we we we now also have expected free energy as a way of trying to figure out what to do in the future where those observations have not yet been observed. And uh this is a a small reference, quite literally small. I didn't have enough room, so I kind of just squeezed it here. But it's just worth bearing in mind the original 2022 active inference textbook, which is indeed freely available. So I would definitely encourage anyone to just download it uh directly. But there are different ways of sort of formalizing expected free energy. And I like it in that um these different variations on it. They're still mathematically equivalent, but they're just different ways of interpreting um you know sort of these respective terms. So just as we can have risk uh and and ambiguity, we can also view this as sort of information gain and pragmatic value uh or or expected energy and entropy. So you know there's different flavors so to speak. there's some that that will make more intuitive sense for those who are doing something like um you know behavioral science or economics versus those who are more interested in strictly um sort of um uh physics uh terms and looking at these as uh as as dynamic systems. So um I I would strongly recommend uh having a look at that. And then the upper right, this was just to sort of illustrate in the A matrix and likelihood mappings. You know, this is an example of a high entropy versus a low entropy um sort of column in an A matrix, right? Like this this first um column here is simply saying that you know the the the probability of the first observation under this hidden state is a 6. Probability of the second one is a 04. It's like, okay, great. Well, your your expectation of of the first observation under this hidden state is a little bit higher, but they're close to being equivalent, right? Then that would mean that well, if I know the hidden state, it's slightly more likely that I get the first observation that I'll see this first observation. Um, but it's almost equivalent uh to seeing the second observation instead. So, there's ambiguity there, right? That's that notion of ambiguity here. Um whereas the second column you know for the second hidden state indexed one with zero indexing this is saying I'm I'm much more certain that I'll see this second observation uh index one rather than the first. Right? So so this hidden state would be very informative as to what observations I'll come into contact with. That's what we mean when we say has less entropy and it's less ambiguous. Right? And so you can use that kind of information from your model to help you determine what kind of policies to pursue and what kind of um observations you can expect based on what policies you pursue. Now um I'm going to wrap up here because of course we usual I'm a little verbose. Um but uh but so a couple things that one uh we will move to chapter 10 next week. Um I think we will spend three weeks on chapter 10 as well. Uh that's in part because like I said, chapter 9 is is is incredibly plentiful. Uh and we're getting everything on discrete state space models all at one time. Uh and so chapter 10 will continue from chapter 9. We'll get into um sort of uh some things I mentioned earlier like preferences learning and habits learning and just the general way like how do we learn our parameters? Um just as in chapter 3 for continuous spaces. um we learned how to update not just our our beliefs about hidden states but also our parameters themselves. So we'll do that for the discrete statesbased models uh starting next week. But then um uh probably for my session uh I will probably blend a little bit of chapter 9 with chapter 10 just because these things are sort of all of a piece and because there's so much content in chapter 9. I mean it's it's literally 50% longer than most other chapters in the in the book. So I see that some you've really tried to give us everything at once. Um and then here these are just two brief resources. I'll probably in future I'll make sort of an appendix in the slides that that has uh ideally many more resources than this. Um I shared some other ones last week for anyone who wants to view the um recording. Just a reminder that you can watch literally any of our recorded sessions by looking at the the KOD page for the fundamentals active inference and all the the links are there so you can watch any previous session. They're all freely available on YouTube to anyone. No need to sign up for anything. And um so so yeah, I may um uh I will most certainly add more material uh going forward. But one one piece was the Ryan Smith tutorial that I already touched on. Um and then uh this was a nice um I just received the email about this and saw that it would be um viewable by the public but uh this was through the theoretical neurobiology group um who are uh you know [snorts] associated with uh the institute but it's largely uh spearheaded by um Carl Fristen himself. And so, um, they're going to give a tutorial on using PIMDP. Uh, that will happen in just a couple of weeks here on the 21st, uh, 1400 in UTC. And, uh, it'll just be at this YouTube link. So, I will, um, just quickly copy and paste that into the chat. Is is that the the um will that be like the TNG's normal time? >> Um >> um yeah, I I I see what you mean. I I'm not I'm not certain to be honest. Um yeah, I'm not certain. I mean, I know that their meetings kind of uh the meeting times kind of fluctuate depending on who's presenting, but >> cool. um is um yeah I mean that that does sound cool or like like that should be distributed. [laughter] >> Yeah. Yeah. 100%. There need to be more tutorials uh on on PMDP. So and I I've met uh you know uh various of those folks. Arun uh has been picking up sort of taking the torch on the more recent uh developments and been you know sharing them with the public and the rest. So it should be should be a pretty good uh pretty good one. So hope everyone checks that out and uh yeah I will oh I already shared that. So yeah uh I will be sure to share more resources going forward. I'll probably reshare this link again next week as it gets closer to the date because I'm myself am very curious. Um but yeah, we will we will stop the recording there and given that we're kind of a little over on time. my apologies but um we will not have much time for discussion. I think there will be a lot more time uh to to have more back and forth for chapter 10 because I think by then we'll have actually been able to cover all of these things regarding POMDP models and um you know I've been avoiding too much discussion now because it's like oh no we'll get to that next week we're not done with the chapter yet you know so um in any case all right so thank you so much everyone um take care see you at the next one Thank you.