Fundamentals of Active Inference (Chapter 9, Session 42) September 4, 2026

All righty. Hello everyone. So it's uh September 4th, 2026. We're here uh for the second session of chapter 9, fundamentals of active inference. So um little bit of a change of pace. We're actually going to be doing three sessions total on chapter 9. So we've already had one. This is the second of my sessions. Uh but we're also going to have a third one as well. Uh chapter 9 is quite long uh compared to the other chapters. And it also marks as we as we have noted in our respective sessions between myself and Andrew, it marks a transition in the book or in in the theory of active inference itself uh between the continuous state space formulas and everything we've looked at so far and now discrete state space stuff. So that can be a bit of a jump uh for sure and indeed we're going to see maybe at the very end of this session but definitely next session um where we bring in this notion of planning and sort of selecting actions by means of considering alternative futures. That's a very big change uh to what we've done so far basically and you sort of need the discrete state space formalism to talk about that. So it it does bear, you know, really meditating upon all of the sort of mathematical machinery that we've um we've now introduced. So we sort of did that last session. My my last session was a little bit dry in that respect. We were kind of stepping through, you know, what is a categorical distribution? How do we parameterize the things that we know and love so far? Things like the observation, uh likelihood, state transition, that sort of thing. We saw that we could uh reparameterize these things in terms of various matrices. Okay, so that's kind of where we we ended up. We did have a look there were there were one or two examples using this formalism to do prediction and post-diction in in a few simple models. Um that is to say hidden markov models. So and we're going to continue doing that. But what what we saw in sort of 9.1 and 9.2 were the case of exact Beijian inference. So you know that's always the gold standard. That's always the the north star. But of course in the general case we can't do this and the whole point about active inference is well how do we do things close enough to be basian inference exact basian inference we min we minimize variational free energy. So that's the story of 9.3 um and also a little bit into 9.4 and that's what hopefully what we'll get through today uh those two sections and they kind of they really do well to my mind at least they sort of tie things together in terms of what we've introduced and where we're going. So, let me share my screen here. I'll just give a few updates on the actual page itself. Uh, here. Oh, no. It appears that they've changed the interface. How terrible for me. Uh, I don't know if I can share my screen anymore. That's that's unbelievably terrible. Host tools. Oh, wait. No, there it is. I'm just uh challenged. Okay, let's do this. So everyone should see the coded page just now. So here we go. Chapter nine. Uh we've got the outline, everything there from before. Um we've got the markdown version and as I say the PDF version. So this is the overall notes for the entirety of chapter 9. Very kind of high level. I've added in if we go way down to the bottom here. Uh so we've got the sort of chapter map. So section by section. Of course we got 9.1 and 9.2. That's what we did last time. I've added in 9.3 and 9.4. Now I do have the rest of the chapter worked out. I just want to make sure those notes are are as good as they can be. So they're not 9.5 and onwards are not there but 9.3 and four are there now. So this is what we'll be doing hopefully 9.3 9.4. We'll have a look at that going forward. Not all of the figures are here now. Uh this however everything for 9.3 and 9.4 is here. So these are all uh relevant figures to what we're going to be looking at. I'll put the rest of the figures there as well I think. and all the equations should be um should be in as well. Thank you very much, Andrew, for that. Righty. Well, uh I'm just going to get stuck into it with 9.3. If there are uh questions, uh I shall intermittently come across and see if anyone's got their hand raised. I can't actually see that now. Um maybe I can figure out a better way to do that. So, let's let's just really quickly sort of outline outlay outline the story what we're doing. So prior to you know we we basically saw last time that we've introduced a couple of things. We've introduced matrices for for doing for parameterizing our model. So the only real difference is that we're now doing things in terms of matrices. Um and we have a pri just like before we got a prior on hidden states. That's our D uh vector. We have an observation likelihood. That's our A matrix. And we have a transition uh uh function or transition matrix B basically. And our generative model is now made up of these matrices and we multiply things do matrix multiplication and slicing to do all the operations that we need to do. So that's the whole story from last time. But now we're going to see look ideally yes we want to use basis rule. This is the thing we always like to use um for the hidden marov model. Okay. really zooming way out. Chapter 9 began by talking about mo motivating the idea that we want to be start solving these things called markoff decision processes. So maybe that's a better place to to kind of initially start from. So here's here's a sort of markoff decision process up here. I there's more to this figure in in a while, but we just focus on the sort of state transition the the Beijian network. We've got states transitioning across time and they give rise to observations basically. Um this is and this is happening in discrete. So this is happening at you know time point one, time point 2, time point 3. Uh and we've got discrete state spaces. This is what's known as a a hidden markov model in the sense that there are observations. You can't directly observe the hidden state. You can only observe the observations the sometimes those are called readouts from the state. But if if you see here, well, this is a bad example because this is controlled. Um, if you ignored the pi for a second and just and there were only S's transitioning and there were only O's being generated from readouts generated from those S's, the what you would have as a hidden markoff model. You would have a partially observable markoff decision process where there were no decisions to be made. The states were just sort of autonomously transitioning across time and each state was giving rise to an observation. That is called a hidden markoff model. If you can control the way that the states are transitioning, uh that's that's now a a more interesting scenario. That's a POM DP, a partially observable markoff decision process. So you can actually intervene on the state transitions and get states to transition to states that you would like them to transition to. It's actually what we're looking at here, but we're going to start out just with the the hidden markoff model yet again because that's where things uh kind of need to start. So we start out in here we have our you know so we had a hidden markup model and we have we want to do beanian inference the initial again just to reiterate the initial belief on hidden states is is now this d vector we can multiply matrix multiply the belief at time t or t minus one depending on where you're starting things so we start if I have a belief in my prior time step then the belief about hidden states now is given by B times that prior belief and that will give me the belief about hidden states. Now, so what all that we're going to change from 9.2 to 9.3 is the method of inference. So we're basically going to be doing chapter 4 redux. We're going to do variational inference now with the discrete um state space model and starting out with the hidden markup model. And so we kind of know that story. We we've been through chapter four. Uh we had a look at the machinery of variational free energy. We saw there were various decompositions of variational free for energy. We're not going to get too caught up in that just now. We're going to just have one canonical decomposition, the G form, and we're just going to roll with that. That's sufficient for our purposes. So the what, you know, what do we do with the variational minim minimization? We know that we want a true posterior over hidden states given observations. That's the that's the north star. That's the thing we want. But we can't really have it. So we're going to introduce a new distribution over hidden states which we're going to optimize in some way to be as close as possible to the variation posterior. And keep in mind now these distributions that we're dealing with these beliefs these are categorical distributions. So they look like uh if I go back to well they look like things like this. So you know I have a these are discrete categories and there's a probability for each category. That's what they look like now. as opposed to the nice smooth Gaussian bell curvy things that we had before. Uh back back I'm not going to step you know through all through my notes one to one but um because I have some code I want to show in terms of some examples but it is nice to start out here. So we have an approximate posterior categorical belief of hidden states now and now we want to minimize the variational free energy. Um, and again, that's really very we're basically going to do exactly the same thing we did in chapter four, except now we have different kinds of beliefs. So, we want to get this to be as close as possible to the true variational posterior. Uh, and there's there's different ways of well, and the way that we do that is by minimizing variational free energy. We saw arguments in chapter 4 as to why it is that minimizing variational free energy makes the approximate posterior close to the true posterior. I don't necessarily want to have rehash all of that although we can if people need to. So if we take that for granted, if we if we understand that minimizing VF gets the var the approximate posteriority to be as close as possible to the true posterior, the question becomes how do I minimize VF? Basically that's that's uh and there there's actually many ways of doing this. A very simple way that the initial way that uh Sanjie does it here in the chapter is to start off doing something called coordinate descent or coordinate search. Um you can also do sort of any any gradient based method you would like. So there is an example uh which I would like to show um which does this just a little bit more before we jump into that as to why motivating again why VF is necessary why we can't do bases rule this is bases rule here so we got uh the tripisteria is equal to the joint probability of observations and hidden states divided by the evidence of that particular observation so this is just our generative model divided by the model evidence that that is bases rule exactly and we know that we can factoriize the general of the model in terms of an observation likelihood and a state prior. Okay, so this for us now in the categorical land this is a this is the a matrix this is the d matrix or the transition matrix from the previous state um and and we have this uh observation likelihood. Now as as with the continuous case the trouble with using this is the observation likelihood computing this is usually just completely impossible or intractable. So the way that you could what what what do we have available to us? We have available our generative model. So we we have specified some kind of generative model. That's what we have and we have a sort of well that's that's all we have really. Um how do how could we get the observation likelihood from this generative model? we have to marginalize the generative model which just consists in summing up the generative model over all conceivable hidden states that there could possibly be and when you do that that's a process of marginalization you would get the uh model evidence. So that's usually just completely impossible to do because usually the the hidden state space is very very large. It's combinatorily explosive um and it's just not possible to do for really simple problems you can do it but then why do this at all if you can if you can exactly marginalize. So the problem is that we just can't do this in in the real world for most interesting problems. We can't marginalize the general model. That sort of motivates why we need to start thinking about minimizing BF. So I thought it was just worthwhile to revise that a little bit as well. You know, given that we have categorical beliefs now and and not uh continuous Gaussian distributions. We saw that the form of the VF last time. So by the way we're just going to look at the G form of the VF in this whole chapter I think or at least in this whole section where the G form is given by the approximate posterior times the log of the well fraction of the approximate posterior over the generative model. In the continuous case this was an integral. Okay. So this was a a nice continuous thing for us. Now it's a little bit simpler in some sense. We're just going to replace the integral with a sum. So we're going to sum over hidden states. I've got hidden state one hidden state two hidden state three. however many I have. Um, and so the the G form for us now in the categorical land is is this here. It's a sum instead of an integral. So nothing's really changed. It's just that the the underlying technically the underlying measure of what we're talking about has changed, but that's we don't need to get into that. All right. So that sets us up. But the so that's we're going to be doing VF minimization. We're going to be doing it with our categorical models. We need an approximate posterior Um so and hidden states are going to be transitioning across time typically you know that's that's sort of the setup we're most interested in dynamical uh where environment is changing. So you know the exact posterior for some uh you know let's imagine we've got some sequence of hidden states and that's giving rise to a sequence of observations. Your posterior your exact posterior would be this. I'm notating the sequence from you know zero to t. Um so the approximate posterior would just be you know the categorical distribution over that sequence. Now what's now we we have made the mark of assumption about how hidden states are going to be transitioning in the model in the process. Okay. So uh that's consistent saying that we believe that hidden states only m the the hidden state now right only depends upon the hidden state immediately previously. It doesn't depend on the whole history of hidden states. Okay, that is our modeling assumption about how the generative process works. In addition to that assumption, we're going to make uh something called a mean field assumption. We've already sort of seen this in chapter 8. Uh if I go down here, so we you know we we're making the Markov assumption about how how hidden states transition in the model. We're also going to make an assumption about our hidden state belief. Namely, that the belief over the whole trajectory that we're considering factorizes as the uh individual beliefs throughout that entire trajectory. So, you know, my belief at time t= 1 over hidden states times the belief over t=2 hidden state and so on. This is a this is called a mean field factorization. So, that's a big that's a fairly common assumption that you make for a lot of simple models. it makes the whole process of uh well it makes everything a lot easier although there's questions about how justifiable that is in certain cases we don't have to worry about that just now but that is the assumption we'll make um so I talk about how mean field independence is not the same as generative independence so in the in the model itself hidden states are coupled so the hidden state at this time depends upon the hidden state at the previous time um and that's an assumption I'm making in my model so you know neighboring hidden states are not independent in the actual generative model. But the factorization of our approximate posterior is an assumption about the belief over the hidden state. And it's it's totally they're two different things basically. So you know we should distinguish temporally coupled variables in the generative model from factorization assumptions that we make about our belief about those things. So all right uh very very good the now what we need so what he does on page 232 is he just kind of states uh equation 9 2.9 uh well let's let's go back from there for a second equation 2.8 or 2.28 28 we we start out with the G form of the VF. We can equivalently write that as an expectation. Okay, so we have our generative model. We have some sort of approximate posterior. We can work out the VF. Um it turns out I sort of go through this hopefully in a little bit something approximating a derivation to to show why the VF is the the the an upper bound on the model evidence. Okay, that's what the VF is. It's a it's an approximation to the model evidence and namely it's an upper bound. So if we make that small we're going to be getting close to the model evidence and that's the thing that we want to compute. The thing that is maybe a little bit of a jump is we can just fully express the VFA in matrix notation. So our generative model is made of A matrices and B matrices and D matrices now um and maybe even one or two more later we'll see. So the VF can be expressed in terms of these things. So 9.29 29. This is the matrix form or expression for the VF or the G form of the VF. So this this kind of expression is the sort of thing we're going to have to kind of get used to basically. And I I sort of try and refresh everyone's memory about where like why this might where this sort of comes about given the kinds of hidden state beliefs that we have and that sort of thing and some some notation around indexing. It's it's a bit boring all of that bookkeeping but it is necessary to understand what's going on. Uh yes, very good. Very good. Now we actually get to the first example. So that's example 9.4. So what I've done is I've created a notebook which um implements example 9.4 and 9.5 uh and also the dynamical case. So I think if we step through that that'll be more interesting than just looking at um equations. But basically the idea is let's say that we have we've got a two-dimensional hidden state right so there's there's two hidden states and there's some observation likelihood it's this matrix here and we have some prior over hidden states in this case it's a uniform prior so we don't think that hidden state one is any more likely than hidden state two and vice versa and we get a particular observation in and in this case observations are one hot encoded vectors so they're a vector with either ones or zeros well sorry exactly one one and everything else is zero which means that I observe whatever thing was indexed at this location. Okay, that's the actual uh setup. So I I want to I want to go across to my my example but just so that we can kind of see what's going on a little bit better. The idea and we're going to do we're going to do precisely um coordinate search to minimize VF. Uh maybe actually that would be nice. Just one one last thing. So the idea with coordinate search is that we're going to start out I I write a little thing in here down here. Down here uh it's better shown with a visualiz. Here we go. Yeah. So you know the general idea is that you'll you'll start out somewhere in the space of hidden state beliefs. There's it's two dimensional. Okay. We got a two dimensional thing. And we're just going to evaluate what our the free energy would be of a belief where in each of the cardinal directions. So norththeast, southwest, but also northeast, southeast, and and so on. So there's going to be eight directions. We're just going to take a step in the sort of manifold of where we are, evaluate the VF of those eight things, and then we're just going to step in the direction of minimum VFA, and we're going to keep doing that over time. So I I do I talk about that here of what that's doing. Have a look if you want. So what we have to do is we have to choose a step size delta and then we just kind of step iteratively. So what that actually looks like is figure 9.13. Unfortunately figure 9.13 is kind of boring because it's a fairly small example. It's only a two dimensional example. Um so what we're looking at is the belief over the the first uh sort of hidden state factor and the belief over the the well sorry first and second hidden state factor. Uh now given that beliefs are probability distributions there if we sum across the entire probability distribution. So sum across in say factor one and say factor two that have has to give us one because the sum over probability mass function the categorical belief has to be one. Basically what that means is that our our our actual sort of beliefs have to lie on this line here, this diagonal line. Uh so it's a fairly simple that's that's called the probability simplex. It's like a shape that you're allowed to sit on as a as a belief. Uh so it's a very simple example for just a two dimensional hidden state. So we start out here in green. That's where the current belief is. But if we can sort of nudge by taking a step up here to the left given that the observation that we got was 01 uh that's going to take us in the direction of decreasing VF going to the right is going to uh increase our VF. So what we want to do is we want to take steps iteratively this way. Now this is such a low dimensional example that of course it it appears obvious to us that we should be taking steps you know to the top left. That seems almost silly that this is what we're doing. But this is mainly to show give sort of intuition about what's happening. Obviously the state spaces we're going to be concerned with usually are going to be very very large and not even possible to visualize in any sense. So um don't take the simple nature of this as as any kind of problem on behalf of the method. So the I'll show you my code as well. But the what what what what happens if we do this is is figure 1 9.14 this here. So we start out you know somewhere well where do we start out you know first of all we start out somewhere on the space of beliefs. Let's just start at our prior belief basically. So we're going to start out whatever with with our dmatrix. What was our dmatrix? It was precisely the uh uniform distribution 0.5. So if I go here uh I mean yeah hopefully you guys we I mean we saw that it was it was encoded here. I don't know where it is exactly. So we start out with a belief over hidden states that they are all equally likely. There we go 5.5. So that's where we actually start out on this this simplex. So here we go five.5. That's where we begin. The true posterior is this this red dot over here. Now what we're going to do is across time we're going to minimize our BF by taking steps in the coordinate descent um formulism in the direction of minimum VF. So each one of these little uh green triangles is a step. There's I don't know maybe there's about 50 of them. Who knows something like that. And then on the right here so that's that's our this is our trajectory through the space of beliefs. Okay. It's a very kind of boring space. It's a line. Um but you can see over time as we're minim minimizing VF that the approximate posterior Q of S is getting similar more and more similar to the true posterior over right on the right we have just the VF itself plotted across time and we have the negative log evidence itself plotted and this is so simple that we can just directly do the the negative log model evidence and over time the BFE it starts out it's an upper bound so it's always greater than or equal to the negative log model evidence over time it decreases and we get you know somewhere around iteration 20 what eight well sorry 35 or something like that we get close enough to the negative log model evidence. So that's example 9.14. What was that? That was a hidden markov model where there was only two hidden states and you weren't allowed to transition. You weren't allowed to perform actions to transition across time. So I've I've implemented this here. This is a a current work in progress. it's not currently on the the page. I'll make it available on on the page and hopefully we'll put it into the the repo itself. So I I try and uh well I implement examples 9.4 and 9.5 as well as figure 9.15. So this this is you know write out everything and I sort of explain the setup here. Um but what we do I should note as well so this this is based upon the code that is not publicly available that Sanjief has. He's currently still working on a couple of things like that. So it is meant to be, you know, high fidelity to to to the actual examples in the book, but they're not one they're not like bit forbit reconstructions just yet. So we're going to use numpy uh just base numpy for everything. Uh that's the idea. So no no other libraries really should be involved. Um just to keep it simple. Obviously there are and there is you know I talked last time there is a library called pi mdp which is exactly a python library for doing discrete state space active inference and it's beautiful it's really good um I think it'd be good for us to play around with instituting some of the examples with pmdp but in order to understand the process of you know what's actually happening it's good to go under the hood and actually play around with matrix algebra and see what's happening so what do we need to do we need to make our a matrix our b matrix our vector. Okay, these are just numpy arrays. Um, I've got a few kind of things in here, you know, trying to get you to think about what might be happening uh before we actually do stuff. And then I talk about, all right, well, what's the free energy machinery at work? Well, we we have a notion of the the VF for the the discrete state space model. That's what I just worked through. And we're going to do coordinate descent in order to to do everything here. So, there's a few there's a few helper functions that I have. One is just normalize takes an array and turns it into a probability distribution effectively. We've got exact bays. So we can compare to the exact bashian case. This is so simple. We can we can do that. So we can see how well we're doing basically. Um and I've also got a function that evaluates the variation of free energy. So that is just implementing this this function up here. Uh which is 9.29 in the book. uh make maybe some better choices could be made about how certain things are are happening here. So 9.4 9.4 is static. Nothing's actually changing in the generative model or or process rather. So that's quite nice uh for initial motivation. So we can just kind of do a bunch of iterations of variational inference and we'll get figure 9.14 at the end of this. Uh so if we do that so what are we doing? we're doing coordinate descent across our two dimensional simplex. Uh I have a function to do that coordinate search. So you can go through here. This this is um effectively algorithm 16. This this little function here. Um it's not necessarily written. So you know all of the algorithms in the book are are pseudo code. If you don't know what that means, it's just a way of saying here's broadly speaking a procedure that I would like to define uh like a recipe and I'm going to talk about it relatively formally sort of whereas this is literal Python code that a computer has to understand. So pseudo code is helpful for general you know we don't have to worry about the syntax of particular languages but this is in code in Python you know the the implementation. So if this was written in somewhere else it would look slightly different doesn't doesn't matter too much. Maybe I should make the implementation look similar. So okay, we we observe that we're in states uh were one. So there zero and one. Okay, that's an actual observation we get in which means that the belief that we're in hidden states uh so there's two hidden states zero and one. We observe that we're in one that's the second state. First state is zero. That means that our belief that we're in state zero that should fall. Okay? Okay, because we got an observation that we're in the second hidden state and the VF should fall accordingly as well. So let's actually see what happens if we run this. So let's run I just you call my function coordinate search. Uh that's going to do coordinate search for me for 60 iterations. Okay, so you can see you can play around with how many iterations you want 5 60 500. You can you can play with that. Uh and we get this. So we start out, you know, we had a uniform belief 0.5. The belief that I'm in hidden state zero is the blue line and the belief that I'm in hidden state uh one is the orange line and I observe one. So across time across well sorry not across time across iterations of the variational free energy minimization algorithm coordinate search. we see that we actually do manage to converge to the the true marginalss in this case which is very nice and of course we have the exact versions in here as well. So this is actually not taking place in time at all. This is taking this is like between time steps and we'll see what that sort of means when we do figure 9.15. I'll stop there and then maybe take some questions because that's a very important point. People typically get confused about iterations of the variational free energy minimization procedure and iterations of the simulation and they're two different things. So this is all static. There is no time in which the simulation is happening. I sort of get an observation and I can just do you know iterations of inference. >> Am I the only one that don't see your uh notebook? >> Oh I hope you I hope you're the only only one. Uh let me come back here. Uh, stop sharing and then I'll reshare. Was everyone seeing my notebook? I I I truly hope that they were. So, do you currently see is showing the COD page? No. Oh dear. Well, that's very terrible. Okay. My apologies for that. Let me just share it again. Uh, whole tab. There we go. I usually share my entire screen. All right. rather than recap everything. First of all, can everyone see the uh the the um code now? >> Uh yes. >> Yes. Okay. Oh dear. Terrible. All right. So, effectively we've got the setup for uh the example 9.4. Okay. Um what I do is I define the model. I do that in in markdown and in code. So, we have our A, B, and D matrices. they're just specified in numpy. Um I then talk about okay well let's institute the the VF functional itself for the discrete case and we're going to do coordinate descent to minimize that. So I have some helper functions to do that basically and also exact bay so we can compare to that. Uh let's actually take in the exact versions of everything it issued in in example 9.4 for and then if we run this well I I have a little bit of discussion about what we would expect to get as a consequence of this especially in terms of the VF as a surprisal bound. I then talk about okay well what is coordinate descent I have a little bit of stuff there hopefully you know motivating that and then I have my little function to do coordinate search and this is effectively algorithm 16 as what I was saying before now I can just run that uh I've run these cells up to now so I won't sort of run them again and here we go this is what I was just talking about but no one could could see so we start out you know uniform belief everything's 0.5 across iterations so there's 60 iterations here. You can change how many iterations you want. I observe uh hidden state one or I observe 01. Okay, so I'm in the second hidden state which means that my belief that I'm in the second hidden state should increase and my belief that I'm in the first hidden state should decrease. Okay, the two hidden states in this case and that's exactly what we observe. So the belief that I'm in the second hidden state that that probability increases and it increases exactly to the true posterior. We can evaluate the true posterior because it's so simple in this case. That's the the dotted lines. Uh and that shows you you can do variational minimization in a discrete uh hidden markoff model with coordinate descent. This is not happening quote unquote in time between time steps of the model. So that's what 9.15 does. Uh and this is the corresponding VF okay across iterations of the uh VF minimization procedure. So this is all happening kind of within one time step. You need enough time quote unquote between time steps to do this because you if you imagine you've only got a certain budget, right? Let's imagine you've only got this is not going to work exactly but maybe only 20 seconds uh to or 20 milliseconds to to to minimize VF to do inference from time step to time step. And if that was directly lined up with iterations of the VF minimization procedure, you'd only be able to go this far. Okay? So you wouldn't actually quite get to the true posterior. Um so this notion about having to do VF minimization quickly enough in our model this is a big big thing that we're going to see later on and we'll see it now. So uh part five this this is just 9.15 or example 9.5 exactly the same as 9.4 except now we're transitioning hidden states across time. Okay. So we we in addition to the observation likelihood and our prior we have the transition function this B matrix right and that's that's going to take beliefs about hidden state now to beliefs about hidden state in the future by one time step single transition. So this is exactly the same except now we have those state transitions. Um so I have simple generative process I define you know the very sort of simple generative process is defined in the text as well. Example 9.5. We can run this for let's say five steps of the environmental process. All right. Now I can run variational for energy minimization with coordinate descent in this model across time and I print out some things here which are so you know at time zero you get an observation one. You see what the prior was you see what the exact posterior was and then what the hidden well the variational posterior was. And we do that across all five. There's only five time steps in this in this simulation. Now the the actual trajectories are the the interesting thing. So let's have a look and see how the beliefs changed across time. Okay, I can run that here. Here we go. So that's effectively what is that here? Figure 9.15. That is showing you that across time. So between time step well you know within time this first time step here my beliefs updated with respect to the observation that I got. So in the first time step I got observation uh one just as I did before. So my belief that I'm in state one should uh increase and my belief that I'm in the other state should decrease. But in the next time step and VF minimization is happening within this this time step here. In the next time step I get a different observation. I get observation zero. Okay, so that's changed now. And that means that my beliefs are similarly going to have to change. So here I transition to thinking, oh no, actually I really am in state zero because that's the observation I get in. I should mention I've glossed over this, but the the observation likelihood, well, I haven't quite glossed over it. If I go up here, so the look at the the A matrix just for a second, this one. So if the A matrix was just a diagonal matrix with ones on the diagonal and zeros everywhere, then if you observed something that would tell you exactly what hidden state you're in. Okay, so this is close to diagonal in the sense that you know the diagonal entries are kind of close to one and everywhere else is kind of close to zero. But that means that there's some uncertainty about the state I'm in given the observation. Okay, but if it was if it was truly diagonal in the sense that it was the um uh the unit I'm I'm forgetting my linear algebra, but you know, totally diagonal with ones on a diagonal, then that there would be no uncertainty about what state you're in given an observation. But that's not the case. There's a little bit of uncertainty about what state I'm in given my observation. And that's reflected across time. And the crucial thing is that's all the way down here between these time within this time step I have to do VF minimization. Within this one, I have to do VF minimization. Okay. And that takes effort that takes computational time basically. uh and we can we can also visualize the VF minimization procedure itself. So this is now so the idea is that between time steps I have a a prior belief about what hidden state I'm in. I get observations in at a particular time step. So these it's pink but I I get an observation at the end of the first time step uh that I'm in a different hidden state and I have to update my beliefs as a consequence of that. Let's go up here. So we're in uh yeah so we start out really able to minimize our VF uh basically in the first time step. So we decrease from there. Um but then the next time step I get on observation zero. So I need to be in that state and then my VF starts out in a different place basically because that's a different hypothesis. um because the trans the the environment is transitioning across time, we see that there's this kind of jumping nature of the VFit. So that's that's example 9.15. Very nice. Now that's excellent. I I I swear I'll stop talking in a second, but um the well I also compare this to exact Beijian inference. That's nice in terms of the um let's go back. I want to share this tab instead now. So that's that's coordinate descent VF minimization in in a hidden markoff model. Uh but the a much nicer way to do it typically a much more common way to do it is gradient descent. Okay. So coordinate descent is a little bit like gradient descent but if we do true sort of gradient descent on the VF I implement that as well but I just want to show you in here. Okay. Yes. Yes. Yes. Yeah. So we start out basically 9.32 what we can do is we can do gradient descent where we we take the gradient of the VF with respect to hidden states that gives us something that looks very similar to a prediction error. Okay. Um and I talk about relations to previous chapters and that sort of thing. Now at a stationary point where beliefs are no longer changing the gradient of the VF with respect to hidden states would be zero. So our prediction error will be zero and we would have exact values. Basically these aren't quite the same as the prediction errors that we've seen in the continuous state space formalism. Um but nevertheless uh the procedure that you would go through is almost exactly the same with coordinate descent except at each iteration you would be finding the gradient of the DF and not just kind of stepping in however many directions you care to step in. I do implement that. Uh let's get back here. Share this tab instead. Everyone should see this now. Ah yes. So I do algorithm 17 which is actual gradient descent for the hidden markup model. That's probably the one to to look at in terms of, you know, if you were thinking about what's sort of the most general way of of doing the minimization, gradient descent is typically what you would do, but it's very much very very similar to what we just did in with um coordinate descent. You can have a look at that there. The last thing is that I have this little plot here plotting the the mean L1 distance of my approximate posterior distribution to the true Beijing posterior with respect to inner iterations of the variational inference loop. So you know here I'm only allowing five iterations. Here I'm allowing 10 and so on. And you can see that the dis what this is effectively showing is that the distance that my approximate posterior has to the true posterior gets better the more inner iterations I have to do variational inference. And that makes sense like if I only allow uh you know five iterations or five steps to do the effect minimization. I'm not going to get as close to the true posterior as I would if I had longer time steps. But of course if you have longer time time steps that costs more in terms of uh computational budget and that sort of thing. So it's a fine line to walk figuring out how many iterations to give the variational inference procedure itself. All right, that's kind of the end why I had a look. If you actually do gradient descent updates, you get much smoother transitions because we're sort of doing well, they're not continuous, but uh it's a generalization of what we did for for the the um coordinate descent version. That's all there. It's not currently available on the page, but I shall make it available. I think it's useful to go through that example, those examples because they really do cement your intuition in a much much more better well a much better way than uh than just kind of looking at equations. Well, uh I'll stop just for a second, come back. We haven't quite got to the end. We we we still need to talk about policies 9.4, but are there any questions thus far on on things? So, apologies. Oh, Andrew, I see you've got your your hand up. Oh yeah, thanks. I I I definitely want to make sure you have time to get to the rest of the chapter, so I'm going to try and stay brief, but uh two things are um I really like the um figure 9.15. Um just for those who are interested in sort of the this the neuroscience there and as Frasier referenced, you know, this is kind of the view is that you're operating on different time scales. So within this kind of discrete time simulation where you're receiving uh you know the agent's receiving one observation per time step you can still see that there can be this fluctuating signal between the time steps that is the agent doesn't just jump immediately to the best answer. is being able to use um you know iterations and gradient descent to to within a time step at a faster time scale carry out various updates and it's just I I'll share a paper there have been a few papers like this over time but for those who are more directly sort of interested in the in the neuroscience around these things um and how that's been viewed in relation to things like um uh neuroiming and and uh EEG and so on like there's actually a lot a relationship between this and trying to fit data to uh things like um you know uh EEG data and and lo computing local field potentials and the like we're actually these intermediate steps are sort of following at roughly the time scale or could be viewed as functioning at the time scale of of the way we collect EEG uh data. So there there's just a lot more to it. uh and of course this book is not purely about neuroscience so I'll just kind of stop there but just for those who are interested in that sort of thing and then secondly there was a question in the chat uh regarding how do we what are actually the units of measurement for things like variational free energy and expected free energy uh and and yes they are so they are measured in nats which uh is a unit of measurement that sort of comes out of um like information theory and studies of entropy and so a nat is just this uh unit that's measuring something like entropy of a system. So actually you want your nats to be very small and you can see how the way that we frame VF and EF is that we're ultimately pinning this is like a minimization problem. So in a sense you're you're kind of trying to minimize your NATS as it were and then uh later in the chapter EF which I'm sure Frasier is going to get into but uh we we break things down into like this pragmatic and epistemic value or what we'll call reward seeking and information seeking. So each of those are like uh computed in nats as well. And then the ef ends up being a kind of summation of nats. So you're you're always on this nats uh field. And uh and then finally we're we're typically using natural logarithms whenever we apply log transformations and that's what allows us to say nats as opposed to bits which would be a base two I think. Um so yeah there's note on the units of measurement but anyway. Yeah thanks. Yeah, the uh just quickly on that. So the the active inference the original active inference textbook from 2022 they actually they do talk about NATS and things like that um in quite a nice way I think I think Sanjie does in the mathematical appendix as well. So chapter two the low directive in the original book and that's free um they you know NATS are spelled out there. So yes all right so uh going once going twice stop sharing for a second any more questions there other than that I think we want to get to 9.4 for controllable state transitions and that kind of thing because Andrew is exactly right. Um expected free energy minimization is really where things kick into high gear with active inference. So um all righty I shall come back here share this just the fundamentals. Very nice. Okay. So that that notebook shall be available on this page um after this session at the very least. So I would strongly recommend you have a look in in that and play around with you know different iterations and that kind of thing because that would give you the best intuition to be able to go forward into what we're going to be looking at just now. So 9.4 doesn't quite get us to planning uh but the whole point of 9.4 is to say all right well we can do the minimization we can do approximate inference now in our in our hidden state space model. Let's do the full story in some sense for the partially observable markoff decision process. We want to be able to choose actions that actually get us to uh transition into states that we would like to transition into. How are we going to start thinking about that? And indeed that is very important. That is kind this is the active bit of active inference. You know this is extremely important this this section here and the following section. So you know I warm up uh everyone again with respect to the notation that's all quite boring to to think about. So let's just sort of go through generally what we're we're we're hidden markov model says well you know state's in some uh my state is in some state at some time and it can transition to another state. Okay but we're going to assume now that we can issue actions or controls that determine state transition. So this the probability that I'm in a given state now determine is is uh depends upon the probability of the previous state and the previous action that I was in. Okay. So I can issue actions. That's kind of the the new new thing that we're going to be looking at uh there. So the the the initial example that we're going to look at is a very very simple setup just a grid world where we've got a 3x3 grid and you know there's certain states so you know state zero state one state two we can index them by rows and columns as well. Um and we're going to have a look at uh controllable state transitions in in this very simple environment. And even this very simple environment is going to uh play host to to some complexity that um will bedevil us with respect to lots of stuff that we do later. So what do we have to do? Well, we we know that uh we have to introduce state transitions. So we have this B matrix. Okay, that's doing state transitions for us. Uh and we had that in the hidden marker model as well. But we now introduce actions. We now introduce things that we can do. So that's going to kind of enlarge the the B matrix and turn it into something called a tensor. Now uh so a matrix is you know table with two dimensions. A tensor has kind of more than two dimensions. It's got you know rows and columns and depth. So this is like a cube of numbers now and they're all going to be well initially they're all either populated with zeros and ones but we're going to see that we can generalize that as well. So really this is a crucial distinction to keep in mind. 9.4 does not tell us this section does not deal with um how to choose a policy or how to choose actions that the agent uh needs to choose. Okay, that's that's a very important point and that's the active bit of active inference. It makes the generative model itself controllable. So it sort of introduces the affordance or the ability to control the generative model. Uh and we're not going to look at how to control things just yet. That's going to be 9.5 and beyond. and also how to it also shows us how to predict the consequences of an already specified action sequence. So we're going to assume that we have an action sequence that's given to us. It's a good one basically. So we need to enlarge the uh the B matrix uh and we're going to do that with our very simple grid. The visualization to keep in mind here is figure 9.17. This is what we have now. So we have a B matrix for every single action that we can take. So if we're in in our in our grid world here, let's presume presume we start out in the center and assume that we can go up, down, left, and right. So there's four actions that I can take in any given state kind of sort of depending on where you are. That means that our B matrix now is going to have to have we're going to need a B matrix for every action that we take that says how we transition from the current state to the next state given the action. Okay? And so that we've got copies and we sort of put them in we stack those copies depthways and we end up with this block of or cube of of numbers. Now that's going to be our state transition tensor. So you can see even for this very tiny example the B tensor is already quite large. Um and this is an unbelievably tiny environment over here. So sort of foreshadowing some computational difficulties we'll get into. Uh the really the last thing to talk about in terms of where we're going down down down is we now have the ability to talk about policies or or sort of actions that I can take sequences of actions that I can take across time. Uh that's going to be with us for a long time. So we know in this initial example up down left right let's just sort of encode that 0 1 2 3. Um, and we can now use my we can use our model to say okay well where would the environment go if I took a certain action. So if I took action let's go here if I took action down from the current state. This matrix here tells me the probability of transitioning to the next state given that I'm in the current state. So uh we're indexing things along columns. There's someone trying to enter into my room here. Uh I'm afraid I'm going to have to call it there. But that is what allows us to talk about um transitions between states and we'll see next time with EF minimization that uh how to actually choose these transitions. That's that's the key thing. The last thing is the representation of these policies across time. It talks about that in terms of the uh the uh what do I have here? the decision matrix. Okay. So, we've got policies, sequences of actions. They go for a certain horizon. We've got so many policies and we're going to evaluate those. Uh, yes. Terribly sorry, I will have to end it there. I don't know, Andrew, if you can potentially take over if I can give things to you. >> Yeah. I mean, >> I don't mind. Um, well, I No, unfortunately the host and does the assign. Yeah. So, but I do strongly because I do pick up on some of where you left off. So, I would propose maybe people check out the recording from this past Tuesday which you can find on the meetings page in the KOD. Um, you know, all the links to all the videos are there and I go into yeah, I kind of um pick up some of the pieces and then of course there's always our third week next week where we can um pick up where we left off as well. So, >> so I'm thinking next week will be true kind of e minimization. we'll see how to select these policies basically and how they affect transitions um and that will really be kind of you know full-on active inference then in the discrete state space and hopefully we'll we'll go through some examples in the same way that we have here I'll make that example available on the on the page uh so see there's a few things thanks until then yeah do do put questions in the in the coder um I have I have time to look at them now uh but that shall be that'll be very exciting coming to to full on active inference next All right, I will end the recording there. So, goodbye YouTube people.