Fundamentals of Active Inference (Chapter 10, Session 46) September 18, 2026

All right. Hello everyone. So it's September 18th, 2026. We're here in fundamentals of active inference uh the reading group for fundamentals of active inference by Sanjiv Nam Jooshi. We are in chapter 10 now. Uh we've come quite a long way and uh basically this session uh this is my my first session on chapter 10. Andrew in his sessions is he's already gone through uh the first of three. So we're going to be doing three sessions on chapter 10. usually is just two for um for each session, but chapter 10, well, chapter nine and 10 are they're really the culmination of the book in terms of the actual fundamentals of active inference. We've already seen how to make active inference agents in sort of seven 89. Uh and we've seen the beginnings of this last topic really in terms of the fundamentals uh namely planning and this having this counterfactual ability. That's kind of where we left off with chapter nine. um in the discrete state space setting. Okay, so we've we've done everything in the continuous state space setting for basically everything up until chapter 9 and we've now switched into this discrete space state base setting and that re that that transition is to set us up for uh being able to look at this sort of planning procedure. That's kind of where everything's going and we've we've had our first kind of look at that in in chapter nine, but we're going to be seeing more of it in chapter 10. effectively uh and well so we're beginning chapter 10 today. Effectively the the point of eight and nine is that nine tells you you know this is this is the the formalism for doing discrete state space active inference planning counterfactual uh modeling of of the future and so on. But as with almost everything that we we do when we introduce a new sort of formalism thus far in the book, we we have to make a whole bunch of simplifying assumptions first of all to initially introduce what's happening and then in later chapters we relax those assumptions. So for instance the the assumption that we've held up until now is that well you know we have a good model of um the observation generating function. We have a good model of state transitions and so on and so on. That's what we're going to be relaxing today. So the idea is we're actually going to be uncertain about state transitions. We're going to be uncertain about observation generating functions and we're going to have to have beliefs about that which is to say beliefs about the parameters of our model. So yet again we're going to be saying how do we do learning in in this new setup now? How do we how do we learn the parameters of our categorical distributions? That's what 10 is all about. Now rather than 10 is not massively long but there is enough information in 10 is there's actually quite a lot of information that I think it would be worthwhile to focus just on section 10.1 today then next week we'll do 10.2 and then from there 10.3 to the end of the chapter. I think that's going to ultimately work out to be quite uh quite worthwhile. That does mean that it's a little bit uh it can be a little bit dry. I'm sure those who have tried to read 10.1, it can feel like a lot of notation and so on. Um, that's that's just sort of unfortunate. I do have um examples though. I have some code examples. They're not currently up on the page, but we'll be stepping through them and they will be available by the time people are watching this on YouTube. So, that's the plan for today. So, if I come across here, I'll share my screen uh and then I'll show you the page as it currently exists. back over here now. So, here's chapter 10. That's where we are. What I have currently is uh I don't have the outline for the entire chapter. I've just got it for 10.1. 10.1 is quite long. Uh and it sets up everything else in the chapter. So, I kind of uniquely um 10.1 is very important for, you know, everything in chapter 10. If we don't get this sorted, then we're we're going to be in a in a bad place for everything else in the chapter. So this is currently just for 10.1. So coming all the way down here at the very bottom we have the figures. The figures are again they're just for 10.1 at the moment and also for appendix C the mathematical appendix. So this is appendix C. Well he talks about this in the in the very beginning of the chapter. Appendix C is um quite relevant to chapter 10. I would almost include it as just an extension of chapter 10 directly. Um so he the whole point is that you know he talks about saying we're going to assume familiarity with dishlay distributions. We'll see what those are in a second. Um in the actual content of the chapter and then appendix C sort of you know talks about what is a dish distribution where does it come from? What does it look like? That kind of thing. So I've deliberately put the uh the figures for appendix C in here as well. really they'll they need to go into their own their own section, but they're there. All of the figures for 10.1 are there. The equations as well, thanks to Andrew. They are in the equations section, which I had up somewhere. Here we go. So, chapter 10 equations, they're all there, which is very nice. There's quite a lot of them, especially for when we get to habits and learning later on. Okay. Now, the last thing, if I go all the way back up, my uh my notes, my latte notes that I I've been writing, they are available as well for chapter 10. They're here at the very top, the sort of PDF version. They're about 13 pages long. Uh they're not just a direct recapitulation of everything in the book. They're designed to be an elaboration, as I keep saying, of maybe some things that are um glossed over. No, not maybe not glossed over, but they're designed to go in a bit more detail into into what's happening. So I have reproduced appendix C.2.3 on the dishlay distribution its relation to categorical distributions and things like that immediately. Uh so we go down down down and then part two is actually where we begin with chapter 10 and then from there there's there's the equations are sort of reproduced so that for those who don't have the book they can follow along at least um and I try and explain elaborate in a bit more detail than I can go into in the live sessions in these notes here. So that is available. That's just 10.1 though. So the other other sections are forthcoming. Uh so as I say 10.1 today, 10.2 next week and then 10.3 and 10.5 the week after. So if we look down here, uh and please do uh I'll I'll come across after I I have my first little spiel, but if if you do want to interrupt at any time, uh feel free to do so. So this is the map for chapter 10. Uh basically we've got um five sections. What we're going to be doing is the whole point of chapter 10 as I say that the the really high level overview is well what if we're uncertain about our model? What if we're uncertain about the state transition function, the observation generating function? That kind of thing. How do we deal with this? How do we learn the parameters of our categorical model in this case? That's what chapter 10 is about. Um, and there's kind of two complimentary uh things that need to happen if this is going to be how we're thinking about things from now on. Uh, and really these are the two points of chapter 10.1 that we need to be able to learn the parameters of our model. That's number one. But number two, we need to be able to sort of seek out and uh find states of affairs that would give us information about our model. That's a bit more of a subtle thing and we'll see what that means later on. Those are the two kind of pillars of of what we're going to be doing in terms of learning. So we just need to be able to learn the parameters of our model, but we would also like to be able to seek out state seek out conditions that will give us information about our model. That's that's where we're going. So if I come back up to the notes up here, the thing is if I I'll go down to part two just for the the beginning. This is what you would see if you were just reading the book from sort of start to finish. We need to now entertain. So we have our our well we have our categorical uh model. We have our A matrix mapping observations to latent states um sorry latent states to to observations. We have our transition tensor B. Okay, transitioning uh hidden states across time but also conditioning on actions that we can input into the state transition process. And we know that we're talking about those actions as as or rather sequences of actions as policies pi. So we've got our A matrix, our B matrix. Really, it's a B tensor because as we saw in chapter nine, we need a matrix for each conceivable action. So it's actually this block of numbers here. I'll just show you visually again. This is a nice figure from chapter 9 9.17. So we have an A matrix but we have a B tensor in general. So you know states current states along the rows, previous states along the columns and then actions uh in the other dimension. All right, that's two things so far, but we know that there's more and there's even more um uh matrices coming R2. We also have our state initial state vector D. This is saying okay well where do we think we start from in terms of the latent state. Uh excellent. And we also have the last thing we saw in chapter 9 was this sort of prior of a policies. Very important point in active inference. That's extremely important where this comes from and we're going to be doing more about that in terms of how we select policies, how we evaluate them, that kind of thing. But the thing that we're adding now is a prior over model parameters themselves. Okay. So we've got our joint probability distribution, our generative model over sequences of actions, sequences of states, policies, sequences of actions, and then also parameters, the parameters of the model. So the parameters of my observation likelihood, the parameters of my state transition and the uh initial state vector. Those are the parameters we'll look at so far. So that's what we're doing. That's what we've we've uh done so far. So that looks like if I come down into the figures, figure 10.2, this one here, this is this is just a a beige network representation of our generative model. Again, so again, as I keep saying, so circles are things that we're uncertain about that. So we have a probability distribution over them. We have a belief over them. And shaded in circles are values that we observe effectively. And arrows say, you know, this depends on this. So that the next state depends on the previous state but it also depends upon my policy. My policy depends on the expected free energy and expected free energy depends on my preferences and so on and so forth. Of course then in the vertical direction states give rise to observations. An observation depends on a state. But now observations also depend on our belief about the observation generating function A. So we're going to have we're going to have to have beliefs about that mapping. And likewise, we're going to have to have beliefs about the state transitions. Those those B's now. So the only thing we've added in is that we've made these B's and these A's probabilistic. We now have beliefs about them, which means that they are parameterized. They have parameters. Uh and the way that Sanjief chooses to notate this in the book is sort of, you know, 10 points two. we we're going to say, okay, well, you know, our belief about state transitions, that's going to be parameterized by this B inverse hat thing, and D is going to be parameterized by by a little D, and A is going to be parameterized by a little A. So, he's got he uses this inverse hat to denote the parameters of uh parameter beliefs. That's a little bit of a a doozy. So if I go back in here, what this actually amounts to is a factorization of our belief about parameters. So if I come all the way down to part two again, what am I talking about? So okay, we need we need a belief about parameters and we're just going to say okay well what are the what are the parameters at issue? They are of the observation likelihood, state transition and initial state prior. They have their respective uh beliefs down here. These should be inverse hats. Sorry, they're tildas at the moment. And I can go back and fix that. And we're just going to assume that the belief over that entire parameter space factorizes as the belief over A times the belief over B times the belief over D. That's that's our sort of uh mean field factoriization if you will over a state a parameter beliefs. Very very nice. So the thing is our beliefs as they are now they are categorical beliefs. They are like this. So back here they are things like this. So we've seen okay you know let's say I've got categorical categories of hidden states and there's a belief in ter there's a probability for each one of them. So you know rainy cloudy sunny snowy very very common example of the weather. We now have these kind of bar charts of probability distributions. Those are the beliefs that we have. But now we need prior beliefs over uh probability distributions. So how are we going to do that? That's that's kind of the question. The thing is um so does you know page 257 and 258 they're quite important for explaining what we're going to do just now. The idea is we need these beliefs over parameters. um and they those parameters parameterize categorical distributions. So what are the beliefs over parameters going to be? That's the question. Effectively they're going to be these they're going to be given by another probability dist distribution called a dishlay distribution. So we've got categorical distributions and dishlay distributions. Now and there's a lot of stuff to be said here. it gets kind of complicated fairly quickly but I I actually start the notes with uh with you know this appendix stuff about you know what are the degrees distributions where do they come from I don't want to go over that just now I want to sort of motivate what's what they're doing for us effectively so if we look at let me come down to the actual figures what we're going to need is something like this. This is um appendix figure C 3 I believe. Yeah, C3. Perfect. Let's imagine that we've got a three-dimensional um well, we we we have a three-dimensional dish distribution. Now, this is a prior belief about parameters. What this effectively means is that if we can imagine well all all of the all of the uh the probability distributions that we could conceivably think of as uh you know beliefs about say a prior state they have to live on this on the surface of this uh this triangle here. That's the that's the idea. So this is this is just the particular example of a three-dimensional um dish distribution and the idea is that every single point in here is going to parameterize it. It was going to be a categorical distribution. So that looks like this. So these the idea is that every one of those points they are potential categorical distributions over one two three uh categories effectively. So and the idea is that I'm actually uncertain about what those that those that categorical distribution needs to be. So I have a distribution over distributions and that distribution over distributions is my durishlay distribution which is saying how uncertain am I about the actual parameters in in my probability vector. So as I say you know it's a little bit meta gets a bit crazy. Um and he has on uh as I say on page 258 and 259 he has the expressions which which you know tell you they're not too handy. There's some some better um figures which will are more interpretable. There are expressions that will tell you how to update your beliefs about uh you know either hidden states uh or you know how to update the parameters of your hidden state belief your your um observation likelihood and your prior over hidden states as well all in terms of these durate distributions. So the actual functional form of a der distribution looks like this. So the idea is I have a belief over parameters. Okay. And you can kind of think about those parameters as another kind of hidden state just like we did in chapter three. We thought of parameters as another kind of hidden state that change at a slower time scale. Um so that's kind of like a hidden state or pseudo counts. And there's there's interesting, you know, subtle points as to to why that's the case, but I don't want to get into that just now. The idea is if I go back to yeah, let's go to figure figure 10.3. So, example 10.1. Let's actually get concrete. I go down to figure 10.3. This will hopefully bring some things into focus. Here we go. 10.1. So what we're looking at here, imagine we've got just a two we've got a um a discrete distribution over just two hidden states effectively. The idea is we're we're going to be uncertain about where we start where the agent is beginning its it's uh what what its initial state is effectively. So we're going to just imagine that we start, you know, with one one, okay, as a sort of arbitrary hidden state at, you know, time equals zero. And the idea is we're going to need to update our belief about the parameters of the hidden state belief over time. And the language for this, we've talked about this kind of um colloquially. You know, we're doing hidden state inference relatively quickly, but we're doing learning relatively slowly. We're going to be updating the beliefs about our our hidden state uh belief relatively slowly and the the the way that we sort of talk about this in terms of dish prior is that we're going to be doing trials. So over a trial a whole bunch of inference is going to be happening. All right, I'm going to be inferring hidden states, selecting actions, yada yada. know that story and my belief about the hidden state uh well the the parameters of of my hidden state belief they're not going to be changing but from trial to trial they will be changing okay so they'll be sort of changing slower than the the actual inference process itself so for a two for a two hidden state distribution uh we have two parameters in the dish lay distribution itself and they we know that if we've got you know a two hidden a vector that the sum of this whole thing has to be equal to one. So what that means is that in this particular case the actual points the actual values of our hidden state belief they have to lie on this diagonal line here. So you know I've got P1 and P2 well P1 is equal to 1 minus P2 and P2 is equal to 1 minus P1. So any categorical belief I have about where I start in this two dimensional hidden hidden state space has to be somewhere on this line. And the idea is that I'm going to have a distribution. I'm going to have a belief over spaces belief over where I am on this line in terms of my initial hidden state belief in the beginning. So these that's what these green samples are. Basically you can kind of sample your derishlay distribution your belief about the parameters of the hidden state belief. That's what this green is. The purple little dot that's the mean of the dish distribution itself. So showing the samples as expected values from dishlay over 50 trials using 51 initial state samples. Green points are drawn from the dish distribution itself. Uh the red cross the true value. Okay. So yeah, the the the purple dot that's the mean of your derishly distribution and the red square is the actual value of the the hidden state belief itself. So you can see across trials, we start out very uncertain about the parameters of our hidden state belief. But as trials proceed, we kind of learn more and more ah no the the value of my hidden state belief the the you can see the mean of that is getting closer and closer to the to the real hidden state which is of course what you would like to see across time. So if we come across to my example I'll just note so what I've done is I've gone through and I've implemented examples 10.1 to 10.4 for as well as a lot of stuff from from the appendix. They're it they they are implemented but they're a bit uh raw. They're not currently available on the code page. So after this session, you will see coming across to the code here. Scrolling down, I'll put um a new entry in here for chapter 10 and you'll actually get to to play around with it. So I I I talk about well let I actually go through this and implement the example. The idea is uh I've got a bunch of things I need for helpers just right now. I start out with the appendex going through you know talking about the relation that categorical probability distributions have to the dishlay distribution and what this notion of a simplex is. I should have mentioned that before. So if I've got a three-dimensional categorical distribution, the space of parameters that my der distribution can take is actually two dimensional and that's what this thing is here. It's called a simplex. So the idea is you know my my belief about my three-dimensional hidden state that lives somewhere here effectively and those that point parameterizes my belief about hidden states. All right. And so there there is a lot of that detail to go through. There's you know stuff about the dish distribution itself. But if we run let's run just need to make sure we run everything in sequence. So this is the in the appendix he has a very similar um example to the one we were looking at but let's get to the actual one just now so that we can have a look at the I re regenerate the probability simplex itself from from the appendix. We can go back I think uh toward the end talk about that. I want to get to my actual example. Yes, section 10.1. So this is this is where we're actually going to be doing the update that I just talked about in in general. So we have our generative model. We have beliefs over parameters. The way that we update the parameters of our dishlay distribution is by means of these expressions here. here and these are on page 258 and 259. These are simply the the consequence of um the the updates that you get when when you have a dish prior. I don't want to necessarily justify that because it can get a bit hairy and it's not too important for the intuition that we want to build just here. So this by the way I'll just note that this at the moment is just a hidden marov model. So we're not we don't have actions. We're just going to look at states transitioning across time because that's that's a bit uh bit simpler to deal with. Very nice. So, example 10.1 all the way down here. So, this is this is what I was just talking about. We're going to have we're going to try and update our belief about the hidden state prior D. That's that's our vector. We're going to just assume we start out in one one. Who knows if that's really where we start start out. And at the beginning of each trial, we're going to have a certain number of trials. In this case, 50. um the agent's inferred initial state belief is going to be uh starting out at 0.9.1. So yeah, the actual the actual parameters of our durishlay distribution are different from our hidden uh initial state prior. So the the parameters of the the der distribution are going to start out at 1 one and we're going to be updating that across time and the actual update is is is here. So I start out with my initial belief over hidden states. Uh but I've got a a history of 50 transitions now. Uh and we're going to go across time and just kind of reproduce that figure that we saw figure 10.3. So if I if I can do just you know after one update I can see what my categorical distribution or sorry the dishlay what the dishlay prior are as a consequence of that. Um I can look at the final concentration vector. I'll have to explain what that is in a second. Uh and the mean of of my dish prior and so on. Uh but if we do this for the for 50 trials, here we go. So this is this is exactly 10.3 from the book. So I start out very uncertain about the initial state and I've only got two uh uh two um parameters of my dish distribution. The parameters of a dish distribution are called concentration uh par parameters effectively or pseudo counts. I'm I I favor a concentration parameter myself. And you can see across time. Okay, this is the initial trial. The the uh concentration vector which is the par the parameters of the ratio distributions 1 one. That's where we start out and the mean is here for the fifth trial. Now we're we're getting 5.5 1.5. So we're getting more precise. The actual value is down here at this this green red square. And towards the end we're very very close to the the real hidden state belief. Right? So that is just raw learning of the initial hidden state vector by by means of changing where we are on a probability simplex. So let me just one more time I'll come back and then I'll I'll answer some questions. So this this is the this is even more complicated because it's three we have three uh parameters but we were just looking at a case that had two parameters although I think the threedimensional case can be nice for intuition. The idea is imagining that we had a three-dimensional hidden state uh prior. The actual uh prior belief is somewhere on this surface and we start out somewhere on this surface and we'd like to be able to move, you know, instead of being down here. Actually, I think I have it in my code. Let's go back up to here. There we go. So you know potentially my this is the this is the mean of the dishlay distribution over hidden state prize for a three-dimensional hidden state belief. But if you imagine if that was the real belief and I started out up here I'd like to change I'd like to move in this simplex from here to here. That's kind of what we're doing down here where it was much simpler. It was only onedimensional you know. Okay. So there there's other examples. I want to go through them. I think they're illustrative. So basically we also want to update the A matrix or learn the A matrix and we also want to learn the B matrix and that's really the end of um this section and then at the really at the very very end we need to talk about novelty and information seeking. That's the the cool bit but before we do I'll just stop sharing for a second. Are there any questions as a consequence of anything that we've seen so far? Has anyone gone through this and and run up against something that they would like to to talk about now before I I go on elsewhere? I could uh chime in Fraser. Thanks. >> Yeah, please. >> Great. Um yeah, I just uh you know I uh I just quickly say that um I think it's great once per usual how deeply uh and clearly that you're sort of going through the the individual derivations and kind of the way that Sanjief presents this material. This past Tuesday I did something a little different. I I definitely kind of stuck to what Sanjieve was trying to do, but I went a bit just a bit off script for for maybe 50% of the meeting. Uh and maybe that's an issue because I have so much uh experience with the discrete states base stuff that uh there there were things that I wanted to address too. Um, so I will just add I I think you've already mentioned it, but the despite how complex the deerishlay distributions may seem at first, I think an intuitive aspect about it is simply that our our dearish priors uh you know, as you said, we kind of accumulate the evidence across trials and it's presented in this trial-based learning way. um for for kind of quickly getting at it. Uh what was very helpful for me and I think some others in learning how these work is simply that your dearish prior can be structured exactly equivalently to the parameter for which they are a prior. So your your a matrix has you know one one sort of slice for each uh observation uh condition on hidden states. So your dearish prior is structured in that same exact way and the indices and elements within your dearish prior exactly map to all the same things in your a matrix. So it's just knowing that your prior is is literally structured the same exact way and it's just accumulating counts of all the same things that you already find in these parameters you've already seen before. Um, so yeah, I I didn't want to spend too much time with that, but it's just I I will say if if you want to just like look at a quick example, I think slide 4140 actually. Um, >> thank you. And I know these are quite jam-packed per usual, but you'll see, um, towards the middle of the screen, uh, and just a little bit lower, um, you'll see this little example with all the little blue arrows. Um uh yeah roughly around equation 10.7 there all the things with the D the prior inference leading up to the trial update there it's simply you can see on the the left with the prior your d trial is is just a vector or here a column with with two indices and then your inference also is you know structured equivalently here. So, we're just going to add just going to keep adding to the D. And it's the same exact structure as your your actual D, right? So, it's just your it's it's kind of like your your dearish is just sort of like this um additive um um log or I don't want to use the word log. Uh journal, if you will. you're just kind of accumulating. >> Yeah, you're just accumulating sort of this experience through all these time steps or trials that the agent is facing and then that uh you know gets sort of added to your current actual D parameter and then you normalize. So it's just sort of yeah accumulating this experience over time which then goes on to weight your your parameters and and as you update them. So I Yeah, I just it was useful for some because it was like, okay, well, what what is this prior? Why is it a dearish? Why does it seem so like all the terminology here is suddenly different? Cuz we're no longer talking about category categorical probability distributions. Why doesn't the deer have to be normalized? It's just it's just this accumulation of experience that then goes on to modulate um you know, your current parameters. Um >> yeah, that's it's it's a good point. I I glossed over that. It's it's very important to uh emphasize the fact that we are yeah we're accumulating evidence over time as a consequence of you know the observations that come in uh and then we can update once we've accumulated a bunch of stuff uh for sure. So I thank you for that Andrew that's uh that is an important point and that's what that's what is meant and you did a great job in in your session talking about the difference between like a trial and you know the the total sort of simulation time or just the time at which inference is happening uh for sure. So yeah, it gets the trouble is that it does get quite hairy uh you know mathematically very quickly trying to talk about these things. But um for the the if if you were just trying to focus if I come back here to the to the example once the example is up if you were just trying to focus on the mechanics of the updating of the belief the categorical belief I would focus on the hidden states um the sorry the initial state belief as as Andrew was saying the the d vector because it's kind of the simplest in some sense you know we just have the same structure uh here as well. So, uh, I would meditate upon this this example. I think maybe I'll go back and see if I can flesh out this particular one in in greater detail. So, thank you. Thank you, Andrew. I'll just come back again. Stop sharing. All right. I might uh I'll go through the remaining examples and we'll get to novelty learning. Well, not you know, information seeking. That's really the the key point. So, uh do stop me if uh if anyone would desire. I'll just share the entire screen because that's simpler. Back over here now. Okay. So that tells us how to update the hidden state uh belief effectively d across time. But of course we have the uh so that's example 10.1. 10.2 then says well how do we update the the actual a matrix itself the the observation likelihood. Um, so we start out just with a a relatively simple generative model where we're always going to be transitioning to another state. So we're never going to be staying in the same state. Um, and the observation likelihood matrix is given over here as well. So the question is how do we learn? We want to be able to find this a star. We want to be able to learn those those parameters. How do we do that? Basically that's the the question now across time. So you know TOAO is indexing maybe I should have t instead of toao the book uses uh toao across time I'm going to be getting in observations right o at at a current time step and I have a belief about uh hidden states across time this is this is talked about in the appendix and a little bit in in the main text as well but the outer product of those two vectors is going to give me back something like my my hidden states sorry my my observation likelihood mapping uh and then we can update the this this is the durishlay uh prime well parameter for for the observation likelihood uh mapping across trials. So I've got r indexing trials and I've got tow indexing time kind of you know across a whole bunch of trials. So it does it does get a bit tricky to to to have a lot of that bookkeeping in mind. I try and and go through in in the markdown to to walk through what's the relation with all these things. It is it is a bit tricky but effectively we can use this as an update for our durishlay belief about the observation likelihood um mapping that's that's kind of the the the upshot so what do we do uh and I talk about hit you know um one hot encoded versions of this at the end of the day the belief that we have about our observation likelihood mapping is going to be this the ratio of of our uh the belief sort of now over the the total amount of counts that we've accumulated. That's that's sort of the the flavor of this. This this happens for all of the parameters actually, but um this is just for the case of the observation likelihood mapping itself. So let's think about this. What do we need? We need the we'll just I'll say you know we've got a true and b switch for the generative process. Okay, that's just from from what I had above. Let's do five trials because that's that's a nice number. uh and we'll go for a thousand time steps effectively. So I'm going to just do a bunch of trials now, five in this case, and I'm going to simulate my hidden markoff model. It's just a hidden marker model, not a PM DP right now. Um I'm going to get back the counts, which is to say these things, uh what's the best one to show kind of these these things here or down here in the denominator across trials. So, I'm going to get those back and then I'm going to be able to perform the actual dish um update itself. So, I've got the actual raw values here, but let's have a look at the the figure. Bing, bada boom. This is figure 10.4. In fact, well, this is a re reproduction of figure 10.4. The coloring unfortunately is not matching the book. I'd like to make it match the book before I put it up. So, what are we looking at across time? We're looking at the probability. So each each entry in the A matrix is a sort of probability. I've got, you know, the top left entry a 0, the bottom left entry a 01. I start out where they all have just the same probability, you know, 0.5. But then as a consequence of accumulating these these trials across time, uh I can update my belief about what should be in the entries of the matrix. And you can see relatively quickly at the end of the first trial here we we end up converging nothing really changes after the first trial to to these particular values. So you know a zero that's around about 7 around about 6 for a 01 and you can see that I've got here in in um in little little crosses the actual values as well. So they they are so you know what do we have? we have 7, we have uh 6 and if we go back up to the actual generative model 76 and then so on with the other with the other values. So that's showing that you you can update the belief about what the A matrix should be across time effectively. Uh now the the other plot this is the one that's on the right of figure 10 uh four this is showing the actual the actual uh parameters of the dishlay distribution across time and these are the the pseudo counts or the concentration parameters. So they are uh the I go back up here. I have I haven't notated the actual I think it'd be good to notate where they come from, but these are so in this case they're um well there's four of them because we have a a four this is showing across time they accumulate. So I'm sort of as I get um evidence in I've got I I accumulate the the dish counts. That's what this is showing effectively across time. this that's that's that's one way to well that is the way in which we're going to be able to update the belief about the the observation likelihood mapping the state transition version of this is very similar but it's uh the wrinkle is that of course we have not only uh hidden states generating hidden states but also actions generating hidden states at the moment I've just got the hidden markov model in there uh which is also what Sanjieve has there is an optional extension at the end where you can you can have the um the full sort of pomd well not pom dp but the well yes the full pom dp version. So we start out here with a state transition the actual in the generative model 2.6.8.4 four we start out with an observation likelihood mapping and we want to learn what B should be basically. So the the rule for this is very similar. We have our way of accumulating dish counts across time across trials rather um and then we can do perform the updates if we go all the way back down here. So let let's start out we can start out with a hidden state belief uh 7.3 and we'll come back down into the code. So the true state transition matrix is given here. We're going to do a thousand time steps only five trials. So we're only going to update our belief about the state transition um tensor five times basically is what that's saying across a thousand full trials. So sorry across a thousand time steps. So I can plug the the values. I can actually just run the hidden markup model. I have a function to do that. I can get my counts. I can form the histogram of of counts and so on much like we did for the observation likelihood mapping and this is a reproduction of figure 10.5 in this case um yeah the the I didn't have time to to make the the colors match but we have the same sort of thing here so we start out radically uncertain uh you know 0.5 for everything and then relatively quickly as a consequence of updating by means of these pseudo counts that come in we can update our dishlay parameter and converge to the true uh hidden state transition function tensor and then we we have the the pseudo counts accumulating across time as well. So that is a lot of stuff. Uh that's a lot of handwaving from me. Uh and hopefully this notebook I'll polish it up. There's a lot more to it as well. Um will be useful for those who want to really kind of delve into the details. I think that's probably enough. I would like to get to novelty and information seeking to maybe round out the session but before we do I'll just come back I'll stop sharing come back is there anything that people would like to talk about there with respect to those those three examples those are the canonical examples you know learning by my hidden state prior observation likelihood and state transition mapping so if there are no questions there and as I If you just want to pick one, as Andrew was saying, I would pick the hidden states uh initial states belief, the you know, belief over D to go through as a first example. >> Fantastic. Well, oh, was that a question? Yeah, just a another minor point that I think you you I'm sorry I'm not sure you might have touched on it, but just a brief note about the the B matrix uh or tensor um is Sanji points out like the learning rule works just a bit differently. Technically, it doesn't have to, but um yeah, exactly. They're the action condition B slices, right? Because it's >> for all of these parameters that we're learning, it's as if, you know, it's as if there's some event that happen that is directly relevant to the parameter that we're learning or that we're updating or that we're accumulating counts for. Those counts are particular events that align with the parameters. Which is why when we update update the D um that is you know it's a prior on hidden states or initial hidden states well through the course of more inference we add in our new posterior inferences to the D because those directly have to do with hidden states. Similarly with the A matrix we look at these sorts of combinations of of state inferences that the agent makes uh in reference to the actual observations it receives. And so there we also are accumulating these specific events that directly pertain to what that parameter is sort of for and what it encodes. And so with the B similarly we will update based on what was the previous hidden state, what was the next hidden state or rather what was the agent's state inference about the previous hidden state and the next hidden state >> exactly which is why we have the Q's there um in in the equation you pointed out 10.4b 4 B and so with the controllable uh hidden state factors where we have a B that also has this this third variable in addition to >> this is the HMM in this example which is so we don't have that controllability but yeah yeah >> yeah right right in the um in the yeah that that was exactly the caveat I wanted to add which is that for the B just as you pointed out here uh you know we um you know we we would just that would be perception only whereas this one that you've now scrolled to the equation 106 six is virtually the same thing except we're also including the the the the action that the agent actually chose. And so now we're saying it's as if the agents witnessed an event of well the previous state I inferred the previous state was such and such. Uh then I did an action and now I infer that the new hidden state is such and such. That particular event is precisely what gets accumulated. so to speak or added as an event count into our our prior at the B. Um, so it's just again making that point of like really we're just accumulating counts of these events that directly pertain to what the the parameter is intended to encode and it's just as a slight difference for controllable versus uncontrollable B matrices. But you could also if if you're coding this, you could still write the code such that it would work for either because a um an uncontrollable B matrix could be structured to include action where you just have one hypothetical action, just a singular action that just sort of does nothing. So what I'm saying is you could write code that's extensible to both scenarios. Exactly this. It's as if you could only move up, you know, but but really in in practice, you would say, you know, this is an agent who can't actually do like a null action, right? Like it's an action that doesn't actually do anything. You effectively just have that single slice. Um it it, you know, is as if it does nothing. And then whenever you incorporate that into an environment and actually run a simulation, you know, the agent is repeatedly outputting this null action, you can just kind of have it go nowhere because it it's it's in an environment where it's action doesn't actually do anything. So then that way you have a nice kind of perception uh simulation without having to write multiple like here's my model you know over here with that's you know has code specific to uncontrollable factors and then here's my other code for one that that can control you can just write code for a single kind of model that you could change you know depending on your particular instantiation of that model does have controllable hidden state factors or no >> yeah that's exactly how PIM PIM MDP does it. >> It's very extensible because then as you're saying you don't have to do two different cases, you know. Yeah. Yeah. >> I think I mean that would be that would be a good thing to um to look at. Uh well, I think there's many more examples that would be worthy to be made for, you know, learning state transitions that are than are extant in the book actually. So, it would be good to to spin up some more of those. I just see there's a question in the chat from Gian Cuomo. Uh he says, "Silly question. Where in the temporal order does the learning occur at every step before any other computation that depends on the parameters?" That's a that's an excellent question and it's very important to to what we're talking about. So I'll just go back and share the I'll go back to the notebook. Effectively the idea is we're not going to be so you know the simulation is happening in time. So I say let's let's this is just for the uh for the B matrix. Okay, so there's a thousand time steps here, but we're only ever going to be updating the belief about the state transition tensor uh dur at the end of a trial, which is kind of a a partition of the total amount of time. In this case, we've got five trials. So you can see between zero and one, well, you know, at the end of of one, I update my belief about where I was at the beginning to be now here. And I I I only do that at the end of a trial. So I guess there's a question about you know how many trials should you have in a given simulation. Should you update your belief about the uh parameters of your uh hidden state belief state transition function observation likelihood every single time step? Probably not. No. Um so there's there's a question about well what should the time scale be? Um and there's there's there's lots of interesting questions around that. It's basically it's very similar to the question of learning. Effectively it is the question of learning. you know, when should I update my belief about parameters? I'm sure Andrew can speak to that more than I can because as as you say, you've got more I have more experience with the uh uh continuous state space formalism and Andrew's a wizard with the discrete state space. So, I don't know if you've got anything to say on that specifically, Andrew, or or not. Sorry to put you on the spot. >> Of course I do. Uh and I was said that. >> Yeah, know I'm sitting here with the little hand raised. Um yeah, no I I think we make a quite good team on these things um with our respective backgrounds. Yeah. So and and and I do recommend um just on this point to maybe check out my previous um talk from Tuesday uh for anyone watching this but uh yeah so it's interesting. So so recall in chapter 3 when we were talking about the continuous stuff we had this introduction of different time scales and we see that different places. So it's exactly what Frasier said. it's that you know learning presumably happens at a slower time scale. Now of course we don't really like that word presumably because it introduces some kind of arbitrariness. Um here u what I think what Sanjieve is doing is uh by by giving this this trialbased learning because you're exactly right in the question like we could have uh learning happen at literally every single time step. So the agent would be learning just as fast as is as it's inferring. Um if we want to talk about realism, doesn't seem very likely that that would sort of happen in any kind of empirical scenario, but perhaps it would. And it's around there that we would want to start linking these things up with different kinds of empirical data fitting and look at actual, you know, if you're going for that. Of course, this textbook is for engineers. And so that that that that really opens uh you know the rule book is much smaller for for that. It's more about your your your scenario at hand. Uh no necessity of assuming that you're that you're superhuman agent that you're making that you need to be highly performative and you're not concerned with realism although you're interested in applying these kinds of biomimetic principles that we see in active inference. You know find some benefit to that uh of which there seem to be quite a few. But um so all this is to say the trial-based learning may be somewhat irrelevant to those who are studying engineering. Um where it's coming from is that much of active inference as we've already seen many times is coming out of neuroscience including um not just um you know neuroiming studies but we're also just doing more general behavioral science behavioral studies which is why um we'll see references in much of the literature at times to like these sorts of uh T-maze examples where mice are are in a little maze and they have to figure out uh is the is the reward the cheese on the left arm or the right arm of the maze. Um, you know, and we use that to to demonstrate um this kind of information seeking and information gain whenever we um, you know, code a a mouse who who actually has a desire to learn, so to speak, such that it um, you know, won't try and jump immediately to reward and potentially risk a shock because it went the wrong way. Um, instead it will check out some kind of informative cue. uh and that that that will give it evidence of where the cheese is and then it will go for the cheese now that it has a more certain um and empirically valid uh inference. Um >> I think we'll actually implement the team's uh this would be an excellent one. Yeah. To do. Yeah. >> Yeah. Exactly. So that the trial-based learning stuff really I think comes out of these behavioral studies where we make these highly structured tasks that will presumably have multiple time steps but a limited number of them. Like trial one will have three three time steps total. Trial two will have three time steps total. And so, um, this is very useful in these empirical circumstances where you're also working with humans where you could use a POMPP to model, you know, um, a person who's doing some kind of, you know, guessing game task or a multi-arm bandit task where they're trying to choose between different slot machines. Um, I've done work where you have a person who um has to pick a a color of an unseen card where the only evidence they get is from another person who gives them some kind of advice and then it leads into all these interesting things because we can model like well how much do they trust that advisor to give them the right um evidence and and all the rest. So it gets a lot into trust but actually pathology and false inference as well in cases where people are very you know distrustful or or or paranoid or potentially even hallucinate in different kinds of ways. Um so yeah the the the trial is a very structured task that might play out in multiple stages where each one of those is aligned with particular time step. Um and and so at the end of each trial, you could enforce a kind of learning rule on your model. And then that way that learning is still following in principle a slower time scale despite the the fastness of of the actual within trial time steps, you know, like, you know, time step one, you start at the starting point. Time step two, you can go to a queue or you can go left or right. Time step three, you can go left or right. You know, it's it's it's interesting because here with the T-mas, it's like, well, there's always three time steps. So, if the mouse goes to the queue, it has enough time to do that and then go to reward with more certainty. But if the mouse decides to skip the queue and goes straight to the reward, it gets an extra time step with the reward, right? Can go right to the cheese and gets to stay there. So, it's kind of getting more rewards, so to speak. Um, you could have a mouse who's very baze optimal in the sense of the first couple trials it's new to this environment, hasn't learned anything yet. And so maybe for the first couple trials it starts going to the queue. It learns more information about where the reward could be and starts going to the reward. And eventually through learning that mouse might become so confident about where the reward is that in the later trials it will skip the queue so that it can get those that extra time step with with the cheese. Um >> it's got a very precise belief about where the cheese is. So it doesn't need that epistemic Q. Yeah, >> precisely. And then the only downside of course is that we as the expend experimenters you know I'm not very much in the business of torturing mice but what you can do is that you can change the context where say for the first 10 trials the cheese is always on the left in which case it would be great for the mouse to learn that the cheese is on the left cuz then it could start soon um you know spending extra time with the cheese because it it learned it starts skipping the queue goes to the left but once after those first 10 trials you could change what we call the context of the experiment such that now the cheese is always on the right. Um, >> of surely the mouse will >> know what happens. >> Yeah. What what happened, you know, and and so the mouse suddenly uh gets, you know, it it doesn't get any cheese at all. Um, you know, if if you're doing the kind of experiment where the mouse gets some kind of aversive stimuli when it goes the wrong way, then it's it's going to have a bad time. Uh some people have used a small shock or maybe they'll use some kind of olfactory scent that the mouse really doesn't like and it gets stuck there till the end of the trial or something. Um so yeah, it it brings up all these experimental uh and behavioral questions about learning, learning rates, um the importance of learning, but also the issue of unlearning of needing to unlearn a previous set of experiences whenever your environmental context changes. So even though the T-maze looks like an incredibly simple example, it actually brings up many questions in different ways of of you know how do we construct a simulation? Does the simulation change context over time? What is it? What kind of phenomena are we trying to model here with respect to information seeking, learning, uh reward seeking, uh change uh uh being able to unlearn uh and and relearn in volatile environments and all the rest. And then of course because we have these highly expressive POMDP models we can look at the inside of the model at every time step just as Sanjie does whenever he presents us with all of these illustrations that show well here were the hidden state beliefs you know at each time step uh with with some dot that denotes which trial in which it was occurring and things like that. So yeah, it's it could become very rich and then a lot of these same principles are very applicable to behavioral science. And then to to finally finish my long-winded piece, it's just that that said, it does stick out as a bit of a slightly strange thing in this chapter because this isn't ultimately for engineers. And so um all this all this behavioral science background that's informing trial-based learning like will seem very irrelevant and and kind of understated. Um, you know, we don't I don't think he fully gives us that context in this in this chapter. You could you could do learning as quickly as you'd like. I think it's a smart move though on Sanjie's part. It's like, well, trial-based learning is one way we've actually done this to try and enforce a slower time scale at which learning happens. Um, y >> you know, instead of just saying arbitrarily, oh, we'll just do it every two time steps instead of one. So that way it's slower, but we don't have any principled reason for for doing so. Right. It's an important point. I I um I let you go there because because it is it is an important point like it's not just sort of god-given when to update and so on and things like that. But um and there's a long history of this. So >> the last thing I know we're a little bit over time. I'd like to spend just maybe five minutes and then we'll pick it up uh in the next session. The really exciting So we've seen okay this this story of uh learning uh you know updating our our dish uh prior beliefs about model parameters. this is how we can learn model parameters and that's excellent. But if you're very much like what Andrew was just talking about if you're a little agent in an environment maybe a mouse in the teammates really what would be excellent is if you can seek out states of affairs that are going to be informative about the parameters of your model. That's a bit more of a sort of subtle question than just learning what the the model parameters are on the basis of sensory evidence. And that's exactly what we have in this next next little section here. So the idea is on on at the end of uh all the examples we just looked at the if I go back up here. So we have our model. Excellent. We can update the parameters. The thing is we need to select the idea is we'd like to be able to select actions that actually deliberately reduce our uncertainty about model parameters. So how do we do that? And effectively what we're going to have to do is we're going to have to augment the expected free energy. So we've seen the risk and ambiguity terms of the expected free energy. We can add in another term called novelty. I'll have to talk about that next time. And the idea is that this is going to be a way for us to to seek out uh states of affairs that are going to give us a lot of information about those parameters effectively. And then what we can do is we can take actions that are specifically designed to give us this this ability to reduce uncertainty about model parameters. I think that's where we'll have to start next time. I will just say that I do have example 10.4 in the in the examples here. So hopefully this is the screen there. Scrolling scrolling. Here we go. So we've got risk ambiguity and now we have parameter novelty as well. I think we'll start with this next time and then we'll go into 10.2 which is habit learning precision and flexible policy optimization where we can learn um much more than what we've been able to learn before. So, I'll put this notebook up on the site immediately after this. Come back over here. If there are any last questions, now would be a good time to to shoot them out. But, um, I think that's hopefully that was helpful. We'll we'll see you guys next time. So, I'll end the recording here.