Fundamentals of Active Inference (Chapter 8, Session 40) August 28, 2026

All right. Hello everyone. So it is August 28th, 2026. We're here in the first session uh or my first session on chapter nine of fundamentals on active inference. So this is we've completed um uh the continuous state space treatment of active inference. Now so that was chapters six through eight. Um and that is a really big thematic block in what active inference is at all. um and the kinds of models that one typically sees in the literature and and and can implement uh with active inference. So that was all sort of continuous uh state space. We had Gaussian distributions and things like that. We're now moving into something completely different and now time for something completely different uh in in chapters 9 and 10 where we're going to be looking at discrete state space active inference. Um and you know basically the story that we're going to tell is going to be the same kind of story. We're going to have the same machinery of prediction. We're going to have the same machinery of variational free energy and and such like. But we're now going to be doing it in a slightly different setup. Basically, we're going to be doing it where instead of smooth uh continuous distributions, we have um uh discrete uh things now. And we're going to we're going to cast the problem of inference in terms of something called a partially observable Markov decision process. So that's that's what we're we're going to be doing um in a very uh sort of tongue-in-cheek manner. The the highlevel difference between sort of what we're doing now and um what we have been doing is that this is we're we're going to be moving into basically things that look like arcade games like you know Space Invaders that kind of thing where you can sort of move up, left, down, right in a very sort of discrete manner. um as opposed to you know a neuronal activation or a velocity or something like this which is a very smooth continuous thing. So that is the very very high level uh preface. The thing is chapter 9 is quite large and it introduces a few things. It only really introduces one new thing in terms of something that's new to active inference per se. Um and that would be this idea of planning. Effectively, that's going to be the big takeaway from chapter nine. It's going to stay with us with chapter 10 as well. But planning is really a sort of different beast to stuff that we have seen so far. We haven't really seen anything that does planning in any sense. But with the discrete state space formulation, we're going to be able to formulate what it means to be doing planning. Um, and we're going to see the first few examples of that. So, we've got subsections 9.1 to 9.7 where 9.7 is review. Uh myself and Daniel and um Andrew, we were sort of thinking chapter 9 is very important. It's quite large. There's a lot of material. It might be best to have an extra week on chapter 9. So, what we're thinking is, you know, this is up to you guys as well. First week will be 9.1 9.2 and then we'll go 9.3 maybe to 9.4 in the second week and then we'll finish it up with a third week. Um I think that's probably good. There's a lot of good examples in here as well which we'd like to see in code. Um, and given its importance, uh, it probably warrants that sort of treatment. Probably the same sort of thing we'll do for chapter 10. Uh, or at least we're thinking about doing that. So, we'd have three weeks in chapter 10 as well just because again very much as we've seen before, we introduce something, we introduce a new idea and then we talk about how we can sort of generalize that idea. So, we go from basian inference to, you know, variational inference. We go from inference only to inference and learning. we constantly introduce something and then generalize it. Chapter 10 does that for chapter nine. It it generalizes the the story that's being told in chapter 9. So that's kind of the plan uh at least as far as far as I think it would be wise to embark upon. So we're hopefully going to try and cover chapter 9 uh 9.1 and 9.2 today unless anyone has any objections to that. Uh let me begin sharing my screen here. Let's just pause. It's a bit easier. So I'll just mosy on across to hopefully everyone can see this now. This is the page the the coda page for chapter 9. Um I have the sort of general overview here in terms of what the objectives are for what we want to try and understand in chapter 9. The nutshell uh key differences between what we've seen so far from chapter 6 to 8. This is very important. Okay. There are actually quite a few differences now um between what we've seen and what's happening now in chapter 9. So this is this is definitely a watershed moment in the book I would say. So you got you've got really you know six to eight they're one little thematic core about active inference specifically and then nine and 10 are a different but related thematic core about what active inference is. So and those those two uh groups of chapters they form part two of the book. So you can go have a look through here. I tried to be as brief as possible. Didn't always work out. I've got the map here in terms of all of the subsections. I've only put up my latte notes or PDF notes for subsections 9.1 and 9.2 just now. I do have the notes I have created notes for the whole chapter, but I want to just review um all of the other material before I put it up. Um so that should be that that won't take very long. Basically, it's just there given that there's a lot of sort of notation and machinery so that people who don't have the book can follow along. they can see okay this is the expression we're talking about with respect to you know where we are and so on. So we've got the the overview for 9.1 9.2 and the general overview up for the whole chapter. So you've got the uh this this is this is a 30,000 foot overview of everything in the entire chapter basically. So, I'd recommend having a look at that this particular document if you don't have the book or even if you do. What I tried to do in these notes is a they're trying to they they aim to be things that people can use to follow along in the maths if they don't have the book, but also to try and maybe add some supplementary explanations and things where maybe I think the book glossed over something a little bit quicker than it should have or you know it's always hard to to know how how much to stand on any given topic at any time. So I try and these aim to be complimementaryary is is what I'm saying not just a transcription. All right. So that's that's where we are. So again, yeah, 9.1 9.2 today. I'll just pause briefly before I launch into my soap box. Do we have any questions thus far? Well, has anyone sort of read chapter 9 and uh do they have questions thus far? I'd be interested. If not, I will I will begin expositing the nature of uh discrete stuff. Going once, going twice. All righty. Well, let's come back. Whole thing share. All right. People can should see my tab now. Um so, the only thing that's missing, I would say, uh we haven't got it in there just yet, are the figures for chapter 9. I do have them. They're over here. So maybe if I get rid of myself, we can have a look at the figures. So hopefully everyone is seeing the outline now for chapter 9. If they're not, please make a lot of noise. I can't really see that just now. Um, so this is the outline. Basically, we're going to do 9.1 and 9.2 today. So we're introducing these things called partially observable marov decision processes, poomdps. These are ubiquitous in the literature. Um, and they predate active inference. There's this they're a very general formulation of what it is to be solving a decision problem basically. Uh, so we're going to we'll have a look at that. Have a look at what that means and how we're setting up things. Uh, and then we're going to go to 9.2 and look at a specific kind of partially observable markoff decision process uh, called a hidden markoff model. So where we we do uh prediction about things that are going to be happening in the future as well as post- piction that's going to set us up today. All right. Now there is a nice table which I don't think I have in the figures. Table 9.1 effectively it's a summary of types of Markov models. So let's just back up for a second and think about what we're even talking about with respect to this whole markoff thing. Uh this this might be review for some people, but um basically the idea is the we're we're now going to assume that we have um hidden states that that don't that are what's called discreet basically. So they I can either be in a hidden state, one particular kind of hidden state or not. um in the sense that you know I might be in the kitchen or the bathroom but I can't be in there both at the same time and there's no real notion about how one kind of blends into the other or anything like that. Obviously for things like we've seen before with we've been talking about the position of an agent maybe in a water column or the velocity of an agent or the temperature of an agent. These are all continuous quantities. They're like a slider. You know what we're going to be talking about now is much more the situation where you can choose between like different buttons. So you can either press a button and it's on or not. There's no notion of uh you know halfway or anything like that. So that's uh just high level intuition. But the the idea I don't have a I don't think I have reproduced table 9.1 which would have been nice to do. The idea is that this idea of solving a problem of decision making um the the there's all kinds of ways that you could approach this. The most common way is to assume that your problem is going to have a specific kind of property. Uh and that is the markoff problem which effectively says that the state of my system now only depends upon the immediately previous state of my system or maybe you know the five previous time steps um or something like that. it doesn't depend on the the entire history of previous states. Uh so there's lots of very nice ways of visualizing this. Um which we we do have but I'm sorry we don't have the uh the the the well we do have figures but not table 9.1. So we're we're in a we're going to assume that the state of the world only depends upon the immediately previous state. I think Andrew is Andrew slides are very nice here. Um we don't show that just yet. Well, we do in in figure 9.3. Maybe that would be good to go through. But there's there's different kinds of Markov uh processes. Uh there's there's many many different kinds. Um the the fundamental differences between sort of what's controllable, the thing the state transitions that you can affect in the world and what's not controllable. Basically, that's a very big uh decision. That's kind of like access number one. Access number two is observability. So can I observe all of the states in the world at any given time? And obviously chess, right? This is the example that he gives in the in the book. You can just immediately see the board. All right? And you can see what all of the where all the pieces are at any given time. The whole state of the thing is totally unambiguous. Um I can just see stuff. But of course, if I'm playing battleship, uh I can't see that uh game, right? Everyone knows what battleship is. The whole point is that again we're in a sort of discrete setup. You know, my my position of my ship can only be here as opposed to there. But I don't have perfect information. I can't just view the board with respect to where my enemy is basically. So this is what's called partially observable. I can only observe my movements and I can't observe some other part of the state space. So that those two dimensions, this idea of what's controllable, what's not, and what's observable and what's not, they kind of sweep out the kinds of markoff decision processes at issue. What we're going to, the whole point about chapter 9 is that we want to work in the realm of partially observable Markoff decision processes. So these are things that are not like chess. They're more like battleship in the sense that you can't see the whole state space. And that should be familiar to us by now in the sense that we know that we can't just directly observe uh the states of the world. We have to infer them on the basis of sensory evidence. Um but that's uh in addition to that what we can do is we know that we can we can uh emit actions and we can change the way that states transition in the generative process. So those two things together they give you a PMDP partially observable markoff decision process. Um, we're going to start though with something that is not a POM DP. It's a bit simpler. It's just a a something called a hidden markoff model where it's partially observable. You can't see all of the states in the world, but it's not controllable. You've just got some states and they transition according to some law, right? So, what are we going to be able to do with a hidden markov model? Well, we're going to be able to do perception basically. That's the fundamental starting point. And then once we understand that we can build up a notion about how to do how to control the the generative process. So come here. All right. So let's let's even let's sort of back up a bit and think about how we're going to start formulating this this problem or this new setup. Always always always we're always starting from our we need to be able to specify a general model over observations and states. Okay. And this is our likelihood of observations given hidden states and a prior unhidden states. That's the same. The thing is now the shape the actual things that these different um factors constitute are now going to be different to what we have been working with. Instead of being probability density functions, Gaussian bell curves, they're now going to be categorical distributions. The idea and and we have a name for distributing for distinguishing between these two things. Probability mass functions those are nice smooth um distributions like the like the Gaussian probability uh density fun sorry probability density function is nice and smooth. Probability mass function is this sort of categorical thing here where there's a bunch of categories and there is a probability for each one basically. So it's like a a bar graph that sort of thing. Um the notation can be a little bit uh wacky initially. I would very strongly recommend looking at um appendix C.2.2 on on the notation how we sort of denote what these things are these categorical distributions. But basically let's think about the two things we need to specify. We need to specify the observation likelihood and the prior on the states. That's always what we have to do. Um but we're going to assume now that these are given by probability mass functions categorical distributions on the state space and on the observation likelihood. So what does that mean? Effectively it means that the we we'll give our our state prior a name uh basically and then well we're going to call it this is just how things have arisen in the literature. We talk about having vectors and matrices for specific parts of our model. So the the state prior is often given the the it's we we call it you know the d vector the the prior on states and that literally looks like this. If I go to the equation it issue uh equations equations equations it's actually something that can be shown fairly easily. So equation 9.2 there we go. So I need a state prior. What's it going to be? Let's assume that we're uh well let's introduce the problem. We we imagine that we're trying to predict what the weather is on the basis of sensory evidence. Now the weather there it's not the greatest u uh example because the weather is sort of inherently continuous thing doesn't matter. Let's imagine there's only four possible states. It can either be raining, it's either cloudy, sunny or snow, right? That's the weather. Those are the hidden states. Well, I have a a prior belief about those hidden states and it's a vector. It's just a list of numbers where the entries in this vector are probabilities. So these are numbers between zero and one. The only requirement that I have is that um these numbers are going to have to sum to one basically. So that's this is the specification of our hidden state prior in in in the categorical sort of language. It's just a vector of of probabilities. Um, now the thing is we can get sensory observations. So I'll show you actually figure 9.2 that shows much clearer. This one here. Here we go. People should see this is this is figure 9.2. If you don't see that, please uh tell me. So imagine there's only four hidden states. Rainy, cloudy, sunny, and snowy. Um the thing is we're going to we're going to say well look I've got some prior belief about what the hidden states are. Um that's my d vector. Okay so that's just a column of numbers that's have to sum to one and that whole thing is what we're calling a categorical distribution. So it looks like this. It looks like a histogram. This is a histogram here or or a bar plot of um probabilities. So rainy has a probability of.3 or something like that. cloudy a little bit less than 0.2. Um this is a categorical distribution basically. So you notice it's not uh a lovely smooth bell curve or or Gaussian distribution anymore. Um that's what we're working with. Thing is though uh we need our observation likelihood because we don't directly observe hidden states. We observe observations kind of tologically. Um there's all kinds of things you could observe. Let's imagine that there's only three possible observations. So this is figure 9.2 by the way. And the only observations you can observe are I can observe water. So maybe there's water on the ground. Maybe it's hot, whatever hot means. And then maybe it's bright. Very good. Now the observation likelihood uh the thing that we've specified before in continuous state space that's now going to take the form of a matrix a table of numbers. So we've got hidden state prior categorical distribution on whatever the hidden hidden states are. Okay, some you know it's a row of numbers they have to sum to one. That's all I that's all that I I need. And I've got a c categorical distribution. Now the hidden states sorry the observation likelihood now says well for each observation what is the the hidden states uh what is the likelihood of being in a particular hidden state given an observation. So that's a table. Basically you take um well you know the rows in this this matrix they correspond to observations. So this is this is you know I'm I'm observing watery stuff on the ground. This is its hot this is its uh bright and then hidden states are columns basically. So that specifies the prob the observation likelihood that I need you know of observations given hidden states. So the idea is to work out the the probability that it is hot given that it's cloudy. What I would do is I would come to my observation likelihood and I would say all right well hot is the middle thing here. uh what I need to do is oh well sorry given that it's cloudy that is uh one two you know this this hidden state the probability that it's hot given cloudy cloudy is this little cell in here basically so that table that gives me my my observation likelihood so we have a vector and we have a matrix and using linear algebra we can now do uh all of the sort of relevant things that we need to do to do basian inference and we are going to do exact basian inference by the way I should I should mention that in in um 9.1 and 9.2 we're going to assume that we're able to do fully accurate or fully exact basian inference and then 9.3 we bring in the variational free energy again so this is kind of like chapter three or chapter 2 redux so equation 9.4 four. That's a nice representation. I'm I'm I'm deliberately being slow, you know, I'm deliberately stressing these things over and over again because they're going to be with us for quite a while now. So, I apologize if it's if it sort of feels like I'm drawing blood from a stone. Uh 9.40. I I don't think uh coder likes to do things likes to format things very well. Oh, good. we have the equation uh the algorithms that effectively this equation 9.40 on on 217 is just what the mathematical description of what I explained to you. Oh, here we go. Here we go. 24. No, not quite. Uh 25. No, not quite. Well, this this is helpful anyway. This is not quite equation 9.4. This is 9.25. The thing is observations. What does it mean to receive an observation now in this cell? Uh well, the idea is that we we're gonna this is this I'm jumping ahead a little bit because really it's equation 9.8 that specifies that. Maybe if I go there. First of all, the idea is that we're going we're going to represent an observation as the vector of all possible observations with a one for the observation you actually get and zero for everything else. That's called a one hot encoded vector. 9.8 9.8 Going to have to go through and change the way that Koda represents these things. Maybe it's in my notes a little bit quicker. 1. So I want to equation 9.8. eight. Uh, and I, you know, I go through here in painful detail and explain things a little bit more um in greater detail in the actual notes. Well, this is actually 9.4. This is what I wanted to show originally. Uh, yes. So, this is the observation likelihood. This is from um in the book. So, we have going down the rows, those are observations. water, hot, cloudy, um or bright, sorry. And then across the columns we have the hidden states. So rainy, what is it? Cloudy, sunny, snowy. Rainy, cloudy, sunny, snowy, and then water, hot, bright. I should really have um annotated them, but I have them here. Okay. And then, you know, indexing into this matrix. So the the first row and the first column that corresponds to the probability of the first row given the well the index of the row given the index at the column. So for one one well depends on zero we're uh one based indexing here which is a little bit different to what we've been doing. So the probability that it's you you get get an observation of water given that it's rainy that's 08. That's the probability there. Okay. And of course the columns have to sum to one because they are probabilities of hidden states. They're categorical distributions each one of them. So they have to sum to one. That's that's an important normalization requirement. Okay. So then our model looks like uh looks like this. 9.5 Basically, we're just going through and you know seeing um how we're specifying things now that in terms of categorical distributions 9.5. There we go. >> Frasier, I just want to Okay, good. Uh yeah, I just want to quickly say I think with that table all the equations are there. I think they might be sorted a particular way on your screen. They sort the way they should be on mine. So maybe it's like a browser cookie thing, however you like. But yeah, >> I'll go in. Yeah, they tend to they tend to format very strangely. So that's might be a browser thing. So here's the specification of our model. Observation likelihood and state prior and they're both categorical uh distributions. Now that's really the only difference. So now a nice way to visualize this because code is well math is often a bit hard to parse is in terms of um uh the uh the Beijing networks. So figure 9.3 is here somewhere. There we go. So this is a a graphical representation. It's not quite like this in the book um of the same information. So uh we now have a vector that parameterizes the unknown hidden states state prior gives rise to observations um and the act well the observation likelihood matrix A tells us the probability of an observation given hidden state. So this this top variant is much more common in terms of that that is the sort of common syntactic representation of a basian network. Sometimes in the literature you you'll see it like this here where you've got the matrix sort of in between the two variables at issue. So you know there's there's no previous hidden state for the state prior. So you've just got that going straight into the hidden state. Um and I you know just for so everyone's on the same page and remembers circles are things that we're uncertain about and circles that are shaded in are things that we observe. So that's that. So what do we have to do? we have to invert the generative model basically as always. That's the whole that's the whole game of Beijian inference. Um and in order to do that we have to be able to calculate the um the normalizing probability of observations. So we're going to assume we can do that uh here and that's an assumption we're going to relax when we go to 9.3. So that's 9.7. It's just bases rule basically. Don't necessarily want to have to spend time on that. So what he does and this this is important to um to grock this particular thing. Let's think about how we actually get um an observation. So he says the uh since the vector must one this means that at most one element of the vector must be one and the other elements must all be zero. So if I actually get an observation in, I have to say, well, one of I'm actually observing something. So the probability of me observing that thing is one and everything else is zero. So that's an example of that is is 9.8. I'm sorry, I don't uh it scramles up. Do I have it here? No, probably not. I don't know why it does that for me. So So then on the basis of that observation, you want to update your belief about hidden states effectively. Um, how do you do that? That's equation 9.9, which is probably all the way down here. I do want to spend time to to find that. I did have it up here before. Seven 9.9. Aha, there we go. So, let's say I actually get an observation in basically. Uh so that's fixed. I I do observe that it's watery. Let's say so this it's either watery, hot or bright. Um well what is what what's the likelihood of this observation? Basically I take my observation likelihood matrix transpose and I times it or dotproduct it with the observation. So what does that look like? Looks like 9.10. This is a visual representation. So, if I go back here, which we don't seem to have, that's very sad. Okay. Well, oh no, here we go. Perfect. Didn't mind didn't know why it come up. So, observation likelihood matrix there are, you know, as long as things are normalized columnwise, it's fine. They can be sort of any numbers. But we actually observe that um it's hot, let's say. So, that's this observation here. this likelihood then corresponds to just this matrix multiplication of that vector with my um my likelihood matrix. So a transpose takes a and it turns columns of a into rows of a and rows of a into columns of a. So what this is really doing is it's just picking out in this case the second column this one here. So if you know if if I got in an observation of watery I would get this row there rows in a transpose and columns in a. So this is the observation likelihood of um all of the the the hidden states given my actual observation. This is not uh a probability. These don't necessarily sum to one. um which is why we have to introduce the normalizing evidence the evidence of of an observation uh and that's finally then going to allow us to do Beijian inference basically. So you know for those who don't know any linear algebra I'm skipping over a lot of material here of necessity do have a look at the notes they explain things a little bit better. So we've got our likelihood of observations arpose given well you know multiply by the observation we get in we then multiply this with the state prior and that's going to this this tells us now well I'll just read from this you know we can now follow bases theorem equation 9.7 we all know basis theorem by now and we can write the posterior of hidden states as proportional to this is just you know likelihood times prior but of course if we really want the full distribution the the full posterior we need to normalize this we need to divide by the evidence so that is just this is bases rule in matrix form notation we're sort of abusing notation a little bit here but suffices for our purposes so the evidence of any given observation is the marginal so I sum across all possible hidden states for my observation likelihood and that gives me the probability of observing the thing that I did observe at And then that ratio that's my variational well that's my exact posterior. Excellent. That is effectively how you do measure inference in the realm of categorical probabilities. The last thing to note is we're going to sort of assume well not going to assume we're going to deliberately use a particular representation uh for of of all the probabilities at issue. We've seen this before. We're going to basically for numerical reasons uh use logarithmic transforms of the stuff that we're talking about just because when it comes to computation things are a little bit nice with not running into overflow and underflow issues. So um this is this is effectively just a sort of pragmatic thing that we're going to do. We're going to represent the same thing that we had before just with logs and exponentials. So this is the fundamental object bases rule. But we're going to do this. If you take the the logarithm of something times something else, you get the log of the first thing plus the log of the second thing. Look up logs on your own time. All this is to say is that this is exactly equivalent to what I just showed you. 9.13 totally equivalent to 9.12. But it is going to help us with things. So there's an example that to to to help you know actually do this. Now example 9.1 discrete hidden state inference. Um that's a nice one to go through. So I'll just really briefly go through it. Uh so basically you start out with a categorical prior over hidden states. That's 9.14. This one here. So, you know, I think, okay, well, I've got a prior belief about the hidden state being rainy versus cloudy versus sunny versus snowy. And in this case, I'm equally I have a uniform prior. I don't believe that any hidden state is any more likely than any other hidden state. So, you know, um that's a that's a typical setup that you know a lot of people sort of start with. Uh let's go to figure 9.4 now. Show some visual stuff. Here we go. Hopefully you guys can see this. So this is figure 9.4. This is my beliefs about the hidden states after observing data. So these are my my posterior beliefs effectively on what the hidden states are. And let's say these are annotated. I observe in the first instance um what is it here? Hot. I can't remember. Yes, hot. So I observe an instance of it being hot. I can then update my belief that I had prior uh you know about hidden states given sensory evidence using my um observation likelihood mapping. So now I believe that it's more cloud I believe that cloudy is the most probable out of all of the the potential options. Um but I still have probability distributed elsewhere. So likewise if I observed instead that things were uh watery well my posterior now is that ah I think rainy is quite likely indeed most likely out of all the other options. So you can see depending on the observation you get in and the prior that you have and the observation likelihood that you have you'll get different posterior estimates and we saw that in in chapter two in the continuous sort of spec space as well. So this is kind of the the discrete analog. Uh so he shows there's a nice um what is it here? Equation 9.15. Move this to the side. There we go. So this is this is just literally an example a worked example of of the the whole inferial pipeline for the particular likelihood matrix the particular observation. In this case, it's the observation of hot uh and then the particular prior. So I have my uh log prior. It's totally uniform. This gives me my um my my uh posterior. And this this sigma here, this is just a a soft max function. I don't want to have to go into that in too much detail. Very nice. Okay. So that's really just the story about how we do exact Beijian inference in the categorical setting. So there's nothing really specific to active inference in this at all uh yet. But this language of um observation vectors uh well prior belief vectors that that are categorical distributions and likelihood matrices. This is going to stay with us now for for a long time. So we need to be very comfortable with all of this. 9.2 two then adds in something interesting which is how do we do inference in a dynamic environment. So obviously the first two chapters of the book everything was static but we want to look at you know actually changing environments now. So that's 9.2. So really this is going to introduce the idea that hidden states they can transition across time basically. So we have we have the observation likelihood mapping you know state prior conditions a belief about states states give rise to observations by means of the observation likelihood mapping that a matrix. Uh but now this whole picture is going to be at a change in time basically. So that introduces another matrix that's typically in the literature called the B matrix. So we have the A matrix specifying the observation likelihood probability of observations given hidden states. The B matrix this encodes the the mark of assumption that we have about how hidden states transition across time namely that hidden states now only depend on hidden states immediately previous to now and not on the entire history of hidden states. So you know we're going to go from hidden state now using B that's going to give us hidden state at the next time step and so on basically and those two things are going to be able to give us the full setup of of what we need for dynamic hidden states. So Bas's rule looks very similar. I'll just show it because it is useful. That's 9.16. Hope this isn't too dry but there is a bit of setup initially 156 perfect there we go so now things now hidden states can transition across time so my uh posterior at the time step t on hidden states given observations is the likelihood uh mapping a times now the the sort of prior on hidden states is this um time dependent thing we have the hidden states oh this should be this should be this is wrong this should be the hidden states at time t + one given the current state just a minor error so you know b tells you what's the probability of being in the next state given the current state basically so minor minor era there and then divided by the sensory evidence of course in general for most interesting problems we cannot compute the sensory evidence because we have to marginalize the uh the the likelihood across all hidden states usually that's impossible So we're going to see in the next section when we get to variational free energy how to do this active inference style basically VF style. So >> you show anything. I don't I I don't see anything that you might be showing. >> Oh dear. Okay. Let me uh I'll stop sharing. Thank you Kus. And then I'll reshare. I've got a couple of extra screens up today. I will say it was working fine for me. I could see what you were sharing the equations. Yeah. >> Can we see um Bas's rule now? >> 9.16. >> Uh I'm fine now. Thanks. >> Okay. All right. Yeah. So there is an example another example where we go through looking at transitions now. So that's 9.5. I don't want to dwell on that for too long. So that is this one. Uh it's not named but uh where is it? Here perfect B matrix. So now of course the way they can transition across time um this this is a nice representation of the B matrix. So the idea is that rows uh going down they tell you these are states at the next time step. Um well sorry rows are are states of the current time step and the probability of being in the next time step. You take the the the slice now well sorry you know this slice now and then you condition it on uh the the the particular row you're in right and that would that would give you the the hidden state of issue. So this is this is always a square matrix d by d in the sense we have hidden states by hidden states basically. So, and that is a categorical distribution. It's just like the observation matrix, although it's a different thing. So, the Beijian uh where is it? The um here we go. Is that right? Dynamic. Perfect. So, the actual um the vision network at issue is is exactly the same thing we saw before except now we have transitions between hidden states and you know, we've got a likelihood observation at every time step as well. One thing that is going to come up is this notion of indexing in time. Um I don't want to necessarily get to that right now, although I will very shortly. So that brings us to there is there's um you know inference in a dynamic environment 9.2.1. I'm going to skip that just now because we've actually seen that before. 9.2.3 2.3 or 23 rather. So the actual generative process is changing in time. How do we represent this? Well, the generative process says we got hidden state transitions and we have an observation likelihood mapping. The observation likelihood mapping we went through before. That's just a transpose times the hidden state. Uh yes. And the state transition function that just takes hidden states at t minus one and gives us the hidden state now or typically you say you know the hidden state now and gives us the next time step just depends on how you're indexing things. So this is correct here that's how the the environment is actually changing in in time the model. So you add the 9.3 filtering with a hidden markov model. So the idea is we could what is the filtering problem? We're going to update beliefs about hidden states on the basis of sensory evidence that's coming in right now and that's it. We're not going to be doing doing any sort of prediction or anything like that about hidden states in the future. Um there's a nice example there 9.2.4 rather. So [snorts] I'm going to have the same observation likelihood different observations. Um well sorry this this is this is the the state transition function. So we've got the probability that it's rainy in the last time step, that it's cloudy in the last time step, sunny in the last time step, and then as we go down the the the rows, we've got the probability that it's rainy now and cloudy now and so on. The probability that it is cloudy now given that it was uh let's say cloudy in the last time step would be 6. If you sort of you know go down the uh particular row well sorry the particular uh how do I say this you go to the particular uh column and then you go down to the row that you want and that tells you the probability addition is of transition right so nice way to represent how these things work is which is much nicer to look at uh this network diagram PR or dynamic hidden markup model. So this is a representation of what I just showed you. This is a a hidden markup model. So we've got, you know, partial observability. We can't observe everything. We're not yet in a position to intervene on this to change how states are transitioning. States are just kind of transitioning of their own accord and they're they're doing so within this framework of categorical discrete um distribution. So the idea is that you've got states given by circles and the [snorts] arrows tell you the probability of transitioning from that state to the state that it's connected to. So you know probability that I transition from the rainy state to the rainy state is 75 from rainy to cloudy 0.15 and so on. So that's a bit more of a nicer way to visualize this this sort of thing. 9.7 shows us the relationship between the model and the environment. Now, so that is do we have it? It's a bit of a funky one. This one here. Um, model and process. I don't know why we have two of them. Draw IO PDF. There we go. So, we've got time going from left to right. Basically, we have our generative model up top and the generative process down the bottom. So, you might imagine that, you know, basically this middle yellow layer. This is the this is in some sense the mark of blankets or the the separation between the the agent and the environment. So everything up up top is the agent. Everything down below is the environment. Just um notational really. So we've got what have we got in the model where we've got prior beliefs hidden states giving rise to hidden states. Hidden states give particular mapping from them gives rise to observations and this whole thing transitions in time or rather the hidden states transition in time. basically uh and of course the actual generative process does this as well um or we can specify it by means of categorical distributions and their transitions um but it doesn't have beliefs about stuff typically. So there's a nice example uh example 9.9 where this is the last bit from this section. I'd like to get it up. Possible [sighs and gasps] uh where you know you you actually sort of see this thing uh throughout time dynamic. What is it here? Results of example two of the first 40 time steps. Uh there we go. Perfect. So what are we looking at here? We're looking at the results of inference. uh using our sort of uh weather model across time. So the idea is that um gray states are hidden are real true hidden states of the world and these magenta or sometimes it shows up as red. These are the predicted states. So across time here. So you know in the first two time steps it was sunny and the model predicted that it was sunny. So you can see the model is actually predicting the true hidden states of the world very well except for in here. So, uh, it was snowy from about 12 to 16, but the model was predicting that it was rainy. And the reason it was doing that is because it assigned very similar likelihood to snowy and rainy given the same observation. So that that meant that the model had sort of scope for uh confusing those two things. So otherwise pretty good predictions though. So very nice. Okay. Yeah. And really figure 9.10 10. This is really just driving home the point that you you can you have beliefs about hidden states. Uh but because this this thing is a probability distribution, you do assign probability mass to hidden states that are not necessarily the most plausible. So the most plausible hidden state here is that it's rainy. But I I do distribute belief about other hidden states in terms of their credibility as well. So that's kind of why you can be wrong in some intuitive sense or why the model could be wrong here. All right, that is effectively the end. The only thing to mention is this is going to be really exciting when we get to planning. So 9.2.2 observations uh and beliefs and indexing. So, if I can get figure 9.11 up, that's the important one. I apologize for my slap dash nature of figure readiness today. This one, this is going to set us up for the next few sessions. I I'll end in the next two minutes here. But the idea is there's two problems basically at the given time step at the at the at the present. Let's say that you know 16 time steps have gone by and we're at the present time where we've got state transitions and observation likelihood. The thing is uh and I can I can do filtering. I can predict what the hidden state is on the basis of current sensory evidence and I can do that for all time steps. Excellent. That's that's fine. But prediction is me saying what do I think hidden states are going to be in the future. You know that's that's prediction. That's what we'd like to be able to do. you can solve harder, more difficult problems if you can predict as opposed to just being able to react to stuff. So the idea is that for some time horizon into the future, I'm going to be able to predict hidden states and also their observation likelihood. So you can see observations that are happening in the future haven't really happened yet. So I'm uncertain about them. They're not shaded in in gray. Okay, I I know what this is. I saw it. I observed it. I know what this is. I observed it. But in the future, observations are uncertain and I can have beliefs about how they they transition across time. So just a quick bit of bookkeeping, we're going to use lowercase t to refer to the time in the simulation. So now it's time step 16. We're going to use the Greek letter toao to refer to the index in the kind of window of future predictions. So there's a there's a a window of length in this case h. So you know from now t to well you know t plus one to h I'm going to have beliefs about future time steps basically and the whole simulation is going to last for some amount of time big t all right so I can do that you know right now I can make predictions about these three hidden states maybe even about the observations I'll get from those hidden states if let's let's say time progresses three steps now I actually observe well you know I go through those state transitions whatever they were they give rise to observations and I'm now at time 19. I can revise my beliefs about the predictions that I had at time 16 from time. And that process of kind of revising my beliefs is called postdiction. So, uh there's these two processes that that are going to become available to us and that are going to be constitutive of sophisticated by by that I mean powerful active inference agents that that's going to bring us to planning which we're not going to all get just yet. Um but this this bookkeeping is going to going to stay with us. There is another example which I think would be excellent to look at although we're essentially out of time. Figure 9.2 uh shows you this process. Basically shows you going through the motions of doing prediction and post-diction. Uh can I show it really quickly? Don't know if I can. Must be riveting watching you. Oh, there we go. Waiting for me to to show you things. So this this is an excellent demonstration of this process of prediction and post-diction. So you start out at the current time, you observe stuff. The future is uncertain. You transition. You you get an observation at the next time step. That's great. The future is still uncertain, the remaining future, but I can have better and better beliefs about that future because I'm sort of closer and closer to it in some sense. And I can post dict. So that is the end of 9.2. 2 and 9.3 then says all right excellent how do we actually do this really given that we can't do um the you know general Beijian inference you you know exact Beijian inference how do we do this with variational free energy so that's 9.3 onwards and that's where we'll pick up next week that's a bit more exciting that's that sort of situation so any questions at all I'd be happy to take them I know that we're almost out of time I see the chat's been very active let me just have a have a gaze through there notation is very odd. Yeah, the notation we're effectively one thing I didn't explain was um these kind of hats. So you know we've got s hat star obviously we've got star for you know real well you generative process things we've unstarred things for for model related things hat is another piece of notation. Andrew did you want to say something very quickly? Yeah, thanks. Uh yeah, no, just in reference to the the notation, uh it took me a second to catch on that uh Sanjie's been using just the the bold s uh to refer to um posteriors and in various equations there. Um I think it's useful for those who are reading literature elsewhere uh and and trying to get familiar with sort of the the the nomenclature and ontology that's used in these sorts of things. It's very frequently that uh in active inference we'll see uh like a Q of S which we're not quite there yet. That that's whenever we do the kind of free energy uh and variational inference techniques that we we already saw in the continuous case. Um but now here we're we're using properties of logs and the like. We're effectively going to make a a variational posterior Q of S. Um, nonetheless, this whole seeing S by itself, the the this is a bit more of a speculative thing on my part, but my guess is that it maybe aligns with um the graphical representations that we're seeing like of the basian networks and the like where whenever we see our nodes like our our variables, we're just seeing like an S in place uh for states just as we see O representing observations. It might be that it's an attempt to maintain some kind of coherency with the uh with the graphical representation. I could see it working that way because then we can think of s as like one variable that itself gets inferred or updated over time and then can immediately directly reference it when we look at our our our basian network or graph. Um in any case, that's what's most useful to me. So, I'd almost suggest viewing it that way because otherwise it it I think it'll be easy to get slipped up on on what the S means versus seeing it as Q of S or P of S condition on O and all the rest elsewhere. >> Yeah, I he uses bold to denote either a vector or a matrix. So maybe that's that's one other area potential confusion. the the mathematical appendex appendix C does does go over all of that for for revision son Zen did you want to uh is there a question there >> uh I just put two pictures uh in the chat it's on the fundamentals page I noticed that the written answer seems to be incomplete it cuts off after how the hidden I'm not sure if that's just how it appears on my side or it's something was missing. Would you mind taking a quick look? Very appreciate it. >> No, no problem. Thank you. I um unfortunately it's not not loading on in in the chat, but what page was this on? >> Uh the the fundamentals of active reference textbook group page. Uh wait wait wait let me see questions questions page and then chapter seven. Yeah. >> Oh uh questions on the um on the website on the on the coder. Is that right? >> Yeah. Yeah. >> Ah okay. Uh I'm I'm going there now. Let me go down here. So yeah know there's there's two questions there. So continuous versus discrete state spaces. You mentioned non-counterfactual. >> No, no, no, no, no, no. The first one. Active inference is often applied to physical. Yeah. >> Yes. Yes. Yes. So, okay. This is this is a question from chapter seven. Uh just so action, let me let me share my screen briefly. Uh just >> it cuts off after how the hidden blah blah. [panting] [sighs] >> Uh let's have a look. Uh there we go. Active inference is oft Yeah, I I had a bit of an attempt here to answer. It's often applied to physical navigation tasks such as a maze or learning motor control. But if I want to build a model that's primarily captures a person's mental states, interior receptive signals, bodily awareness rather than physical, would there be different criteria to consider? Yes. Uh in the sense that the things that we're modeling or the assumptions we're making about the hidden states is that they are either continuous or categorical. Um typically so if you've got some sort of maze uh task and this is very common in active inference. um you you know you'll come up again and again and again this sort of t- maze learning task where you've got a a grid of states you can go to a que that que will give you information about whether some other state is rewarding or punishing this comes up a lot so if you're modeling this kind of scenario you know should I go to the queue or should I go directly to where I think things are rewarding uh but then there's costs associated with that if you're modeling that kind of thing it's typically very useful to model uh uh to use a POM DP [clears throat] partially example mark decision process and categorical distributions but you know you might want to have a continuous representation of this. So I was trying to get at that difference in in my my uh reply here. So I did I did I not quite finish maybe I didn't quite finish my my reply. Um >> um no big problem just a little confused. Yeah, I I'll go back and review that. But that to the degree that there would be modeling considerations about the differences between things like navigation and things like interiorceptive um >> modeling. Basically the biggest divide you have is between continuous state space and active well discrete state space models. So you would imagine that for something like a maze okay it makes sense to do continuous uh you know categorical modeling but uh yeah it's it it has basically it comes down to particular problem at issue. So you want you want to you want to select a model that's well suited for the for the problem at issue basically. Hope that helps. >> Okay. Okay. Thank you. >> Are there any more questions? I'm sorry. Um it was a bit sort of prefactory um today but this is it does pay to to really go through the the motions on the notation of things and how how we're representing the model now given that we're kind of radically changing from what we have done. So basically so you know 9.3 that takes us into the realm of variational free energy minimization and then 9.4 four is where we really kind of kick into high gear and start using these models uh for for kind of genuine quote unquote active inference. So grid world and controllable state transitions 9.4 four. That's that's really the kind of core of um what we want to do. And do we introduce planning there? No, that's a 9.5. 9.5 is going to be that's going to be a watershed moment, that's for sure. So, all right. Going once, going twice, going three times for questions. I'll stop the recording if there aren't any more for the recording and then anyone who wants to ask a question can. But other than that, thank you very much. We'll see you guys next week for 9.3 to 9.4. That's what we'll do next week. So, thank you.