Fundamentals of Active Inference (Chapter 8, Session 39) August 25, 2026

Okay, once again, greetings everyone. Thanks for joining the active inference textbook reading group. We are reading uh this year's 2026 publication fundamentals of active inference by Sanjie Nam Jooshi published with MIT Press. Um this week we are starting chapter 9. Uh while chapter 9 is still within the broader part two of the textbook, as we get more specifically into um modeling in terms of active inference and not simply based on um sort of bases theorem alone or or making different kinds of linear assumptions alone, but rather bringing everything together now and looking at these core algorithms. Um, nonetheless, chapter nine is actually going to put us in a very different kind of space, a different state space, uh, as we'll be focusing more on discrete state space models. Um, we'll spend two weeks on this chapter today and then, uh, once again next week and then chapter 10 will extend the state space models uh, discrete state space models even further. But for now, we're just getting familiar with the sorts of nomenclature as well as uh conceptions of how we view hidden states and observations, all the rest within a discrete setting. So, first um before I go any further, I just wanted to briefly share this for anyone who's interested. Um this is the Oh, that pasted well. Um this is the computational psychiatry course uh being hosted this year online with uh uh University of Zurich and ETH Zurich. Um it's a really fascinating course. I won't spend much time on it because of course not everyone here is necessarily interested in psychiatry in particular. Uh however uh for the past several years uh Ryan Smith who has contributed uh of a lot of literature and and uh kind of progressive empirical work in the realm of active inference and including in especially discrete state space models that we'll now be covering. um he's given a tutorial on active inference including how to set up POMDP models which we we will be hearing a lot more about as we proceed further. Um but uh oops sorry. Yeah. So um so he will be doing a tutorial once again. Uh there are links somewhere through here to find some of his open- source code for those who are wanting to get into programming themselves. And um furthermore everything for this except for his tutorial I think unfortunately. But all of the sessions that are that are being hosted um can be accessed online and it is very affordable to get a ticket uh to watch everything online. So it's uh running from September 7th uh to September 12th of this year and can be watched entirely through Zoom at a much cheaper rate. Um to be clear, we are not sponsored in any way by TNU or or uh University of Zurich. simply want to make that available to everyone and let them know that these re resources are available and they also do have lectures from previous years. gives you a sense on different ways of in uh sort of integrating these things that we're learning about computational modeling at least with sort of models of the mind including in the context of uh you know uh psychiatric disorders, symptomatology um understand different ways that cognitive science uh and other fields that are not necessarily um um focused on psychiatry nonetheless uh have dialogue with psychiatry and vice versa and the development of these models And then interestingly of course we know that the textbook we're going over is for engineers uh you know not necessarily those who are specifically focused on the brain nor any sort of um psych psychiatric disturbances the brain is experiencing right so it's just to say that there's a lot of uh sort of dialogue going on between these fields and they all sort of learn from one another whenever we have those who develop more complex uh but but interesting or accurate algorithms versus the kinds of uh empirical work that's done with respect to behavioral studies and neuroiming uh data analysis and and so on. So wanted to make that clear. Uh and then furthermore for those who are interested in um coding I want to share we we've brought both of these up before um but this first one is SPM. Uh this is a you know a library that's existed for uh decades at this point. uh I think it was either the late 90s or early 2000s that it was started and it's it's grown significantly. Um and uh everything that we're about to witness in chapters 9 and 10 uh does have dedicated scripts in SPM for carrying out these kinds of functionalities. Um, so I think we'll want to see um the only thing is that many of these files have, you know, very technical names as they've sort of grown over time. Um but like this file here S spm MDP VB_X um this is [snorts] effectively code that we can use to make what are called hidden Markov models or Markov decision processes uh including partially observable Markov decision processes that's that POMDP model I mentioned earlier um these this pres pro provides a significant amount of just kind of syntax tax and something to sort of look at and get familiar with. Of course, SPM is a library for mat lab. I'm able to share this code with you because it is open- source, but mat lab itself is proprietary. There's no requirement that you learn mat lab specifically in order to do uh active inference writ large. Um, but nonetheless, as I said, SPM has been highly kind of specialized and and developed and uh improved over the years. There are many published simulations and active inference including in discreet statesbased models that specifically will reference how they used SPM to carry out their simulation and and design it and all the rest and hopefully they've released that code which will then work so long as you have mat lab and sp so that you can try it yourself. Um but here we'll see uh all of the different sorts of variables that we will be encountering uh in chapters 9 and 10. um and we will see kind of how these different uh matrices are constructed uh categorical prob probability distributions uh and all the rest. I will pause there because of course we're not we're not quite there yet to be able to start catching on to what is going on here. Um so what I would really prefer to do is first um bring us back a little bit in fact back to chapters one and two roughly um because as I've mentioned we're switching into these discrete state space models and so it's not as simple as taking one to one everything we've learned thus far and then relying upon it in order to develop these discrete state space models. However, what I do appreciate that I I think I interpret from how Nam Jooshi's written the textbook is that now that he's established everything for the continuous models, we're able to see these kinds of analoges between how continuous models and discrete states base models are created and how they run inference and learning and all the rest. that is they're obviously not equivalent, but conceptually if we sort of lined everything up between each of them, we would be able to draw connections between how one component works in the continuous case versus how it works in the discrete state space. So I think that learning by analogy is probably the fastest way to make this sort of quick transition from chapter 8 into chapter 9. Um so with chapter one we recall we were presented with the inverse problem which is can we infer hidden states using observations and the other way around can we uh if we know uh a hidden state can we then predict what the observation sort of should be. Um, we covered this very broadly uh and in depth over the first eight chapters, but now we're simply going to change how we're thinking about our X and our Y, our hidden states and our observations. That is how is everything encoded? We saw that in uh you know chapters one and two we were presented with these normal distributions or Gaussian distributions um and they have their own respective parameters with a mean and a variance term. And so this is where we're thinking in terms of scalar values. We're thinking about values like 1 and 7.9 and 342.62. um those are the kinds of sensory observations we're receiving. Those are the kind of values that we find whenever we're inferring hidden states, right? Um and then whenever we have a multivariate case, we have multiple hidden states uh all being linked to uh potentially multiple uh observations that are being received simultaneously such as you know um observations through the eyes and and through sense of touch and the rest uh all getting linked up to these different continuous hidden states. But still we're thinking of these continuous numbers or values. Whereas for the discrete state space we're switching to instead of using these normal distributions we're now going to be using predominantly categorical distributions and another kind of distribution known as the deerishlay distribution which we won't cover today although uh it'll be very clear whenever we get to it uh how it relates and it's incredibly simil similar. um a categorical probability distribution instead of viewing things as a continuous scale of you know if we could say in the continuous case uh you know the mean the the expected mean of a hidden state went from 2 to four and so we actually have this continuous scale we can look at we can say it increased the mean increased by two you know 4 - 2 is two uh as opposed to the categorical case where we can't quite do that we don't have a continuous a scale. We're instead thinking and this is where it's worthwhile to become at least a little bit familiar with basic probability theory uh including things like you know um what is the probability of heads whenever we flip a coin uh assuming it's a fair coin we know that probability of heads would be 0.5 and probability of tails would be 0.5 they're equally likely they're equally probable let's say um and those are two mutually exclusive states that the coin could be in or from our standpoint it's two mutually exclusive observations that we could receive when we look at this coin. Um, so we could represent that as a categorical probability distribution instead of it being a normal distribution that's parameterized parameterized in this way with a mean and a variance. Instead, um, categorical probability distributions can be much simpler uh, in the sense that it's just a vector where each index refers to a particular mutually exclusive state and the value represents probability. So we could have technically you know uh heads or tails would be 0.5 for heads 0.5 for tails and anytime we do some kind of computation we'll just remember that the first index is referring to heads and the second as tails as our two mutually exclusive states. And so these slides which I've been sort of slowly um adding to over time whenever I'm able um uh I've just linked them in the chat. Um this is a much earlier slide just trying to portray like these are the different ways that we approach active inference from the stamp and and not just active inference but more broadly all of the problems that we approach whenever it comes to machine learning uh and and deep learning and in certain ways even what LLM do um we're working with these different kinds of uh not just distributions but generally thinking about how do we encode uh information how do we encode the world so to speak that then allows us to uh sort of run computations over those those various encodings. Uh and whenever it comes to trying to do empirical modeling where we are the experimentter trying to build a model that represents um a a world or a particular phenomena uh we do have to imagine like how would our model conceive of and encode these different kinds of information. Um so with categorical probability distributions again we're not looking at these continuous values. There's also no assumed ordering right like 0.5 0.3 0.2 to for these hidden states sunny, cloudy and raining, rainy, we would say those are like hidden states like these are the states of the environment. We have to infer you know perhaps the observations are uh is it wet? Uh you know is there is there um you know are there being is the amount of light that I'm seeing in my visual field very high or very low? Um you know are there different kinds of shadows that I'm seeing because of clouds in the sky. uh however you like. It's just supposed to be an intuitive example, but it's just say if we treat those as three mutually exclusive states, we're not saying that cloudy is quote unquote more than sunny and then rainy is more than cloudy, right? They're all sort of on their own flat sort of playing field here and we're not comparing them in terms of one of them uh having some kind of higher value than another. Whereas that's exactly what we were doing in uh continuous active inference where we're looking at where do values lie when sampled you know and and how does that distribution look and we're able to make those kinds of statements of something being inherently bigger or smaller than than something else right um and then finally uh these whenever we compose uh a distribution like this all probabilities need to sum to one right um so we're always thinking about This is sort of like proportions. You know, if you if you were to to add up all the bars of sunny, cloudy, and rainy in this probability distribution regarding these three mutually exclusive weather states, you know, if we were to add up all these bars, they would just sum up to one. So we're thinking in terms of sort of proportions or what is we can say that maybe something is more uh probable than like one state is more probable than another hidden state but that has to refer back to these values right in the distribution. Um so um after that slide you know we proceeded on to chapter 2 said now we'll move to chapter nine. There's quite a bit more going on here because I I attempt to make these slides as sort of um not quite as dense as possible, but I'm attempting to sort of get at a lot of information at once um such that one doesn't have to repetive repetitively go back and forth between different slides and the rest. Um so what I've done here is section 9.1 uh tried to lay out this continuous versus discrete state space um perspective on on the modeling that we're doing. Uh in the top these are all things that should be familiar. Some of them go once again back to chapters one and two. Um we know that we are always here starting with bases theorem. We're not even talking about variational b uh yet. We're not talking about free energy yet. We just know that we're always employing basis theorem in this kind of basian framework. In the continuous case, you recall um you know we have our posterior uh being equal to our likelihood times a prior all over model evidence. That's the kind of nomenclature that we can use. It's effectively X representing his states and Y representing observations. And then whenever we kind of write that a bit differently um we can actually take the joint distribution between X and X and Y and we can say that is our model. Our model is something that uh you know kind of encompasses this broad distribution over hidden states and observations such that if you wanted to [snorts] ask of the model what is you know the probability of receiving a particular observation given some hidden states or some other such thing we could effectively have that whole joint distribution needed along with as long as it has some priors and a likelihood as its parameters we could compute its probability. Um so so we're able to sort of query this model and this this broad uh distributional space that it that it encompasses in order to find probabilities. That's kind of what we do with our our models, right? We we predict probabilities using hidden states and observations. And then whenever we look inside of what the continuous model looks like, as I mentioned earlier, uh we see that we were we were using these normal distributions to set up our likelihood and our priors. Now something very simple uh whenever we switch to the discrete state space is that we are still using bases theorem. In fact if you compare equation 1.2 and 9.7 being for continuous and discrete respectively they are the same thing. Um some may or may not like that we're switching out x for s and y for o. Um but it in my own personal experience as well as seeing that it's already been pretty well established in the literature, it is very useful to just remind us of what kind of state space we're working within. And it's also a good reminder in that X of course in the continuous state space is going to look like some kind of distribution uh here a normal distribution whereas our S in the discrete uh space or S hidden states are going to look like these categorical probability distributions um that I was referencing. And so if we compare uh what how the model is defined and what it contains in in the simplest cases we know in the continuous setting our model contains a likelihood state prior uh it contains a generating function which is essentially just um sort of how we get to uh the the likelihood we get that's how we get to the model's ability to relate hidden states with observations. And so um from there we just have some parameters. Now this will look much simpler in the discrete state space setting because technically our observation model which is like our our likelihood and our state prior which is conveniently titled the same thing with continuous. Um both of those also represent priors over hidden states and they represent uh the relationship between observations and hidden states. Right? So here we have uh and we name each of these uh a particular letter. We use a as a shorthand to refer to the probability of observations conditioned on states. We use D to simply represent the probability of hidden states or prior hidden states. So you can see how they're very analogous, right? Probability of Y conditioned on X, probability of O conditioned on S. Those are the analogs between continuous and discrete. And similarly, prior uh excuse me, the probability of X versus the probability of S, right? Those are our priors. Um so I've attempted to sort of link these things together. We can see how we can go from a prior distribution in the top case down to what looks like a discrete prior uh distribution on hidden states in the discrete case. Same thing for likelihood. Another thing that we can do is that for any particular observation in the continuous case we would use this yhat symbol uh where where we're effectively clamping an observation. That way we can query our continuous model. Well, if I receive a particular observation, observation with a particular value, can I say what its probability is? And there's a little bit more to it than just this cuz you recall that, you know, the probability of any single particular value is technically uh zero when we're looking at a continuous distribution like this. But uh whenever we're able to kind of slice things up and discretize, that's where we are able to make statements about probabilities. Whereas here in the discrete state space, it's actually uh a little simpler and more complex at the same time. It's simpler in that really we're still just looking at these mutually exclusive states and observations. So here we have four hidden states both of both of these boxes for the prior and the likelihood for this example discrete model. U they're they're coherent with one another. uh so that this is the prior for that discrete model and this is the likelihood for that discrete model. Again call that uh all probabilities sum to one for states and then for the likelihood similarly all of these columns will sum to one. So we're still very much in that kind of domain of making sure that it's a proper probab probability distribution. We can't have one value being 0.5 and another one being 1.3, right? it would it would kind of overflow our limit there. Uh we would need to normalize the probability distribution to put it onto a scale of 0 to one. Um so that is something we could do and we'll get back to that with durishlay distributions in future. So we're here we're looking at a case where we have four hidden states rainy, cloudy, sunny and snowy and then three different observations that the agent could receive which is water, hot or bright. Uh it's not the most intuitive example, but I think it's good enough to get at the point of here are sort of the the three different sensory observations the agent could receive. We're going to consider all those to be mutually exclusive. That is an agent could only perceive water or hot or bright at any given time step. we're in the we're still in this this world of of time steps being run over the course of a a simulation or a simulation that is comprised of you know capital t time steps where uh a step of inference happens at each time step. So at a given time step an agent might witness water and then infer that it is rainy because there's a high probability of that hidden state. Um, we've not actually talked about uh sort of uh inference in the proper sense just yet. We've just talked about priors and likelihoods and those will actually act as uh kind of formally how we use these words parameters. Those are sort of the parameters for our model. Um we're going to I'm not going to speak too strongly on that point because it can be conceived in different ways and I want to make sure that I understand how Sanjieve wants to present that kind of material. Um we're just going over sort of the the the fundamentals of discrete state space models today. Um so uh but but in any case you you can just see from comparing like this was the simplest model we essentially saw in chapter 2 this discrete uh excuse me this continuous one. And whenever we move to discrete state space settings um the model is even smaller right like it these are kind of effectively the parameters that they that contain all the information we need they contain these full probability distributions such that we could probably arrive at basis theorem from there um given that we receive some kind of sensory evidence right we as [clears throat] long as we receive some new observation we could we could run that kind of computation Um what gets introduced after that in roughly sections 9.2 9.3 is where we also add state transitions. And while I haven't represented this exactly how I would like to uh on this particular slide, I at least have enough information to get at the points that I wanted to sort of illustrate here uh which is that not only do we have a prior and a likelihood, but you recall that in roughly chapter six uh and going forward from there for continuous models, we have this situation where we introduced um flows to hidden states. we say that hidden states transition over time. Um that's in the case where we have a model that's capable of making inferences about a dynamic environment. So, not only do you have your A likelihood where you relate hidden states to observations, as well as your D, your priors on hidden states, um, but now, uh, with with this new B transition matrix, we're also able to have an agent who can make inferences about how the hidden state changes over time. It allows for sort it effectively adds this sort of temporal dimension to an agent, right? um in a static environment. Uh if we're just thinking purely in terms of computation, not too much respect to um let's say, you know, we're not using an example of a person who's making decisions and and thinking about empirical work. We're just thinking more more generally to understand these fundamentals. Um if you have a prior and a likelihood, then that means any given observation you receive, you could compute some kind of probability. Um but nowhere in any of that do we have some sense of how time works or how the hidden state might change over time um that the that entire dimension is lacking right it's just um you know you push a button and you make an inference and you're done um here uh we now that we have the B we're able to sort of compose everything together and say that the probabil first our generative model for for all time steps where the the agent actually has a sense of how one time step can shift to another and hidden state can then shift from uh the the current to the next time step. And I want to be clear and sorry for mixing up my words here but um we're still working with these kinds of marovian assumptions. Um that's why we have these phrases hidden marov which effectively is what we're looking at here. Um a hidden marov model has these marovian assumptions baked in. And so there's this kind of timelessness um uh sense where we're not necessarily baking a bunch of historical information into our model. It's simply saying uh that it's able to make an inference about how the world changes from one time step to the next. Not necessarily from one to the next to the next to the next, just one to the next. Um we'll we'll we'll kind of supersede that uh going forward by the end of this chapter, I think. Um, but for now we're just looking at these kind of foundational pieces to the hidden lockup model, discrete states base model. So we're effectively just saying that this agent has some initial priors about hidden states, you know, that the states of the the weather, you know, the their respective probability. What's the prior probability of it being rainy or cloudy or sunny or snowy? Um, we can imagine if one were in Antarctica and these were the only four things we could choose from, maybe their prior on snowy would be incredibly high. Uh, it might be the only thing. Um, it's this, of course, the reason why this isn't intuitive is that we can imagine it being sunny and snowy at the same time. In this kind of model, um, we're saying that can't happen. These are mutually exclusive states, right? So, you know, there there are many other examples that one can use aside from this um to try and get at the idea. Um heads or tails with the flip of a coin is much more intuitive, right? Because we know it necessarily can only be heads or tails in reality. Um so, we have our priors, we have our observation model or our likelihood. Again, that's that a and it's how we relate observations to states. Then here we have the state transition model. So again with the continuous setting we would have an agent who had some kind of transition model within itself. Uh it it would it effectively giving it a belief that hidden states can change over time uh irrespective or without any uh reference to what's going on with observations. It's rather that all this information kind of gets integrated together uh over the course of in inference. Right? It's like given this one observation like what do my priors say about hidden states? What does my observation model say about hidden states with respect to this observation I got? And then what uh is the hidden state based on how I think the hidden states change over time. Now integrate all that information and that that's that's sort of the general notion of inference that we've been following. uh barring from talking about action which here in in in chapter nine uh or at least early in chapter 9 we're not talking about action at all similarly to continuous states space models earlier in the book we didn't jump to action we only got to action when we whenever we reach chapter uh seven I believe right it took some time to get there so now here we're not immediately getting into action we're just talking about perception which is where we receive an observation and infer hidden states so these are all the sort of components that go into that for discrete state space models and then we can we can view how that looks a little bit more clearly and these equations. Um first we're going to frequently we saw that um there are all kinds of nice properties of logarithms and whenever we do these kinds of log transforms um back in the continuous setting right um by taking our precisions and using a log transform we're able to do different kinds of operations with them whilst uh helping them stay on a non- negative scale um so that you know everything can work properly but then we're able to sort of convert back between log and non-log um in order to do anything that we need to do. Uh so we're going to do a lot of that with discrete state space models as well. uh whenever we take the logs of these matrices at times um and I also included this equation in part because it's I think something that gets overlooked whenever people construct these models that um while it is very important to supply your D vector or your D matrix or tensor it's going to depend on if you're in a multivariate case or univariate case we're just looking at the univariate case all all of These have been for the univariate case. So far we're just looking at one set of observations you could receive like one observation modality and then one hidden state factor. You know they there's only one hidden state of the world. It could be one of these four. Uh there are three different observations in this world. At any step I'll only get one of them. Right? Um so in this univariate case um our our d will be a vector just as we've seen. um D is only kind of used in inference during this very initial time step, right? It it is our p our prior and in all senses of the word and that's our starting point. If I go to Antarctica and I haven't observed anything yet, uh my prior is probably going to be that it's going to be very snowy there, right? Um, I've not seen any observations yet, but as soon as I receive an observation, if I were sort of following this the the way the mathematics are composed here, as soon as I get that first observation and I incorporate my my prior belief that Antarctica is going going to be very snowy, um, is once I get that observation and we move on to the next time step, we're no longer in this world where we're thinking about um, what those prior values were because I've already updated my posterior, right? I've already updated that and that's going to effectively become my beliefs going forward. So the D is yes an essential part of our model. But I think what can happen whenever people start modeling is that they take D to be this this you know extremely important thing that's going to have a major impact at every single uh inference step. Uh this not necessarily the case. We will get there in terms of talking about learning in that whenever we went over continuous models, we moved from talking about perception and inference to learning and then finally attention where learning is updating our model parameters. We can we have that same kind of process with discrete states based models where the parameters can get learned or updated. Um in that case we will actually update the D we'll update the A and all the rest. Um, if you're running some kind of simulation with an agent, you know, that that would be expected to occur over time to where there would be learning of parameters and you might run some kind of behavioral experiment where where you know um a participant or or a mouse or whatever your your test subject is. Um, you know, say they they undertook multiple trials where we run multiple simulations using the same uh sort of agent there. um then in that case you would employ D at the first time step of each of those. So if you did have learning on the D, then that would matter. It would start to matter because you'd be changing the D over time. Um for all intents and purposes, if you're just running some simulation because you're trying things out, you're still learning how all this works. um or if you're actually employing a model for the very first time and you're trying to fit it to some empirical data uh but you have no idea what the prior should be. In those cases, typically the D will be initialized as being fully uniform. What does that mean? Uh a fully uniform distribution would look very close to how this um cloudy uh uh illustration here looks in the in our A matrix, right? every single state has the same exact probability. Um that that relates to this notion of the maximum entropy principle uh from from a mathematician physicist ET James. Um and the the general sense is that by beginning with a very uniform prior you're allowing the model enough flexibility to start learning in a good direction. that is you're you're not giving the model any preconceived notion that any particular hidden state is more um probable than the rest, right? It's kind of like going into a situation uh if you go into a situation where you're going to learn a new game for the first time. If you come in assuming that it's going to work like all the other games you played before, you're going to find yourself getting really frustrated whenever you get confronted with all the novel aspects of this new game and the fact that you do in fact need to learn it, right? So if you come in too biased towards thinking, oh uh you know soccer is just like uh baseball or something and as soon as you're presented with this that you know the fact that you need to hold a bat uh or something like that um you know you're you're going to be hit hit with a lot of surprise of course. Um and then it's going to you you potentially like uh conceptually would have to do some kind of unlearning that you have to unlearn uh you know that you that that you kick the ball with your feet. you know, instead we're going to be hitting a ball with a bat or we're going to be throwing it. It's going to depend on what position you're playing and all these other sorts of distinct rules, right? So, you would need to the the general idea would be come in without any uh strong preconceived notions, right? Um so that's one way of thinking about the D that is it only occurs at the very initial time step in any simulation and then it can be learned but learning it only becomes relevant if you're running multiple sim simulations where you will once again start at a new time step zero at the in [clears throat] the next trial or the next simulation. Um and then also if you're unsure of what to do from a a computational or or modeling or programming standpoint, uh it's very fine to initialize it as being uniform. Of course in a you know a psychiatry setting or in some other kind you know whenever you're developing a model a reinforcement learning agent model that you want to be safe and therefore you know it you want it to sort of avoid certain states or you want it to uh you know have a certain understanding of states of uh the world to where uh you know it's trying to realize certain ones and avoid other ones then it might be the case that you would you would give it some kind of of bias right um you would bias its priors um in different kinds of ways. So I also wanted to include back in uh chapter 2 and then over the course of chapters 2 through 8 we saw various kinds of graphical representations of these models. Um graphical in the sense that they are quite literally a graph where we have nodes and edges. uh nodes are the sort of circles representing our variables and then the edges are just referring to there's some kind of relationship uh between the u the two things that the the edge connects right so for um figure 2.8 8. This is was a continuous model where uh our agent infers X uh as a variable and it's going to use the um uh variance and mean of X as its parameters because it was a normal distribution and then our other node Y which was our observations. um there's some kind of parameter uh we didn't get quite get there in chapter 2 is in chapter 3 that we saw more about these in learning u but nonetheless uh it's effectively just these different parameters connect back to it for y there's some kind of um variance or belief about the variance but uh y itself is known because the agent directly receives it that's why we don't see a mean value there it's just it actually directly observes the the y um but then there's some kind of hidden parameter so to speak that's conditioning uh y and potentially conditioning the relationship between x and y. It's quite a mouthful just to say that we can represent these models in terms of graphs. So figure 9.3 um also makes the point that we can do a very similar thing for discrete state space models and there have been different kinds of ways that these have been graphically composed. um the this figure on the left uh is closest to the kinds of basian networks that we've been looking at uh uh earlier in the textbook and and um Joshi does make the point that he will continue to largely focus on uh these kinds of basis network graphical illustrations just to keep things more consistent and um you know sometimes it's a matter of preference or sometimes it's a matter of trying to find different ways of illustrating different aspects of models that makes one graphical architecture um sort of or or or notation uh you know it makes certain things easier to represent or not with a model. Uh nonetheless of course we don't want to be um you know we don't want to stray off course with with um you know any kind of arbitrary graphical designs and so largely focusing on basian networks different kinds of factor graph models is what we'll frequently see them phrased as or fory factor graph models another kind of factor graph um that conceives of nodes and edges a little bit differently but we're still able to rep represent a lot of the same information. >> [snorts] >> Um, I also pulled in uh this is from the previous textbook um and I I since it's freely available here um uh through MIT Press's website, one can just simply download the book. Um for anyone who wants some additional assistance with uh you know getting into discrete state space models they can especially look at chapter 7 uh active inference in discrete time. This this was the textbook co-written by uh Giovani Psulo and um and uh Thomas Parr and Carl Fristen. And it's here that they they also split up active inference into sort of continuous models and discrete models. Um and so we get these different kinds of illustrations. we get representations here of all the different ways that we can kind of compose these different parameters the A and the B the D. Um it also opens up with those and talks about hidden markup models. So you'll see a lot of similarities. So I think that if if there's any kind of confusion that arises especially here in chapter 9 where we're just getting at the fundamentals of discrete models um you know is a matter of opinion I think that nami has done a really good job actually of getting at these without saying that anyone has to hearken back to the previous textbook. I'm just simply making this known that it's available um and they give their own examples from previously published work. Uh so uh in a little bit more context on uh some of the different dynamics and components of these models that uh still are simply using the A and the B and the D the same material but we get them from different perspectives. Like here's an example where um where uh you have a person who uh knows what uh the script is for a particular song, right? So they have their own beliefs about what the correct notes are for a song, but then they're also listening and so receiving these like audio observations, audible observations um of the real notes that are being played. So the the the the person the musician kind of make inferences uh about what note they heard versus what note it should have been. And then we can see what happens whenever they uh you know the wrong note is played versus their belief about what it should have been. And using different forms of of plots and graphs and uh shading to show kind of how these categorical probability distributions look uh as as things play out over time. I also very much like the 2022 textbook for those who are interested more precisely in in neuroscience and neurobiology and and the theoretical and computational aspects. we actually get these different kinds of like waveform uh uh patterns that start to align much more with a lot of the initial inspiration for this kind of modeling. And its purpose was to be in relation to uh neuronal activity such as local field potentials uh raw amplitudes as well as like um uh frequency um analysis and kind of spectral analysis uh of of different frequencies. Um, in any case, uh, Sanjie nonetheless does a really good job, I think, of presenting us with something pretty coherent in relating continuous to discrete models. So um before I go on too much further, I mean I know I've spent almost the entire session so far talking about these, but um I think it's really important in that if we miss the fundamentals of discrete models um or it's going to be much more difficult to sort of catch up whenever we start expanding these models. Um, Sanjief has his own illustrations, but you can see in in that chapter 7 from the earlier textbook I mentioned like already we go from this hidden Marhoff model that we were talking about where we have an agent who effectively receives observations relates them to hidden states through their a uh parameter their a matrix or a tensor um as well as using using their their initial priors the d which again you can see here it's the only starts the simulation. It doesn't technically um persist, right? That's that timelessness of marovian processes. We've sort of already updated uh our our beliefs or posteriors from there and then those can be used as uh themselves kind of prior to be updated after that. So the D is kind of left behind. Um in any case, so so all of that information is is already something that needs to be understood fairly well. Um and understanding kind of how these graphical models work um and and how these matrices are composed and how you know the basics of probability these are mutually exclusive states, mutually exclusive observations. There are probability distributions that must sum to one between all of them. Um, we soon get into uh the POMDP model which I brought up much earlier. Um, these are very similar and that they still have an A and a B and a D, but now we've added a few things, right? We've added a prior uh over observations themselves. And we'll actually once again we have all these analoges between continuous and discrete models. We'll see that the priors over observations uh in our our C parameter are a bit similar to very specifically the bump Gaussian bump example um that we saw back in chapter 7. Here the agent um had capacity for action and so it had these beliefs about exogenous forces. V and what we did was that we we fixed them. We kind of clamped them, right? And so Nam Jooshi referred to this as a kind of preference prior. It's almost as if by clamping or kind of kind of making the the belief about V static or stable. It's saying that this this now no longer functions as something that the agent updates rather it's a sort of fixed homeostatic set point. uh that it's as if the agent wants V to be that value. So here the idea was V uh was set to be zero in the agent meaning the agent wanted to be uh you know at this vertical height. So whenever we get when if we imagine the the agent in a 1D world of of water where a wave coming through can can push the agent upward whereas the agent was to remain uh flat on the ground at at the first coordinate without getting lifted up that is the the agent doesn't want to be pulled away or pulled upward by this wave the agent says oh I want to be um at this coordinate so I'm going to exercise action that realiz izes that that's this notion of self-evidencing. It's in simplest terms, it's saying that there's something in the agent that it sort of wants or something in the agent that perhaps it even needs uh biologically or something, right? Like our bodies kind of expect roughly a similar number of calories per day or roughly a similar number of hours slept per eating uh and the rest. And so um by kind of having this belief in advance about these exogenous forces and all the rest um whenever they're fixed means that we can act in order to kind of bring them into reality. Right? This is what we want to happen. And so action here is sort of in the service of realizing our expectations. So if I believe that I'm hungry or if I believe that I need food or I'm observing that I need food and I want to see that I'm no longer hungry I will eat I will act I will eat food have caloric intake in order to realize that preference that I have that that sort of homeostatic point set point of not being hungry uh is is is the way we can sort of phrase that and so we get very similar logic whenever we set these priors over observ observations um in in the particular way that we're learning about these models. [snorts] Um there will be more in future about you know you could have preferred states as well. You could have do do these things in very different kinds of ways. Um but the most canonical I think is kind of what's set out here in the textbook in 2022 and the way that Sanjieve picks that up in his textbook in 2026. And um and I would say from there, anything else that I'd want to introduce you all to is that we've also heard many references to uh PMDP which let me pull up their better website. Um very cool. Uh they they gave a a very nice presentation with the institute late last year. Um there were also developers at the last IY international uh active inference workshop or international workshop on active inference last year which was held in Montreal. Um they've done many updates to this library. It's now uh kind of successfully reached a 1.0 version. Uh so that there are many in the sense that it's it's more or less intended to be fully functional with respect to the degree of functionality that they included. Um so I do think it's very worthwhile for those who want to learn to uh code in active inference and especially discrete statesbased models um worthwhile to get into PIMDP. Um as well as uh the the different examples and tutorials that they provide. I think uh much of it was pretty well built out. Um we'll see references to this T-maze example. Um, you can see worded references to the team's example back in the 2022 textbook. Um, I think that Sanjieve may cover it. Um, uh, given I'll be leading these these groups, uh, the the textbook group for the next few weeks. Um, I I will absolutely be giving at least one code de demonstration in the next few weeks. It's more about which one to choose. There are many different ways that we could go with that. And again, it's very important to understand the fundamentals to creating these models before jumping too far ahead, right? Because I I I know that it can be quite interesting to to create really complex models. Um but um you know to to sort of use this this uh this old adage uh simple as possible and no simpler, right? That's kind that's part of the general idea of what the free energy principle is is about in some senses. uh almost philosophically and so I'll say that here too. Um sometimes the the model that can be best especially whenever it comes to getting started in learning is to choose something that's simple as possible and no simpler that is it can demonstrate uh and entails all the things that you're you're trying to learn but don't don't expand it beyond that. Right? um we don't have to jump to things like hierarchical models right away or or different kinds of uh complex relationships between hierarchies and all the rest. um we we will in fact come to that and I think being able to really just understand the the fundamentals, define them appropriately and then being able to sort of define what your sim simulation should be where you have a simulation that actually um you know at each time step over time in a real time setting it itself elicits these um you know clamped observations right it needs to be able to give an observation to the agent one of these mutually exclusive observations, then the agent needs to be able to run inference. It needs to be able to do perception. [clears throat] Uh that is infer hidden states from that observation and perceive from the environment. Then finally, if we're in a case where we have action involved where the agent can actually act in that environment, then the agent needs to be able to emit an action which then gets sent back into the environment. And so that kind of thing should take us back to, you know, the various kinds of figures um that we've seen um over the course of the textbook and this general sense of um action perception cycles. Please Well, in any case, we we we've we've seen this many times, this this idea that, you know, that the the um agent and environment are separated, right? Here we go. Um you know, we have the agent, we have the environment, there's no direct connection between the two of them. That's what this whole partially observable uh sort of framework or setting that we're in is is really about. That's where a lot of uh notions of like subjectivity. The idea of basian mechanics being a kind of subjective mechanics or or or mechanics of subjectivity. It's that we can't know the true hidden states of the environment, these true external states. So the best we can do is infer them. And so what separates us from the environment uh but allows us to have some kind of vicarious relationship with it makes some kind of inferences about it is our capacity to emit actions and our capacity to receive sensory observations. So to bring that back whenever one is you know setting up these different kinds of models you have to create an environment that has that it needs to be able to emit observations and it needs to be able to receive actions. Or if you're only working on perception, then it doesn't need to receive actions, right? It's just it's so in that in perception, it's just a matter of the agent is sort of passive and just perceiving what's going on without doing anything about it. Um, and it's the ability to kind of put together these priors and your D and your uh likelihoods that relates the hidden states to observations in your A. Then the way that hidden states change over time, the kind of discrete analog to continuous flow uh [clears throat] in the continuous case. Um being able to compose this of how one hidden state transitions to the next. So the probability of rainy transitioning to rainy staying rainy uh is what is that roughly a 7.75 you know um and so having [clears throat] those kinds of of distribution set up and then you can run this kind of marobian chain-like logic where you are just employing bases theorem over and over and over and it's you know from there that we already start imagining well what if this kind of inference is not tractable what happens whenever our state space is is huge uh and all the rest and uh it's it's in that case that um we do need to think about variational inference and that's where suddenly the free energy principle comes once again back into the picture. Um so given that we're quite low on time, I just wanted to briefly see I know that there are some some questions uh in the chat. Um yeah the D and the observations it's a cool right okay so yeah um I would say in the the kind of example with this hidden marov model here um that that uh we've discussed so we have our priors over hidden states So just just so we remember how this was in the continuous case, our D here prior over hidden states and we have this one just denoting this is initial hidden states saying at the very first time step like before anything's really happened. This is our prior that we kind of come to the situation with. We're initializing our simulation, right? Um it's here that our priors will just be a single vector. Um in this example in this simple univariate example um it'll just be a single sorry this doesn't like to scroll sometimes um it'll be this singular vector at the bottom here. Right? So we have we have this distribution over these different mutually exclusive hidden states. That's our prior um that is that is all that the D is. And whenever we run inference it's there that we will employ it one time only when we're at the first time step. Here uh Noshi's use zero to use zero indexing. So this is the zerith uh time step, but we're going to call it the first or very initial time step, right? That's where the agent will use its a likelihood just going to relate hidden states to observations, right? Because if you come to a situation where there's states and observations, but you only have a prior on states, then we've said nothing about observations. The agent has no encoding, no understanding, no sense of probabilities and relationships between how hidden states relate to these observations it's suddenly getting. That's where the A comes in. the a the likelihood um that's what allows us to make these statements about what is the probability of observations given hidden states or of any particular observation given a distribution of hidden states or given a particular hidden state we can run those sorts of computations so so without the A there's there's simply no relationship in our model for O and S right and if we have a prior on B that's what what gives us that starting point for basis theorem on what the states are so that whenever we get to you know this general uh uh way of looking at basis theorem where we know that our posterior is always a likelihood times a prior divided by evidence and we know that this evidence term can just be a particular observation we get at a certain time that's why we in the continuous case we're doing this clamping same thing, right? Y equals a particular observation. Then all you need from there is you need a likelihood and a prior. And that's exactly what these A and D U distributions, categorical distributions are all about. It's B that then gives a sense of flow to the state. It says, oh, but also the hidden state itself could change over time, right? an A and D alone would be enough in a static environment. You could, you know, you could just say like, yep, uh, you know, there are states, there are observations. What hidden state is it? Great, it's rainy. But what happens whenever the weather changes, right? This is nothing about the idea that states could change, right? It's just making probabil probabilistic statements, but there's nothing in here that has any logic of B being uh excuse me, of hidden states being something that can change. Um, so it's here by adding a B that the agent now has a sense of time and change over time. It has some sense that actually hidden states of their own accord can have some kind of transformation over time. In fact, the weather does change over time, right? Um let me let me just make sure I can uh so at that time D is but what about V? Yes. uh first there is no necessity for D to be uniform that would that that that is simply a recommendation for particular circumstances that is it you know someone who's doing computational psychiatry who has some theory that you know u an individual who's prone to uh exacerbated stress due to a stress disorder they might have a higher uh belief uh you know about something about the environment. You know, they might be biased in their view of how the environment works or they're perceiving another person trying to make inferences about another person. Uh you know, they might be uh inclined to believe certain things about that other person in a very biased way. Um if we wanted to represent that as a kind of probability distribution, we could very well not have a uniform D. We could say, oh, whenever, you know, our agent confronts that other person and they have this bias and then, you know, one of these instates will be much more probable. You know, um, you know, maybe our agent is is not very trusting of others, right? So, they they see another person and there are two probability uh two different uh hidden states for that person, right? Either either this person in front of me is trustworthy or they're not trustworthy. A uniform distribution would look like a flip of a coin, right? It would say, "Oh, trustworthy 0.5, untrustworthy 0.5. I'm not sure. I need to do some inference, right? I need to observe things about this person and potentially act as well to either try and elicit even more observations or, you know, find other ways to interact with this person." Um, right. It's it's there that updating and then begin. But the the D is simply that very starting step of what do I think about you given that I haven't observed anything about you yet. Right? There's no O or or or Y uh variable in this. It's just the S. It's just the hidden states. Right? Um so it does not have to be uniform. I was saying that you can make it uniform if you're unsure of what it should be. you yourself as the experimentter, right? If you're if you're trying to fit the model, if you're trying to, you know, you have some real empirical data, you're trying to run some kind of inference over it to try and do something, um, then, uh, yeah, you can you can absolutely make it uniform, which will make it essentially not biased at all. All of its beliefs will be kind of equivalent to one another. It's like well you could be trustworthy, you could not be um you know that will allow me to approach this in a less uh you know sort of predisposed uh you know biased way. Um and then similarly you could do that with the A matrix, right? Um if you did that that means your model has the the the architecture to relate observations and states, right? You have an a uh parameter here, a whole a whole tense or a whole set of these, you know, distributions that relate each observation to each uh hidden state or set of hidden states. So great. But if it's fully uniform, right, then that means it's not going to be able to distinguish anything. It's going to observe hot and say, "Oh, hot. Well, I have a uniform distribution over states for that." So, it could be rainy or cloudy or sunny or snowy. Each of them with a probability of 0.25, right? [snorts] Right. Like that would be a uniform a matrix. So it would mean that anytime you get an observation, you don't you're you're really not going to up your date, excuse me, you're not really going to update your beliefs much at all because you have no idea what this observation has to do with hidden states. And that's where all these questions about mappings like many to many mappings, one to many mappings, many to one mappings, um these are all, you know, modeling problems that have existed for, you know, decades. And um they're the kind of thing that allow us to make claims about precision that we were seeing before. You recall in the continuous case we're seeing precisions where uh you know you might have a model that looks um sharper uh you might have a distribution that looks sharper versus kind of more more spread out. And that relates to variance where the higher the variance, the more it's kind of spread out around the mean rather than very sharp and precise around the mean or in this case mode. Um but um uh so so that's the variance. Then whenever you take the reciprocal of the variance, that's where you have your precision, right? So very high precision would say we have a very precise kind of distribution here. uh whenever it's much sharper [clears throat] like this. A kind of analog to that is whenever you have a distribution that is not uniform, excuse me, that is not uniform. You can think of these uniform categorical distributions as having very high variance, right? Like it could be water, it could be hot, it could be bright, it could be any of those three things. It's not very concentrated around one of them, right? Whereas in this rainy matrix here or tensor here, uh water is much much higher. Right? Um the only reason why this doesn't look like the way that we've been representing these normal distributions is these sort of you know uh bell curves or or or humps or as however you like um is that there is no ordinality between water and hot and bright. This isn't a continuous scale. These are just three different things and it's very convenient to to plot them as as bars, right? We can we can plot them as as um um you know bar charts. There are also other interesting ways of using like shading. Um here there's a very faint shading uh in this which was a this is effectively an agent's beliefs about hidden states where the the black refers to um probability of one and then it was here at the this third time step that the wrong note was heard versus what it sort of should have been according to the agent's beliefs. And so using this shading to represent, oh, there's more uncertainty here. This this black is not a perfect perfect 1.0 black. It's a little a little bit lighter than that. It's rough, you know, perhaps it's like a 0.7 or something. And this slightly darker here is that this is the note it heard versus what it should have been. All the rest. So, um, I hope that that was useful in some some kind of way. And it is very much the case that D and A can be hypothesis driven. Um that would I mean they can be hypothesis driven depending on your your your scenario that you're setting up, right? I think in almost all cases it's going to be hypothesis driven unless you don't have one. If you don't have one, then going with a uniform distribution that that will kind of be sufficient. It's really going to come down to to the situation at hand, right? Um so so we'll talk more about that in the coming week uh in the coming weeks and uh again I do strongly recommend having a look at at PMDP uh having a look at SPM I know that SPM SPM can be quite a bit to get into whenever we have uh the the thing is that these scripts are really welldeveloped right they have so much functionality that being able to get a sense of how they work and not only that but like starting with some of the easier examples. I mean, you know, again, Ryan Smith, uh, who's going to be speaking at the the CPC in a couple weeks, um, he has a nice tutorial on active inference. And it's I mean it's it's probably been the end all be all quick guide to getting into active inference in discrete state spaces all the way down to making direct references to uh mat lab and sort of um I believe it provides some snippets of code in the appendix and then it makes references yep uh to you know that SPM MDP VBX it's incredibly long kind of legacy style uh file names um but so so I would have I would strongly suggest that you know it's really what do you want to get out of this textbook um uh yeah thanks Ahmad for sharing that uh for sure the um he has I mean he's given a gift >> advertisements uh fun Let's see. Oh, Berlin. Uh, yeah. So, this was several years ago. Um, he's given many of these talks and uh um they're they're really great. I mean, one thing that we are not getting in this fundamentals textbooks that we are not getting very much relationship to sort of the neuro neuroscience stuff. I mean, we are in the sense that we're getting all these philosophical and conceptual notions of what perception and action are. All of it is absolutely coherent with the with the active inference framework in general. Um but we're not going to see very much about local field potentials, right? We're not going to see very much about um uh neurotransmission or modulation other than a couple references to it when we were talking about attention in chapter 8. Right? So I mean there there there's so much more kind of to learn in this field. I think that Sanjie's uh uh textbook really is just something that will clarify a lot of the mathematics and it clarifies the mathematics and it gives you a much broader fundamentals toolkit for thinking about how to to make these models and that said um it is a whole world whenever you look at continuous models versus discrete models. So, it's great to draw out all these analoges uh between the two since we spent all this time on continuous. Uh but um in any case, I've certainly gone over on time, but I appreciate everyone's attention and I think that maybe next week and especially the week after that, I'll I'll allow more time for sort of discussion and question asking and answering um so that you know if there are any continued um you know uncertainties or points of interest that people have or otherwise um we we can actually talk through them. I just really wanted to be sure that we get through some of these uh fundamental aspects. And so again, you know, the this this the reference to the slides um are all on the first page. Thank you folks so much. Um and uh yeah, we will see you uh hopefully this Friday with Frasier's session and then I will be back same time next Tuesday. Um so yes, once again, thank you all so much. will now stop the recording but we'll also stop the session. So we'll see you soon and per usual please ask questions in the discourse and questions page on the the superhuman koda for the textbook group. Um thanks so much. See you at the next one.