Lesson 06 – A softer perceptron, part II: likelihood and loss

So, welcome to this lesson number six. I don't know why you're at fourth of usual students. Uh, more quizzes. I don't know. We see. Let's see what's going on today. So, lesson number six, we have a softer set. Go. All right. So what was the last topic that we covered in the last lesson? In the last lesson, we were talking about flipping coins. We did write a probability distribution that's called the Bernoli distribution. And we didn't write the name. We're going to write it today. And then we flip it four times and we had a specific outcome and then we [clears throat] expressed that the probability that the model assigns to be having observed the data set is a function of a specific parameter P. Finally, we saw that that function was something that was going between zero and one, right? Because P is going to be a number that can vary from 0 and one. And then the amount of probability that the model assigned to that specific sequence of outcomes was changing depending on different Ps. At a specific point, which was 14th, we had a maximum probability assigned to that specific observation of or events. And that is called maximum likelihood. You find out for which parameter you have the maximum ability assigned to a specific observations of outcomes. We're going to be using something similar today to find the weights of our software perception. But since we forgot everything and there are no new entries today, thank you for coming. I will just make a quick recap from last episode. Okay. [clears throat] All right. So back the first thing we were talking about was the perception prediction which is our y the t means is an approximation. been decided to have this one being equal the activation of my perception. And the activation is defined as whether the inner product between the feature vector of my specific object x with my weight is going to be equal or larger than zero. So this thing here is going to be just either zero or one depending whether the inner product is larger or not than zero. which translates whether we are falling on one side or the other side of the decision boundary. The decision boundary being the equation where the inner product is equal zero. Right? All right. Second one, we introduce the softer perception. So how do we soften this thing here? So we have softer perception. So what is another name for this function here where I have the square brackets larger than equal than zero. That's the heavy side step function. How do we soften the heavy side function? We introduce the logistics point is called. And so with software perceptron, we had that our prediction y is going to be this the logistic point for again the inner product between my weight and my feature vector. It's a number that goes from zero to one. and therefore was representing the degree of belongness to the positive class. Right? So how do we express this with this funny symbol? This is the degree of belongness to class number one for our object X. X is an object. Y is going to be an integer mean either zero or one a binary but here parenthesis right parenthesis. What happens if we bring the coldness to plus infinity? Or in another word, we're going to be considering the zero temperature limit for our softer perception. Where does the softer perception freezes to >> the perceptron? >> The perceptron. Why? Because the soft logistic sigmoid converges to the heavy side. And so if we take the limit for the temperature that goes to zero, then the softer perception, it becomes the perception, right? So the sigmoid beta becomes the heavy side just a reminder. We are not going to be using it for the moment. Then we went on to the point flips distribution and we said that P of Y which is in this case is a function right? Y has possibly has two possible values. So what is the probability for the specific outcome Y? And so here we wrote that if we consider lowerase P as being our parameter for our Bernoli distribution, we can express the probability of having a specific event Y as being our lowerase P to the Y times 1 - P to the 1 - Y. So if y is one then the probability is going to be just the lowerase p. If y is zero this one goes away and you're going to be getting one minus p right this is no no matching so far. What is going to be now the class belief? Well we have it here right. So let's write the class belief not just for the y equal one but for the generic case. So given an object an email X the class belief right the class assignment class belongness right if you want it's going to be given to us by something that is similar to this one but it's different because I look at a specific object so here we're going to be writing something that looks like what is the class for my specific object object X and how is this going to be correspondingly to this one? So in this case P is the same P for all my coin flips. I have one P. I don't look at the coin. Do I have different coins? No, I have coins. I don't I cannot look at the coin. In this case, I have different coins. I have different emails. So each email will have a different P. So we have a P per email which is called Y, right? So y is this probability for that specific object x to belong to class number one. So if I'd like to rewrite this expression here where we were not looking at the thing now we open our eyes it's going to be slightly different. Instead of having lower case p we're going to have y to the y times 1us y to the 1us y. Okay, it's exactly as this one, but now this item over here is different per email. If I check the email and I see, oh, uh, would you like to buy this Viagra or something like that? Well, it's going to be likely spam, right? That's also the things that come maybe to the inbox or if you see, oh, professor, can you add me to campus wire? Also, maybe spam, right? Or maybe just send it to the TA. Meanwhile, if he's like, "Oh, I would like to give you $500 if you come for a speech." That's ham, right? So, I can I can get some. >> All right. Sweet. Okay. So, uh what we have? Yeah. >> So, basically, we're replacing the P the probability with the perception. The perception acts as a probability here. >> That is what we said, right? So the softer perceptron gives us the probability that our X belongs to the class number one. >> Okay. And this is always the case where Y is equal to zero or one. >> That is correct. So if Y is the probability for X belonging to first class then one minus Y is going to be the probability to belong to the other class. Right? If you're 70% spam you can be 30% H. Right? One minus the probability for one class is going to be the other one since we are dealing with a binary classification. We saw that yesterday too, right? We said that this was equal to sigma of the inner product and the other one was either one minus sigma of the inner product or sigma of the negative inside. Right? We saw that yesterday. Well, the previous lecture. If you don't remember, review the content. That's why you're supposed to review things otherwise you don't know what we are doing anymore. So what was the last topic? We had a data set of several tosses. So again given here the last thing given a sequence of point tosses. We have that that the probability the model assigns to this sequence they are independent events right we just multiply all of these things here right so this is going to be a big multiplication n goes from one to capital n of what p to the yn * 1 - E to the 1 - Y right cool is there any doubt is there any uncertainty besides my probabilities in this blackboard for you is anything weird or you don't understand the actual meaning right I'm not asking about the symbols but each symbol has a meaning should be there should be some understanding of each of the things drawn on the board are we All good. Are we sure? >> Yeah. So is the only reason that we're able to use Bernui here because um we only have a binary classification. >> Correct. So next episode we're going to be expanding this to the multiclass classification. We're going to have to change something. But again, I just did mention the Berni but is in parenthesis. I'm not teaching probability. I don't care about the name. We're just using for now this thing here. Right? Okay. So can you guess what we're going to be doing now? If you are following this logical set of operations right what is missing next >> huh >> updating y >> updating >> like the weights the y >> the weights right so we actually fine that is a possible continuation I didn't think about that but let's go there so in the perceptron we did know we we knew how to figure out the weights how were we finding the weights so weight finding the weights is defined as training the model. How did we train a perceptron? I take a sample, I take an X, I compute the feature vector, I send it through the model which is computing the inner product. And now there are two options. Either the prediction is correct or is incorrect. If the prediction of the perceptron is correct, what do I do? Nothing. It's already correct. Why would you change something that works? Now, if it's not correct, we have a mistake. There are two possibilities either y is one but my prediction say is zero y is zero. So what do we have to do in that case? >> We want to make it larger right? So we want to increase the linear product such that is becoming positive. So both the prediction and the target are going to be both one. The other case if my ground truth was zero but my white is one I had to make it more negative. try to lower the inner product such that it doesn't flip on the other side. Okay, how about soft perceptron? We don't know. So we have to like here we didn't know about P but did we did find P eventually we found a P that was maximizing the probability of having observed that specific sequence of pauses. In our case we're going to be doing something that is very similar. We're going to be collecting a data set of emails and corresponding labels and then we're going to be trying to maximize the probability that the model assigns to having observed that specific set of emails. Okay, what did I say? Many words. Do we understand? No, it's okay. There is the recording. Anyway, in the next few four like the next blackboards are going to be going through all these steps which is exactly what you mentioned. So if you if you're lost just get back to now we're restarting. So here we were talking about the given sequence of tosses where I'm tossing these coins but I'm not looking at the coin. Now we're going to be doing exactly the same but for a sequence of events. [clears throat] So let's say uh we're going to be collecting we are going to collect a data set equal X n= 1 end of emails right and corresponding labels. For each email, we are going to extract it feature. So finally we want to compute The model likelihood, what is the model likelihood? Once again, the likelihood is going to be the probability that the model assigns to this specific data set as a function of its own parameters. What are the parameters of my software perception? >> The weight vector. Cool. If you're going to be performing maximum likelihood, we're going to find out what are the weights that are going to be giving me the largest likelihood for the observed data set. Okay, so let's write one more line here. Finally, we want to compute the model likelihood which is the probability of this data set with respect to the model weights and maximize. All right. So, let's do this. Right. So far, we we have a plan. So let's try to do what we put forward like what we are suggesting we want to do. So we would like to compute the likelihood of my data set right d which is I just write it only here x n y n one to n it's going to be a lot I just copy right it's going to be basically the same thing as here this is assuming that each email is independent from the other one right there's no se sequential order I'm just picking up my emails one after the other. So what is going to be different here? P was the same for all the points. It's the same same coin. Instead now instead of having a single pin, we have a specific estimate for the probability for each email I have based on what >> it's written there something right. What is the second last line? >> Second last line. Second last line. Okay. The the first word of the second last line, feature vector. >> The feature vector, right? I check an email again. If I see Viagra, Viagra, Viagra, I mean, it's likely going to be, you know, unless I have issues a spam thing, right? So, we are going to be coming up with some probability that is dependent on the content of the email itself. So, here we are looking at the content. You're not just blindly checking. Oh, how many students are emailing you? Too many. Right? So that that was the blind one. I don't check the content. Now I check the content. I see things. So my estimate for the probability of the specific email will be dependent on the content of the email itself. That's why there's a major difference between this expression over here and the one that is below here. Here this expression here says that you are looking at the object itself. In this case you're not looking at the object. You're just taking the mail is inside the envelope. You're not looking at at the thing. You're just thinking, oh, it's a spam or a hand. I'm not looking. The other one I actually open and I read and I see, oh, this is likely spam or not. Right? So how are we changing this? We're going to be changing this piece per uh per each object. So I just write y t which is exactly this one but for the n sample and then to the y n * 1 - y t n to the 1 - y okay [clears throat] what is this number this number goes between >> z >> 0 and one this stuff is also going between >> I multiply capital n numbers between zero and one small numbers >> becomes very tiny number. We don't like tiny numbers. Why don't we like tiny numbers? >> Hard to compute. >> We want to log somehow. >> The floats have >> floating points. You are computer scientists. You're not using real numbers. You're using fake numbers, right? I mean floating points numbers. Still f the floating points have holes, right? Like real numbers are continuous. You know, you have pi, you have e, you have all possible things. and Q Q numbers have the holes. Similarly, floating point holes you cannot move continuously because you have to jump you have to jump around. Okay. All right. So let's put a log in front such that so this is called the likelihood like right. Then instead we're going to talk here about the log t of my data set is going to be what do we do? Let me if I do the log of the product we get the >> okay very good. So I have the summation over here and it goes from one to capital n then I have another product. So I will have this guy it falls down and then I have plus and then this guy falls down and then I have I didn't make any mistake. Cool. So this is the log likelihood but I don't like it. Why don't I like it? So we said these guys here why are why are they between zero and one? because of the signal. >> Because of the signal, right? So this thing here we were we want to think about this guy over here. We have the S and then we have right we have this thing here which is up to one. But then I apply the log. How does the log look like? This is sigmoid, right? How does the log look? Can you draw the log your with your finger? So the log goes like this, right? Yes. No. Okay. What is this point here? >> Okay, we know. Very good. So this is going to be plus one. Cool. So if I apply the log to something that is going from zero and one, it's going to be negative. Do we like negative numbers? No. Maybe. But so how can we change this thing? Let's swap the the sign. So I will put a minus in front here. I will put a minus in front of here. Okay. So this is going to be our negative log likelihood. Okay. All right. One more step. It turns out that this thing here has a name with a minus inside as well. So this thing here we're going to be writing it. There we go. So this is going to be the binary cross entropy between yn and y. If you don't know what it means, doesn't matter. We're going to be uh yeah, it doesn't really matter for the course. But what does this measure? If y is one and then y tilda is also one how much is going to be the cross entropy zero. How about if yn is one but y n is almost zero. [clears throat] It's going to be negative infinity there's a minus in front plus infinity. So if they are similar if this is one and this is almost one this stuff is basically zero. If this is one and this is almost zero and this stuff is very very large and so this is measuring some sort of distance between those two items inside here. This distance is going to be taking the name as cost. Okay. So this is the binary cross entropy cost. The cost is a function that measures let's call it distance in this course. It's not correct but up to the potentation. Okay, should be divergence. All right, let me introduce you a few more symbols here. So I will say that here this negative uh low likelihood is going to be n times something that I define as being the loss for my weights and given my data set. Okay, so this is called the loss and the loss tells me how bad my weights are for this specific data set. So if the weights are good, the loss should be small, right? Meanwhile, if the the the weights are bad, the loss is going to be a large number. And therefore I can introduce one more thing. This item over here is going to be our capital L weight XN and YN. So if this is the loss which is telling me the badness of my weight. This is the badness tells me the badness of my weight for a given data set. Okay. Uh meanwhile here I have the single prediction is going to be the badness for my weight for just a specific x and y pair. Okay. And this is called the per sample house. So what is the meaning? This stuff tells me the badness badness of the weight for a given Y and X and Y. So these are definitions. We define n times the loss to be the negative likelihood and we define this single item here to be the percent sample loss. Okay. What is the relationship between curly l and l sorry >> it's the sum of all of the [clears throat] >> right but there is n times the loss is going to be the sum of all those things. So what is going to be the relationship between the L >> average the average right? So this thing here it's simply the average badness. So we can just write it down here. So my loss for my weights and the given data set is going to be the average deadness or if you don't like the the symbols from from physics I just write it down here. It's going to be G average percent loss. Okay. So this is the loss. Why do I have to put both arguments here? It's important, right? So the loss tells me something about what? in looking at the things written on the board. Yes, it's not incorrect, but the loss talks about what two things. >> It tells me how bad my weights are >> for a specific data set, right? So, both things are relevant quite relevant. So, therefore, the loss curly L for W and a specific data set is the average badness, right? Badness of diamond, right? Which is right. What I just wrote there. Okay, just yeah notation. It's just definition. There's no much going on. The only thing that is going on is that here we have the expression. So I give you one minute to to digest this and finish to copy and then we're going to be trying to shed some light about how this function behaves. Okay, like otherwise we don't understand like we understood some definitions but we have to understand actually how it works. So overall what was our objective here? So if this guy here is telling me the badness of my weight, which kind of weight would we want for our model? The least bad weights, right? And so how can we find the the best possible weights? So finally training that means another word for finding finding the optimal weights right it's going to be uh trying to find out least bad least bad least worse what's broken English so arg min arg mean of this guy triangular which we call remember how we call it I think I told you yes that's the W check >> okay >> so W check is the arc minimizer of our loss yeah >> we start the training with all the ways set to zero >> for example yes >> then give all the neurons have their way >> there is only One neuron. >> Yes. One when we move to multiple neurons, we have to change that. But we only have one neuron right now. One neuron with many weights. As many weights as the number of features. But yes, that's a good good question. Are we okay with this part here? We understand. No, you're confused. Why are you confused? like where does the set of weights come from >> coming from that we're like comparing >> over all possible all possible weight all possible weights >> so if a weight vector is in R2 >> then your search space will be all of R2 >> correct right so if I have a let's say R2 like he mentioned I can consider all possible wave vectors right I have all possible wave vectors here And one of these wave vectors with a specific length with a specific orientation will be giving me the least average badness. Yeah, don't worry. The next part of the class is going to be me finding this by hand. So it's going to be half the lesson is still waiting. >> Other questions? >> Are we assuming that that whole function is complex? Because we are assuming that there is some width that minimizes it. >> Okay, we don't have to assume. I can prove to you that this is actually a convex function. Uh I will not do it in class. I can I will provide some resources in the reading material such that I can properly address this question. Okay. >> All right. Yes. >> Losing. So you have the log the negative log likelihood and that's equal to the loss function. >> No the negative log likelihood is n times the loss function. >> Okay. >> It's written here. n times the loss equal the negative log likelike liquid. >> Yeah. >> And that's equal to the the accept fun sorry heavy side function. >> No no no no no. This is binary cross entropy. Ah there there is double bar. >> Okay. >> I know entropy with a h. >> Yeah. Yeah. Entropy right. Cross entropy actually. >> [clears throat] >> Yeah, >> the loss function always gives a negative value. >> No, the loss function is going to be always a positive value. So this stuff here, let me write it here. This is going to be always larger than zero. Why why don't no why I'm asking you why? >> Why is this non negative? Because the log function goes negative as it approaches zero. So the summation of the whole thing become negative. So we put a extra negative make positive. >> Sure. So the the log will be always negative because is going to be always lesser than one. Therefore by putting a minus in front it's going to be just summation of positive steps. Yeah. >> Good. Moving forward. Okay. Let's try to do some uh stuff by hand. Oh no. First before we have to understand what this expression is is kind of scary. So, let's take a picture and then erase the board for the next part of the class. Yeah. And boom, binary cross entropy. So the binary cross entropy which is our cost right binary binary cross entropy cost expresses this kind of distance between those two item over there. One item is a binary object either zero or one. The other one is going to be a number between zero and one with all possible values in between zero and one excluded. Right? So let's have a look at this expression over here. So this is going to be the y log y tilda minus 1 - y * log of 1 in y tilda. Okay. So where is y tilda coming from? from the activation of its activation of >> Y comes from >> WTF of X WTX. So this is going to be S but what S before there is something right >> the logistic thing right so let's have this guy here sweet and this is defined as 1 + x of xus one right all right so let's put this stuff inside the log the minus one comes down it erases the other minus So this stuff is going to be equal to y log 1 + x - x and then we have plus 1 - y* log of what? So what happens here? 1 - y it's can also be written as sigmoid of minus s. Remember we I show you that last time and so if I have sigmoid of minus s this one becomes a plus s. >> Yeah. >> Okay. So I just write it here. So I have log of 1 + x of s. All right, getting better. So, we can start to understand something maybe or maybe not depending. So, let's figure out what's going on. If my y is equal one, then I'm going to have this expression over here. This expression it's called soft class of minus s. That's if y is equal zero. So this is y= 1. This is y= 0. And then here I have this is going to be my soft class of s. All right. So let's draw this function. How does it look? Right. So I'm going to be drawing this thing over here. Uh yes, you are required to be able to draw things for your test. So yeah, first [clears throat] question. What is the height of the soft glass in zero? >> Uh zero. >> What is the height of the soft glass when s equals zero? >> Each two >> uh >> it's log two already. >> Yeah. How much is it? >> E >> calculator. Is it the natural log or? >> Yeah, of course it's natural log because we are in the math department, right? >> 693.7. Yes. So, we're going to be crossing the uh thing over here at 0.7. So, this is 0.7. Then, let's draw the blue guy first. The soft is going to be something like we come down like like this. And the other one is going to be there. Right. So this is going to be soft of minus s and this is going to be soft plus of s. Are you familiar with the plus function? No. Check on Wikipedia what it is this one or the soft or the plus function or positive part function. We're going to be using this extensively for the deep learning part of the course. Well, this is not deep yet. Still very shallow. Anyway, that was like an aside. So this is going to be a function for all possible discrete y's, right? The two possible values for y. If I just want to split that in two, I have the cross entropy for y= 1 and y tilda. This is going to be this guy over here, right? a soft class with a negative s and then on the other side I have a cross entropy for my y= z and tilda it's going to be soft plus of positive. So in the last part is going to be a lot of drawing with a lot of colors. So let's consider easy case where we have only a single feature. So let's consider my feature vector as being one and then f and then my weight vector being w0 w1 and also let's define fbar negative W0 divided by W1 such that I can express S which is W= F as W1 * F minus F. Okay, so far no magic besides this thing here. What's f with a line of >> negative bias divided by >> we just introduced this notation correct also I'm going to just erase it in a second so let's also assume w0= 0. Therefore you have that f just goes away. Okay so let's for now forget about fbar and the bias. We're only going to be interested in this multiplier. Okay, you want to have a look at which one? Let's go with this one. The cross entropy for the class being the first one. Okay, we're going to be drawing the following. So where is Fbar zero? Here, right? So here we have F bar. So f is the location of the vertical axis. Okay. So what happens now with the first option? Let's consider w 1 = z. Here we're going to be drawing h of y1. How much is my cross entropy for the class number one given that my w1 is zero? And we already assume that w0 is zero. How much is s >> okay s is always zero. Sweet. So oh I didn't tell you what is happening here. So here I have my single f. So this is my feature space. Be careful. This is feature space. This is s space. Right? two different spaces S space F space okay let's see how many of you get confused don't get confused so if S is equal zero for any F how is going to be the cross entropy how can I draw this come back to life people don't don't get lost okay where did I lose you could you repeat the question >> yes of Of course I can repeat as many times as you want the question. So we are trying trying to draw this cross entropy [clears throat] for different possible values of my W1 weight. Okay, we assume that W0 is zero. We don't care about that. So how is this function here which is this is we define it as the soft plus of minus s right? So this is soft plus of minus s. How much is the soft plus of minus s when s is equal to >> zero >> zero? >> Uh it's yeah 7 [clears throat] right. Are you sure? >> Yeah. >> All right. So if this is 7, this is going to be the cross entropy for this weight over here. This choice over here. Okay, sweet. Let's make a different color. So which color do you want? Okay, I pick green. What happens if I have W1 equal + one? It's it is good s would be equal to f. So it will basically be uh [clears throat] so plus minus s. Okay. >> When w1 is equal to 1 s is equal to f. When s is equal to f we basically get sub plus of minus s. Same paras. >> It is correct. How about the rest of the class? It looks like a monol one to one tutor session. You are supposed to contribute to this thing. Okay. You because you like the talk. >> Oh, no. No. I'm sorry. I just uh for some reason I thought we had to apply sigmoid first but yeah for for uh we actually don't have to apply sigmoid. We do directly on the s. >> All right correct. So the sigmoid head that exponential and so on. We replace it already here such that now we have a functional expression for those two cases. We are concerned to check how the cross entropy behaves for whenever we are considering classes. the red class. So this is going to be this function here. This guy here was just talking about this scalar fun scalar value s and so this is going to be fixed. Now we are going back to our email our features where we have that the s is a linear function of the feature. So we are interested in looking at how this cross entropy changes based on how my feature changes. Okay. So it's kind of slightly different perspective. This is across different S's. This is across different Fs. Yeah. >> Um we can use what like the kind of natural interpretation of W0 versus W1 is >> here. >> Yeah. >> So we have a weight vector. Remember my weight vector has two components. Uhhuh. >> Now I'm trying to try a few components to see how this cross entropy changes. So >> W is just an example where in this case it has it's a vector to >> correct. So I'm trying a few. So you were trying to do an argument minimization before you asked me someone asked me oh how do we try uh the other guy asked how do we find these weights? I try a few values right now. So I just fix one to zero. So we only had to care about a single value W1. So we are going to be interested in looking at how this cross entropy changes its shape given that I change this single parameter. Okay. Then we're going to find out which is the parameter that is minimizing the overall loss. Right? That was the second pass. We are trying to do by hand what we are going to be doing with computers later on. >> Okay. >> So he answered correctly. Yes. Question. Given the expression of f bar then you >> I'm confused now. >> Sure. Uh I already told W is zero. So just we get rid of this thing. So just forget about this fb bar. You can perform the multiplication. Right. So if you multiply w1 * this thing here you get the classical expression w1 * f plus w0 right so that is our initial inner product right this product here is w0 plus w1 * f_sub_1 right or f there is only one f I just write it this way because in the second part of this thing we would like to move this left and right so if you have a subtraction I can just change this f bar to move everything horizontally. So this allows me to do shifts. But we can just consider this as being wus w1 * a single shift and this is exactly our problem right and now I said this is not existent is zero. So we are just looking at s equal this item over here. Thanks for asking. Now I can answer. Yeah, I'm sorry. Is F equals >> Don't be sorry again. Try again. >> Is F equal to the vector one and F? >> F bar underneath is the vector of one and then single F. >> Single F. >> Yeah, single F. Right. Usually I have F1, F_sub_2, F3. I have a single F. So it's just F. Good question. Other questions. You're very confused. Yes. So we have uh X coordinates as S. What's what is the ycoordinate? >> The ycoordinate, this guy here, it's the cross entropy in this case is the red guy, right? So the red one is going to be the soft plus of negative s. The blue one is going to be the soft plus of s. The cross entropy is the summation of the two things but multiply by the corresponding selectors. So it's either the red one or the blue. It's not the sum. I mean it's the sum but multiply by something that is either zero or one. So you the cross entropy can be expressed as having two possible value given that y= 0 or y= 1. So it's like a selector. So this is the cross entropy either the red or the blue given that you are looking at the positive class or the negative. Other questions? No. Okay. So we were looking at the green case where my w1 is just equal plus one. So I just end up with S= F. So if S equal F, it's going to be exactly the curve. Which color over there? Red. The red one. Right. Sweet. So we're going to be drawing in green. Right. We come down this way. And we are going down this way. How about I draw in purple? How about W1= + 2? How it's going to change? Okay. Where is the soft of 2 * F cross the vertical axis? Same place. Same place, right? Of course. So, it's going to be crossing this thing here. But now, is he going Faster or slower? >> Faster. >> Faster. Multiply by two. So it's going to be twice as fast. So instead of being the bis sector of this thing, it's going to be half the bis sector here. So we can come down this way and then we go down here. Right? Is this true? When you have exponentials, you just do bis sector and you just go by it. It works for this function. Log 1 + x. If x if this is large the one disappears the log and the x disappears you just get the s or the negative s in this case. >> Okay. Thank you. Understand? Okay. All right. So let's do the other one other colors right. What color do you want? I have yellow. So how about I do w1 equal1? It just be the softness of s. is going to be just the soft plus of S. Right? So I have this one over here. So I come down this way and then last color. Which color we are missing? Okay. I don't have enough colors. I use white. How about W1= just going to be faster, right? Because I move faster. So I make it. >> Thank you. See, I make sure that you're following what's going on. So, we have this one, right? We come down here. Boom. So, these are all my cross entropies in the feature space given that I change my single weight w. So, which one should I pick? How do we how do we how do we decide which of these weight is the less least bad? What do we need? >> H0 hy= z muted mean >> the hy the one the other one is going to be just flipped right. So all of them are mirror of the other. If I consider the h of y equals z, it's going to be exactly the same diagram flipped vertically. I'm asking which is which weight should we pick? How can you answer this question? You cannot >> see >> depends on why >> depends on why not only how what did we write here before two two blackboards ago? Yeah, >> you can sum up all sum up all the values that you compute. Then you take the average. >> I guess to sum up for what? What is the summation for? >> All across entries and you like you take the average of that. >> Yeah, the average over what? There's a specific word missing here. >> The data set. >> Data set. We need a data set. You cannot figure out which weight you want unless you have the data set, right? So [clears throat] let's make a a possible data set here. So let's say I have my positive classes here. This is going to be still my feature. Then I have my negative classes over here. So how do I have the least? How do I pay? So direct cases are going to be the one that are paying these costs, right? How do I pay the least amount of cost for the red guys? Which of these one, two, three, four, five curves should I pick >> for the red guys? >> Tell me the color. >> Uh, purple. >> Purple, right? The purple has the least amount of cost for my red guys. How about for the blue guy? So for the blue guy, since we have to use the soft plus of pluses, I have to take all the colors and flip them, right? So which color would you use for the blue to have the least amount of cost? >> You have to do to flip, right? So to have the least amount of cost, you still going to be considering purple, right? So the largest my W1 and the sooner this purple will go down to zero at the after the crossing point. Right? And so in order to be paying the least amount of cost, I will choose choosing W= purple + two gives us the least cost for both uh classes, right? This is only the case where we fix W. >> Forget about W for now. I bring W after we are done with the next line. There is a question. We have four minutes left. >> Yeah. Yeah. Why does it work with minus chip? Because like for the blue the host cost is very high. The loss is high. >> No, no, no, no. For the blue. >> You have to flip it, right? For the this blue guy, you have to consider h of y equals zero, which is going to be the flip the mirror version of these guys. >> Okay? >> And the one that has the least amount would be the purple. Again, you have to flip them, right? >> Okay. >> Yeah. So, um here we're using like set choices for W1. I just tried five different weights and I figured out which of these is going to be least less less least worse less worse. Yeah. So when we flip the the white curve would be the one for W= W1 equals plus two >> when when we're considering the blue points it'll be the >> Yeah. So the the purple one will be the the white becomes purple one for the blue guy. Right. Okay. Good. So last line here. So here we chose W1 equal plus two to pay the least amount of uh cost. How about I have a different configuration of my data set. This is still my Sbar. And in this case, I'm going to have this case over here. What happens if I still go with the purple curve? >> If I go with a purple curve, I pay so much for this guy, right? So going with a purple curve. This one pooms, explodes everything. So we can no longer choose the purple curve. Similarly for the blue guy, blue guy would be going with this one. This would be the the purple one. Also this one goes to plus whatever huge huge number. So choosing W1= + 2 would give me a very large uh cost if it have like some mixed up in the middle. So how can we pay less? How can we pay less? Well, you just decrease this guy. Perhaps we can take the green. The green one makes me pay less for this one. But this one will pay a little bit more, right? So, that's fine because for the purple, this one pays so much more. The green one, this one pays much less and this one a little bit more. Okay. So for this specific case uh choosing W1= + one seems a better alternative to W1 equal plus 2. Okay. All right. Last word then I'm done. How about my points are not centered here but they are centered here. Let's say this is my Fbar. This is where my points are centered. So my points are like here. How can I move this task to the left? >> W you change >> I change well F is no longer zero. Now I have that w is going to be negative f * f w1. Right? So given that we already found out the w1, now I can just shift everything by changing this item over here. So now I can just move I can I can change a single parameter which is now dependent on the previous the value we with decided. That's why I took it that's why I took it off and that's why I use this, right? Because if I have F minus F bar, I can just perform a translation without having to worry about the fact that the value for the bias depends on the actual value of the first part. Okay, so the le the lesson is kind of done. Let me give you one more drawing then I I I I shut up. So let's say I have this specific configuration and then we are done. Question for you. Which of these two are going to be likely the solution that this arc minimization finds? This going to be this one or it's going to be this one. Which of these two orientation of the uh of the weight the pardon me this is the decision boundary and the contour contour levels. Which of these two will give me the least amount of loss right and why? So these were possible both were possible solution for the binary perception because if we are not making mistakes good binary perception doesn't care but you can you can see now if we are switching to maximizing the probability the likelihood or minimizing the loss one of these two solution will be preferred to the other one because it will is make us pay the least amount of cost right think about it questions No. So that was lesson for today. I know it was pretty much a lot of stuff. Please do review the content, go through the different steps for next week such we know what's going on and we understand. If we don't understand, tutor and okay, thank you for listening. I'll see you next time.