This AI segments ANYTHING ... even VIDEO!

This is pretty insane. So, if you get a video and you have some text description, you can segment or isolate objects in the video. Now, we're going to take a look at exactly how this happens in this video. So, let's get to it. So, back in the 2010s, the way that segmentation used to work was through pixel level classification. This problem though is only great for something called semantic segmentation. But, we can't really detect every instance of an object here. In order to solve this problem, we can do something called a mask classification, where we'll take an image and we can produce multiple masks simultaneously, and each of them will be associated with some probability distribution of the object of interest. So, for example, we can have, you know, this mask be an actual, let's say, a building, and then this probability distribution says, "Hey, this is the building." Same with this being a sky. And we can have multiple such buildings, right? It doesn't have to be just this one building. There could be another building as well in one of these masks. So, not only are we able to do semantic segmentation at the class level, we are also able to do instance segmentation by isolating individual objects with this paradigm. But, an issue here now is that we have this predefined set of object categories. So, we would have to train it on, let's say, building and sky and dog and cat. But, if we wanted to detect like a boat, we would not be able to do that with this framework. So, how do we actually truly segment anything? And that's where the hero of our story, Segment Anything, comes along. And the idea here is that instead of pre-training on certain object categories at the time of inference, we can accept prompts. So, this prompt, for example, could say, "Hey, I want only to segment the object corresponding to this like white dot, but exclude the object that corresponds to this red dot." And so, we can like isolate the frog here. And so, this segment anything model is trained on something called promptable segmentation. Where we take an image, we can take a prompt, pass it to this model in order to get a valid mask. Let's now take a slightly detailed view of this model. Where we'll take an image, we will pass it through a transformer image encoder. This image encoder would perform self-attention in order to get an enriched representation of the image. We'll then take the prompt, which could be a rough mask, it could be some points on the image like we saw before, a box, and pass it to a prompt encoder in order to get vector representations of each. Pass this to a mask decoder. The mask decoder will perform cross-attention between this image vectors and these prompt vectors. So, it enriches this image vector to be conditioned on whatever the prompt is. And so, what we can get from it is a segmentation mask, as well as a confidence score. A natural extension of this is instead of just making it available for photos, why not be able to do this for videos? And that's kind of what we're seeing over here. And this was introduced with SAM 2, which is the segment anything in images and videos. This is the architecture for SAM 2, and it actually looks very similar to SAM 1, but with these added memory components over here. So, we have the sequence of frames here of this like chicken here. Let's take a current frame right here of this chicken. We're going to pass it to an image encoder. As we said, the image encoder is going to perform self-attention. So, the output is going to be a bunch of vectors that represent collectively this input frame. Now, we perform memory attention. It's going to take information of the image and perform cross-attention with the historical frames that are stored in this memory bank. And specifically, all of this information from this memory attention to this memory bank is going to be storing information about a specific instance of an object. I recently found out that this is known as the chicken's comb. Pretty cool. In this way, we can kind of isolate the chicken comb in the current frame. And the mask decoder is going to help like before get the mask and some confidence for this segmentation. Now, this chicken with the highlighted comb over here in this current frame is going to now be stored in the memory bank so that it can be used in the next pass for the next frame. And so, you can imagine this memory attention and mask decoder acts as a great way to track this chicken comb throughout the video. And so, we're able to get like a pretty clean representation of tracking an individual object. But the issue here specifically now is that we can really only track a single object as all of these masks, these points, and these box only correspond to a single object. Well, how do we detect multiple objects or all instances of an object? This is now handled in SAM 3, where instead of just detecting like one instance of a cat, we are now able to detect all instances of a cat. And we also can use a text prompt to easily just guide our segmentation. So, let's see how this actually works. This here is the architecture under the hood. And so, you can imagine though the memory attention, the tracking with the mask decoder, and the memory bank, these blue components are remaining the same from SAM two. What changes now is that we have this detector here, and we have a masklet matcher along with of course a text encoder because we're taking text input. So, if I want to give a text like a chicken comb, this is going to be passed to a text encoder which is a transformer architecture, and because of its self-attention mechanism, it's going to create a sequence of vectors that represent a chicken comb. The geometry here is just going to be like an exemplar. For example, we could just draw like a little box around a specific object of interest to better guide like what we're actually looking for in addition to like the text prompt. And so, you can imagine that there's going to be three sequences of vectors, one from the text, one from this exemplar prompt and geometry, and the other of course from the image encoder. All of these three sequences are going to now be fed into a detector. Now, this detector is going to isolate or create masks around all objects of interest that are described by this text in this image and correspond to the exemplar. So, that's the output over here of like five chicken combs. Now, on this parallel track over here, let's say that we take the latest frame of the chicken comb, pass it to the image encoder, pass it to the memory attention and tracker. For every single, you know, past frames until the current frame, we can track a single chicken comb. This is effectively one masklet to track one object of interest. Potentially for every object we'll have a masklet and we can have multiple such masklets in this memory bank. And so what this masklet matcher is trying to do is, okay, I have a current frame over here with like five chicken heads and we're able to track, let's say we're tracking one chicken head. We want to do like a one-to-one matching between what's currently in the image frame and what is currently being tracked across historical frames and we can act on those. So, for example, for those, let's say five chicken heads, if we have only like one or two of those chicken heads that are actually matched to like um a masklet, like we're tracking it, then that means that well, the other three are not being tracked, which means that we want to initialize a new masklet in our memory bank. In the opposite case, let's say that we have been tracking like six chickens, but one little chicky went away and now we consistently only have five chickens that are detected in the detector, but we have six masklets here, but due to masklet matching, we might have one chicken that was being tracked that is no longer in frame at all. And so we would remove that masklet. We would remove that set of masks from the memory bank. And so we end up with this cool output where we have an image conditioned on text and we can segment any and all objects of interest in the video. And so for the AI that segments anything, the SAM model turns segmentation into a promptable task. So tell it what you want and it predicts the object's mask directly. SAM 1 segments an object in an image from a point, box, or mask prompt. SAM 2 adds the ability to segment an object in a video and it does so by incorporating the idea of memory. And then SAM 3 adds the ability to text prompt and detect all instances of objects in a video, and it does this with a transformer style detector and masklet matching. Together, the SAM family evolved from segmenting a prompted object to tracking it through video to finding and segmenting every instance of the concept, bringing AI closer to truly segmenting anything. And that's all I got for today. Thank you all so much for watching. If you think I deserve it, please do consider giving this video a like. What other kinds of broad topic videos like this would you like to see on this channel? Comment below, and I just might make a video on it. Thank you all once again, and I'll see you in the next one. Bye-bye.