Video summary
The evolution of AI image segmentation began with traditional algorithmic approaches that grouped similar pixels into regions based on intensity differences, such as the Felzenszwalb graph-based method. The landscape shifted dramatically with the rise of convolutional neural networks (CNNs) following the success of AlexNet in image classification. Early fully convolutional networks applied these architectures to segmentation tasks, but they faced a critical bottleneck where deeper layers lost fine-grained spatial details while gaining high-level context. To overcome this, architectures like U-Net were developed, which utilized skip connections to merge high-resolution feature maps from early layers with contextual information from later layers, thereby preserving the pixel-level precision necessary for accurate segmentation.
Further advancements addressed these limitations through architectural innovations in CNNs, notably the DeepLab suite. This approach eliminated the traditional bottleneck by employing atrous convolutions, also known as dilated convolutions, which expand the receptive field of a kernel without reducing spatial resolution, thus retaining more spatial information. Additionally, conditional random fields were integrated to refine edge details and improve boundary accuracy. These techniques enabled semantic segmentation at the class level, where all instances of a specific object type are grouped together. Subsequently, Mask R-CNN introduced instance segmentation by combining object detection with mask prediction; it utilized region proposals and ROI Align to generate distinct masks for individual objects within an image, allowing for the precise delineation of separate entities like multiple airplanes or people.
The field underwent another paradigm shift in 2017 with the introduction of transformer neural networks, which brought attention mechanisms and superior scalability to computer vision through models like MaskFormer. Unlike previous methods that predicted a class for every single pixel, MaskFormer reframed the problem by simultaneously predicting multiple masks along with their corresponding probability distributions, effectively decoupling segmentation from fixed category labels. This era culminated in 2023 with the release of the Segment Anything Model (SAM), a foundational model trained on a massive dataset of one billion masks generated across eleven million images using an automated data engine. SAM operates as a promptable system that accepts various inputs—such as points, boxes, or rough mask sketches—and generates precise segmentation masks without requiring prior training on specific object categories or tasks.
In summary, AI learned to segment anything by progressing from simple pixel grouping and fixed-class classification to sophisticated, class-agnostic systems driven by prompts. The journey moved from early region-based algorithms to end-to-end pixel classification via fully convolutional networks, then to spatially aware models using skip connections and dilated convolutions. The introduction of instance segmentation allowed for distinguishing individual objects, while transformers enabled the prediction of multiple masks simultaneously. Ultimately, the development of foundation models like SAM removed the dependency on predefined categories entirely, allowing the system to understand and segment any object simply by interpreting user prompts, marking a significant leap from rigid classification tasks to flexible, general-purpose visual understanding.
Read the full video transcript
How did AI learn to segment anything?
When we talk about segmentation, we mean
drawing outlines of objects of interest
in an image. Before the rise of
convolution neural networks, algorithmic
approaches to segmentation were very
prevalent. In this example, this is
known as the Felzenszwalb algorithm,
which is an efficient graph-based
approach where we consider every pixel
as a node and the edges connecting them
are the differences in the pixel
intensities. And we just start merging
these pixels together into superpixel
regions indicated by color over here.
This kind of segmentation could be used
as a preprocessing step or a feature
extraction step to other computer vision
applications at the time, like, let's
say, object detection. Now, the revival
of convolution neural networks actually
came with this paper on ImageNet
classification with deep convolution
neural networks. This paper introduced
the AlexNet architecture, which is a
convolution architecture trained on
image classification. And this
convolution neural network architecture
performed better than other handcraft
approaches in image classification. And
this eventually spread to all parts of
computer vision, including segmentation.
For example, in 2015, we had fully
convolution networks for semantic
segmentation. So, what this involved was
taking an image, we'll pass it through a
sequence of convolution, activation,
normalization, and pooling layers, which
is just like AlexNet, in order to get
these pixel-level segmentations. Now,
the main issue here is this bottleneck
over here where we have low spatial
information but high contextual
information. But, segmentation actually
requires very fine-grained pixel-level
precision. And so, losing pixel
information in the center degrades
performance. So, how do we deal with
this? Well, one way to deal with this is
using something called a U-Net style
architecture where we take an input
image and while, you know, these earlier
layers have high spatial information, we
still make the transformations as we
would, but we also use skip connections.
So, while we have, you know, these later
tensor layers have high contextual
information and low spatial information,
we can still append it with earlier
layer tensors that have high spatial
information. And this can be used to get
more accurate segmentation maps. Now,
another solution to this was by the
DeepLab suite of segmentation models.
And the idea here is why should we even
have this bottleneck to begin with? And
so, they decrease the bottleneck using
atrous convolutions and fully connected
conditional random fields. So, atrous
convolutions are pretty interesting.
They're also known as dilated
convolutions. It involves taking a
kernel and spacing out or adding holes
into the kernel. So, for example, if
this is a kernel of dilation one, this
is a kernel of dilation two. And then
what we can do is use this dilated
kernel, these like dark moving squares
as it's being applied, in order to
perform a normal convolution operation.
And so, because it sees more space, you
have more spatial information that's
preserved in these units. And so, you
have an input image, you perform an
atrous convolution in order to get
tensors that have high spatial
information than a normal convolution
network. And then what we're going to do
is use bilinear interpolation to upscale
it back to the pixel space.
We use these conditional random fields
in order to fine-grained understand and
highlight the details of the edges,
which is required for better
segmentation. And that's kind of what we
get. We get an output which has better
segmentation. And in this case it's
semantic segmentation because it's
happening clearly at the class level.
All of these airplanes are the same
class after all. Another really
important breakthrough for segmentation
was these Mask R-CNNs. So, Faster R-CNN
was used for a problem known as object
detection where we effectively draw
bounding boxes around objects of
interest. But what Mask R-CNN is doing
is that first, let's say here there is
an image and we will have multiple
bounding boxes which indicate region
proposals. They are potential areas
where objects could be present. We then
take this bounding box, we'll perform
something called ROI Align which will
basically create a 7 cross 7 tensor. We
determine what is this object
representing, what are the adjusted
bounding box coordinates. But it
additionally, we also create a mask. And
because of this, you could see that we
are able to now perform instance
segmentation where we are highlighting
individual objects of interest and also
drawing bounding boxes around them. Now,
all of these cases that we mentioned
were convolutional networks. But in
2017, we had the creation of these
transformer neural networks that had far
better scalability and they also had
these attention mechanisms to them. They
were originally introduced for natural
language processing, but they creep to
other parts of computer vision. One
place where they creeped up into
computer vision was in this MaskFormer
architecture. And the idea here is that
the MaskFormer redefined how the
segmentation problem was done.
Specifically, before when we used to do
for every single pixel, we would try to
predict the pixel class. That is for
every pixel, we have like K predictions,
where K is like the number of objects.
In order to get the final mask, now what
we're doing is we're taking an image and
we are just going to predict multiple
masks of different objects of interest
simultaneously, along with N
corresponding probability distributions
of what that mask represents. For
example, we have this first mask that is
created and this probability
distribution could represent and say,
"Hey, there's like a 95% chance that
this is a building." Whereas this one
over here would probably say there's an
85% chance this is a sky. And you know,
we have this last one over here, which
could just say, "Hey, there's like a
really strong chance this is no object."
And so, we have multiple mask
predictions along with probability
scores directly. It's around this time
that we saw the rise of foundation
models. And these are essentially large
models trained on large corpuses of data
on broad tasks, so that they can be
reused without training at all on
downstream tasks. In 2023, we had among
the first foundation models for image
segmentation and this was the segment
anything model or SAM. Because SAM is a
foundation model, it needs to be trained
on a general task. And this general task
that it was trained on was called
promptable segmentation. If we kind of
blow up SAM a little bit, it'll consist
of a prompt encoder, which will take the
prompt, encode that into vectors, and
it'll take an image, encode that with an
image encoder,
and encode this into like vectors, and
it'll perform like a two-way transformer
attention in this mask decoder in order
to create the valid masks of interest.
And of course, we need a lot of data for
this. Data which is usually not just
available on the internet because
segmentation is not such a prevalent
task. And so, what we'll do is we'll use
the model to annotate some data and
continue to train this onset data as a
data engine effectively in order to
generate 1 billion masks across 11
million images. And so, we now move to a
new world where given an image, we can
actually directly create certain
segmentation masks without the model
actually having been trained on that
task specifically. So, how did AI learn
to segment anything? If I were to kind
of summarize it would be early methods
grouped similar pixels into regions.
Fully convolutional networks made
segmentation end-to-end through pixel
classification.
UNets preserved spatial detail with skip
connections, while DeepLab used dilated
convolutions or atrous convolutions.
Mask R-CNN predicted a mask for each
detected object, and MaskFormer reframe
segmentation as predicting classified
masks.
Finally, SAM removed the need for fixed
categories. Given a point, box, or mask
prompt, it can segment regions without
needing to name them.
AI learned to segment anything by moving
from grouping pixels to generating
class-agnostic, prompt-guided masks. And
that's all that I have for today, and I
hope this video kind of made sense. If
you think I deserve it, please do
consider giving this video a like, and I
will see you in the next one. Bye-bye.