Video summary
The evolution of object detection began with the introduction of the Detection Transformer, or DETR, which revolutionized the field by framing detection as a direct set prediction problem. Unlike previous models that relied on non-maximum suppression to handle overlapping boxes and required significant post-processing, DETR utilized bipartite matching during training to assign one predicted box directly to one ground truth object. This approach allowed the model to output final predictions without any additional processing steps, streamlining the detection pipeline by leveraging a convolutional neural network backbone followed by a transformer encoder-decoder architecture where object queries are transformed into specific class and bounding box predictions.
Building upon this success, MaskFormer extended the concept of set prediction from boxes to segmentation masks, effectively shifting segmentation tasks from pixel-level classification to direct mask classification. Similar to DETR, MaskFormer employed bipartite matching during training to align predicted masks with ground truth masks while suppressing incorrect ones, enabling simultaneous output of all segmentation masks without post-processing. The architecture involved passing image features through a pixel decoder to create embeddings and utilizing a transformer decoder with query vectors that performed cross-attention to predict both the class and the corresponding mask, creating a robust framework particularly effective for semantic segmentation where distinctions are made at the class level rather than the individual object level.
Despite its strengths in semantic tasks, MaskFormer faced significant challenges when applied to instance-level or object-level segmentation due to the inherent inefficiency of transformers when processing long sequences. To maintain reasonable inference speeds, the model was forced to use smaller image feature tensors with lower spatial resolution, which inadvertently made it difficult to detect small objects and degraded overall performance on instance segmentation tasks. This limitation highlighted a critical trade-off where optimizing for speed compromised the ability to handle high-resolution details necessary for distinguishing between different instances of the same object class within an image.
The solution to these limitations was introduced with Mask2Former, which retained the core architecture of MaskFormer but enhanced efficiency through the use of masked cross-attention and multi-resolution feature inputs. Instead of feeding a single low-resolution tensor to the transformer decoder, Mask2Former passed multiple image feature tensors of varying resolutions in a round-robin fashion, allowing the model to process high spatial resolution data without sacrificing speed. By restricting attention mechanisms to only the relevant image features corresponding to specific objects, such as focusing on a dog rather than the entire image, masked attention significantly improved computational efficiency. This advancement resulted in a general-purpose architecture capable of excelling across semantic, instance, and panoptic segmentation tasks, ultimately solving the previous bottlenecks and setting a new standard for how segmentation models are designed and trained.
Read the full video transcript
What if DETR predicted masks instead of
boxes? Let's level set this with the
Transformer neural network introduced in
2017. It was originally introduced for
language translation, and this quickly
spread to other natural language tasks
and eventually to computer vision tasks.
One such computer vision task was object
detection. This was done with the
detection transformer or DETR.
DETR framed the problem of object
detection as a direct set prediction
problem. This means that DETR will
assign one box to one object in an image
without the need for much
post-processing.
And this is unlike previous models,
which needed non-maximum suppression to
remove overlapping boxes, which would
significantly affect performance. So,
how does DETR do this? DETR accomplishes
this during training by using bipartite
matching. It matches one predicted box
to one ground truth box and suppresses
all other predicted boxes assigning it
to no object. And so, during inference,
we take an image, we pass it to a
convolutional neural network backbone to
get a tensor that represents the image.
We pass this to the transformer encoder
and decoder architecture. Then the
decoder has these object queries, and
they are vector placeholders that will
be transformed
using the image tensor
into a predicted class and bounding box.
While only four predictions are made
here, we only see two bounding boxes
because two of the other predictions
were suppressed with no object
predictions. And this could perform
object detection quite well. Now, the
main question is instead of having these
object queries output boxes, can we have
them output segmentation masks?
MaskFormer does exactly this.
Previously, segmentation was done by
pixel-level classification where we
classify pixels into predefined set of
objects. But with MaskFormer, we define
segmentation as a direct set prediction
problem, just like DETR. So, it outputs
segmentation masks simultaneously
without the need for post-processing.
MaskFormer accomplished this this during
training using bipartite matching. It
matches one predicted mask to one ground
truth mask and suppresses all other
predicted masks, assigning it to no
object. And during inference, what we do
is we will take the image, pass it to a
backbone to convert it to image
features.
This is passed to a pixel decoder to
create embeddings at the pixel level.
And both of these units can be
convolution neural networks for
efficient processing. And the pixel
decoder could also use a feature pyramid
network to further enrich this image
tensor. Separately, we pass the image
features to the transformer decoder and
perform cross attention with these query
vectors. These query vectors are
placeholders that will be transformed
to predict a class and a predicted mask.
And we get all the masks simultaneously
without any post-processing required.
And this forms particularly well with
semantic segmentation, which is
segmentation at the class level. But it
struggles to perform well with
instance-level segmentation, which is
segmentation at the object level. So,
why is this the case? This is because
transformers are inefficient for long
sequences. That means larger this image
feature tensor, slower is the
transformer operation. And hence, we use
a smaller image feature tensor so that
the transformer inference speed is
reasonable.
But, small tensor means that it has a
smaller spatial resolution. So, smaller
objects are much harder to detect well,
and thus degrading the performance of
instance segmentation. So, how do we
solve this? Mask 2Former solves this
exact problem. It follows the same
architecture as the MaskFormer, but
instead of just passing one
low-resolution image feature tensor to
the transformer decoder, we pass
multiple image feature tensors of
different resolutions to the transformer
decoder in round-robin fashion. To
handle these higher-resolution tensors,
the transformer needs to be made more
efficient. And it is made more efficient
using a masked form of cross-attention
instead of full cross-attention. In this
image, let's say that the green query
vector is somewhat predicting around the
dog from its previous layers. In full
cross-attention, we would perform
attention with the entire image. In
masked attention, we perform attention
only with the image features that
correspond to this dog. And this makes
the cross-attention
more efficient, enabling larger image
feature inputs, too.
And so, Mask 2Former performs just
better overall on all kinds of
segmentations.
So, in summary, if we were to ask, "What
if DETR predicted masks instead of
boxes, well, DETR framed object
detection as a set prediction problem.
It assigned one predicted box to one
object, removing the need for
post-processing.
So, MaskFormer asked, well, what if we
applied the same idea to segmentation?
MaskFormer thus transformed segmentation
from a pixel-level classification to a
mask classification.
And while it performed well for semantic
segmentation, it struggled with instance
or object-level segmentation.
Mask2Former improved this framework by
making the attention more efficient with
masked attention and allowed high space
resolution image tensors to be
processed. And this created a general
architecture for semantic, instance, and
panoptic segmentation. And that's all
that we have for today. If you think I
deserve it and you like the video,
please do consider giving this video a
like. Thank you all so much for watching
and I will see you in the next one. Take
care.