Video summary
Transformers originally revolutionized natural language processing with architectures introduced in 2017 for tasks like translation, followed by pre-training paradigms seen in models like BERT and GPT that achieved state-of-the-art performance through scaling laws. Around 2020, this success inspired Vision Transformers (ViT) to apply the same encoder architecture to computer vision by treating images as sequences of fixed-size patches for classification tasks. While ViTs demonstrated strong scalability with large datasets, they struggled with object detection because their uniform patch sizes made it difficult to capture both extremely small objects and large structures simultaneously; a single 16x16 pixel token might encompass an entire building while missing the details of a tiny pole within that same region.
The core challenge for early transformer-based detectors like DETR was balancing spatial resolution with computational efficiency, as standard self-attention mechanisms scale quadratically with input size, making high-resolution processing prohibitively expensive. To address this, architectures such as Swin Transformer introduced hierarchical feature maps by progressively merging neighboring patches to reduce token count while maintaining a hierarchy similar to convolutional neural networks. This approach utilized windowed multi-headed self-attention and shifting windows to linearize the computational complexity relative to image dimensions, allowing for efficient processing of high-resolution data without sacrificing too much context or spatial detail.
To further enhance object detection capabilities across different scales, researchers combined these hierarchical transformers with feature pyramid networks and U-Net style skip connections that fuse low-level spatial details from early stages with deep semantic context from later layers. For real-time applications where latency is critical, architectures like RT-DETR integrate convolutional backbones to generate multi-scale features efficiently before applying transformer attention selectively on lower-resolution tokens for global reasoning. This hybrid strategy effectively merges the speed of convolutions with the accuracy and long-range dependency modeling strengths of transformers, resulting in systems that can detect objects at various scales while maintaining low latency and high precision.
In summary, learning to see at every scale requires moving beyond single-resolution grids toward hierarchical designs that progressively merge patches to create multi-resolution feature maps akin to traditional convolutional networks. By combining spatial details from early stages with semantic context from deeper layers through fusion techniques like those in Swin Transformers or RT-DETR, these systems overcome the limitations of fixed patch sizes and quadratic attention costs. Ultimately, modern transformer-based vision systems achieve robust object detection across different scales by strategically trading off between latency and accuracy, leveraging convolutions for efficient feature extraction and transformers for enriched global understanding within a unified framework.
Read the full video transcript
Greetings, fellow learners. In this
video, we're going to talk about how
transformers learn to see at every
scale. So, let's start with the
transformer neural network architecture
that was introduced in 2017 for language
translation. And specifically, it was
used to translate like English to French
or English to German. And this performed
quite well, becoming now the soda for
language translation. Over the course of
the next year, we saw the creation of
BERT and GPT, which introduced this
pre-training and fine-tuning paradigm.
We pre-trained on a general task like
language modeling, and then fine-tuned
on very specific tasks with less data.
And so, on various tasks within natural
language processing, we saw BERT and GPT
become the state of the art. And so,
these transformer architectures quickly
took over natural language processing.
And around 2020, we saw scaling laws
that came for these large language
models for these NLP tasks. And it turns
out that if you increase the compute and
you increase the training data set size,
as well as the number of parameters of
the network, we could continuously
decrease the test loss or improve the
transformer performance without like an
actual attenuation going on. And so,
transformers scale very well for these
NLP tasks. Seeing the success of these
transformers on these natural language
problems, we have a vision transformers
that tried to use this architecture for
computer vision. And specifically, they
just took the same transformer encoder
architecture with very minimal changes
and tried to treat an image as a
sequence of patches in order to solve
image classification. In doing so, they
actually did kind of see some sense of
scaling even for these image cases. As
if we increase the training data set
size to like of the order of hundreds of
millions, we see that these, you know,
colorful circles, the transformers,
start outperforming even convolution
neural networks, which is this gray
region over here. So, transformers could
also scale well potentially for
image-related tasks. Now, while this
particular network was trained for image
classification, we have other problems
in computer vision that are very
sensitive
of objects in these images. And just
like using this transformer architecture
to solve, let's say, object detection
becomes quite difficult because you can
see here, for example, let's say in this
image, we are trying to detect a little
pole versus a large building. You can
see that when we break this image up
into like 16 pixel by 16 pixel patches,
these are fixed-size patches, you see
the image pole is just relegated to one
image patch or like one token, whereas
the building spans like most of the
tokens over here. So, you can imagine
when we go into the transformer
architecture and we're performing
self-attention,
we really don't get too much information
about these, you know, extremely small
objects. Well, the why don't you just
make these patches smaller? Why do you
need 16 by 16 pixels? Why not make it
just one pixel by one pixel or four
pixels by four pixels, something much
smaller?
But the issue then here is that this
multi-headed self-attention is actually
quadratic in terms of the input size.
Now, in 2020, we actually kind of saw
this issue being combatted with
detection transformers. Now, what this
DETR did was instead of processing the
image with a transformer network, it
actually processes it with a convolution
network in order to generate some image
features. And it's these image features
that are passed to the transformer
encoder and decoder in order to detect
objects of interest. And if you look at
performance here, you can see that DETR
is actually very competitive. The issue
though is for small objects for DETR, it
actually much more underperforms like
faster R-CNN.
So, why does DETR still underperform
faster R-CNN? And to understand this, we
can look at a slightly more detailed
architecture for DETR. And the idea here
is let's say we have this image of two
birds, right? We're going to pass that
raw image into this convolution neural
network backbone. That's going to give
us image features. And it's these image
features that we pass to the transformer
encoder decoder architecture.
But what are these features exactly?
What do we pass into it? Now, we have to
choose one of these tensors over here to
pass to a transformer neural network.
But we already just said that
transformers are pretty bad with
image input token sizes. So, if we were
to pass this early spatial layer into
the transformer, this we would have to
flatten it out, right? And this would be
55 * 55, which is 3,025
tokens, and each token is of like 48
dimensions.
Whereas this last layer over here is 13
* 13. That's 169 tokens of 128
dimensions. So, it's far easier for the
transformer to process this tensor as
it's like at least 10 to 20x less tokens
than these early spatial layers here.
But the problem of passing these later
layer tokens is that they don't have as
much spatial resolution. And so, we
can't detect those small fine-grained
objects. And so, passing, you know, that
tensor over here through to the
transformer causes more degraded
performance on detecting these small
objects. So, how can we actually deal
with these issues? So, one way to deal
with this is using something called a
Swin Transformer, and this borrows the
idea of hierarchy from Convolution
Neural Networks. And what I mean by that
is we have an input image over here.
We're going to create like a patch
partition, which basically means that
we're going to partition it into, let's
say, 4 pixels by 4 pixel grids. So,
that's now H by 4, the width is now W by
4, and the number of channels is 48 over
here.
We then create an embedding for it, pass
it to a Transformer block. So, the
number of tokens that are going in,
which is H by 4 * W by 4, is the same as
number of tokens coming out. H by 4, W
by 4, and this is the channel dimension
C.
So, we have this very high spatial
resolution tensor as the output of this
first Transformer block over here.
And then we do another stage of this,
where we will first merge neighboring
patches, that is, like 2 by 2 patches,
we will merge them together. So, instead
of 4 by 4 patches, they become 8 by 8
patches, pass it to the Swin Transformer
block. There is no change in the number
of tokens, and so we get H by 8 cross W
by 8 cross 2C.
And this has more contextual
information, less spatial information.
And we can do this for multiple stages
again.
And so, you'll have a hierarchy of
tensors effectively, just like we had
with Convolution Neural Networks. And
well, before we said, "Oh, but you know,
this Transformer block is not performant
enough or not efficient enough to, you
know, just partition with high
resolution." Well, to deal with that,
we, in the Swin Transformer, it was
actually made more efficient using
something called a windowed multi-headed
self-attention and a shifting window
multi-headed self-attention. Ultimately,
these are just techniques to ensure that
instead of being proportional to the
squared number of tokens, which is H
cross W in terms of images, like number
of pixels,
it is now proportional just to the O of
N or order of just the height and width.
So, it's linear instead of quadratic.
And so, this architecture now becomes
more feasible. And because now we
created these hierarchical feature maps,
we can use feature pyramid networks or
U-Net style architectures. With feature
pyramid networks, what we can do is
enhance the contextual information of
these earlier layers so that they just
become more performant. And in this
case, there's like a top-down approach
of combining whatever the later layer
information with the earlier layer
information over here in order to deepen
its context. And so, effectively, you'll
have these earlier layers just become
better at detecting smaller objects. We
can even use similar principles like
U-Nets. And while they were designed
mostly keeping in convolution neural
network tensors in mind, the same
framework now can be applied to these
like Swin Transformer output tensors,
too. And the idea is that we are using
these skip connections in order to
concatenate earlier layer spatial
resolution with later layer contextual
resolution. And so, if we compare like
ResNet architectures, like convolution
architectures, with the Swin
Transformer, you now see Swin
Transformers just perform better in
general. The issue now lies in this FPS.
It's basically saying the convolution
network is still faster than this
Transformer architecture. So, how do we
deal with this?
Well, if we want real-time object
detection, we can look at the
architecture for RT-DETR.
Now, with RT-DETR, we actually use
elements of efficiency from convolution
networks with more accuracy from
transformers in order to create this
architecture. So, you take this image
and you can pass it to a backbone, which
in this case could just be a convolution
neural network. And because it's a
convolution neural network, we can get
multiple stages of feature maps. S3 here
is going to have high spatial
resolution, but S5 will have more
contextual information.
And now what we do is this AIFI is
basically going to be a transformer
encoder. And because we know that
transformers are far more efficient with
less number of tokens, it's just going
to take S5, which has the least number
of tokens, perform like self-attention
or to get this more enriched tensor F5
over here.
And what we do now is a
convolution-based cross-scale feature
fusion, where we will fuse these much
lower resolution features with the
previous layers over here that we see.
So, this is going to be fused. This F5
is going to be fused now with S4, and
then that output is going to be fused
with S3, and then we do an upward fusion
as well in order to get three different
tensors of different scales.
And then we will flatten all of those
out into just individual vectors, and
from these multi-scale selection of
vectors, we will select a few of them.
They'll act as prior object queries.
These are just like placeholder vectors
to the decoder in order to come up with
bounding boxes around objects of
interest in the image. And this RT-DETR
works extremely well as it now has like
higher precision as well as low latency
despite it now using the transformer
architecture. And so, if I were to
summarize, how do transformers learn to
see at every scale? Well, transformers
learn to see objects at different scales
by moving beyond vision transformers
single resolution grid of tokens.
Hierarchical transformers like Swin
progressively merge patches to create
multi-resolution feature maps similar to
convolution networks. Using feature
pyramid networks and unit style
architectures, we then combine spatial
details from early stages with semantic
context from deeper stages. And for
real-time systems like RT-DETR, CNNs can
effectively create the multi-scale
hierarchy, while transformers are
applied selectively for global reasoning
and object detection to reduce latency
while maintaining accuracy. And
together, these approaches allow
transformer-based vision systems to
detect and segment objects across
different scales, with different
architectures trading off between
latency and accuracy.
And that's all that I have for today.
Thank you all so much for watching. If
you like this video and you think I
deserve it, please do consider giving
this video a like. Thanks again, and I
will see you in the next one. Bye-bye.