Video summary
The video explores the historical shift in computer vision from the dominance of Convolutional Neural Networks (CNNs) to the rise of Transformer architectures, questioning whether the latter has rendered the former obsolete. The narrative begins with the 2012 revival of CNNs via AlexNet, which enabled machines to solve complex visual tasks using learned features rather than handcrafted ones. While various CNN architectures flourished throughout the 2010s, the introduction of Transformers around 2017 initially revolutionized Natural Language Processing through a pre-training and fine-tuning paradigm. This approach allowed models to scale effectively with increased compute power, data, and parameters, leading to Vision Transformers (ViTs) that could process images by treating them as sequences of patches. Consequently, for large-scale datasets containing hundreds of millions of examples, these attention-based models began to outperform traditional CNNs, marking a significant transition in the field.
However, the central argument presented is that CNNs have not disappeared but rather evolved to address specific strengths that Transformers lack, particularly regarding efficiency and real-time performance. The video illustrates how researchers are now integrating Transformer learnings into CNN designs, such as using large kernel sizes or patchification techniques within convolutional layers to mimic global self-attention without sacrificing the inductive bias of local pixel relationships. A prime example of this evolution is ConvNeXt, which demonstrates that CNNs can absorb modern design ideas from the Transformer era while maintaining their structural advantages. This hybrid approach suggests that rather than killing CNNs, Transformers have forced them to adapt and incorporate new methodologies to remain competitive in large-scale vision tasks.
Beyond large-scale applications, CNNs remain indispensable for scenarios where computational efficiency is paramount, such as on mobile devices and edge hardware. The video highlights MobileNet architectures, which utilize depth-wise and point-wise convolutions to drastically reduce the number of learnable parameters, making them ideal for running models directly within apps on limited resources. Even a decade after their introduction, these networks continue to be refined and developed. Furthermore, in the realm of real-time object detection, the YOLO series remains the state-of-the-art, with recent versions like YOLOv6 relying entirely on fully convolutional backbones that outperform transformer-based variants in speed and practical deployment. These examples underscore that while Transformers have won the race for scaling with massive datasets, CNNs are still winning the efficiency race in many critical applications.
In conclusion, the video answers the titular question by asserting that Transformers did not kill CNNs but instead compelled them to evolve into more sophisticated forms. The current landscape features a division of labor where Transformer architectures dominate large-scale vision tasks due to their superior scaling properties, while CNNs persist and thrive in domains requiring low latency, small model footprints, and real-time processing. The future of computer vision appears to be a convergence of both technologies, with hybrid models leveraging the global context awareness of Transformers alongside the localized efficiency of convolutions. Ultimately, the rise of foundation models and pre-training has not rendered CNNs obsolete; instead, it has created a nuanced ecosystem where each architecture plays a vital role depending on the specific constraints and goals of the task at hand.
Read the full video transcript
Did Transformers kill convolution neural
networks? In 2012, we had the start of
the revival of convolution neural
networks with the creation of AlexNet.
This marked the point of transitioning
from handcrafted features to actually
using convolution neural networks to
solve more practical and complex
computer vision problems. And then from
2013 and through the 2010s itself, we
had a slew of convolution neural network
architectures that solved different
facets of computer vision. They were
trained on individual tasks. Around like
the 2017 time though, we had the
creation of transformer neural networks.
This network was an attention-based
network and was originally created to
solve the problem of language
translation. But within the very next
year, we had the popularization
of a pre-training and fine-tuning
paradigm where we would now pre-train
these transformer models on a broad task
like language modeling and then we can
fine-tune it on any number of downstream
related tasks. And because of this, on
almost like all NLP tasks, we saw this
pre-trained fine-tuned transformer
architecture just dominate across the
board. And through the end of the
decade, the scaling of these
transformers was understood more and
more for NLP tasks. And the idea here is
that if we increase compute, the data
set size, and the number of parameters
in the network, we would see a
continuous decrease in the test loss for
these transformers. This showed that
transformers scaled very well for NLP
tasks. And then around 2020, seeing the
success of these Transformer
architectures in natural language
processing, we have these vision
transformers that say, "What if we just
take an image, cut it up into a sequence
of patches, pass it through the exact
same Transformer architecture, and then
use this for a downstream, you know,
case like classification?"
And it turns out it actually works
pretty well. For small data sets, they
are quite comparable to convolution
networks, but as the data sets become
much larger, like 300 million training
examples, you then start seeing vision
transformers just outperform convolution
neural networks. And so we went from an
era of convolution networks to solve
most computer vision problems to now
these attention-based transformer
networks to solve these problems. And
also because of this idea of
pre-training and then fine-tuning,
potentially, we have some of these
networks which are trained on more
general-purpose
problems, but can be used out of the box
as is as for downstream tasks. And these
became known as foundation models. But
the main question here is now, did
convolution neural networks now
completely go away?
And the answer here is actually that's
not the case. And convolution neural
networks are largely still there for
cases where we care about efficiency. So
with the rise of the transformers in the
2020s, there was now kind of a thought
of, "Okay, can we try to just take a
convolution neural network and then
adjust it to actually make use of the
learnings of transformers and see how it
performs?" And what I mean by this is we
take like for example a ResNet
architecture and we replace it with some
transformer base learnings. For example,
we use a patchification of the image
because you take an image split it up
into patches to pass it into a
transformer. We can mimic this operation
with convolution operations.
The same thing goes for like large
kernel sizes here. One of the good
things about transformers is their
global self-attention.
Whereas convolution operations because
of their inductive bias to images look
in local regions because images tend to
have pixels that are related to other
pixels around it.
And so we try to experiment with
increasing the kernel size to better
mimic this global self-attention. And
then we also make some design changes
here, small ones here, in order to make
the convolution network quite literally
look like a transformer neural network.
And it looks like pound for pound or
parameter for parameter, this
convolution neural network can still
have an edge. The second example that I
want to walk through is for real-time
object detection using YOLO, you only
look once. This was originally
introduced as a fully convolution neural
network architecture where you pass in
an image and then we will get this
tensor as a response. And from this
tensor we can generate bounding boxes
around objects of interest.
And this was all back in like the
2015-2016
era. Fast forward nearly a decade, we
are still creating versions of YOLO that
are the most performant today. For
example, this is the architecture of
YOLO 26 and it is fully convolutional
where we have a backbone which takes an
image and converts it to a tensor. Then
we have this tensor passed through a
neck which is largely responsible for
ensuring that objects of different sizes
can be appropriately
detected. And then we have multiple
detection heads, one for larger objects,
one for medium-size objects, and one for
smaller objects, all purely
convolutional. And the results here are
actually the state of the art, beating
out other transformer-based variants. A
third place where convolution networks
are still very much prevalent is on
mobile nets. For mobile devices, we need
very small models to be run on them
because the model has to be run within
the app itself. And convolutional neural
networks provides a great way to do
this. The standard convolution
operation, we can split it into two
parts. One is a depth-wise convolution,
and then we can do a point-wise
convolution. And this can dramatically
decrease the number of learnable
parameters, and hence the size of the
model in these mobile applications. This
one was introduced originally in like
2016, and even 10 years later here, we
still have developments that are being
made. And these new mobile nets are
actually pretty cool because they can be
fully convolutional, or they can be
hybrids of convolution and
transformer-based architectures, taking
advantage of the best of both worlds.
And so, did transformers kill
convolutional neural networks? Not
quite, because they just forced
convolution networks to evolve.
Transformers became the architecture of
choice for large-scale vision because
they scale well with data, compute, and
pre-training. But convolution networks
did not disappear. Transformer won the
scaling race, but CNNs may very well win
the efficiency race in many cases.
For example, ConvNeXt shows convolution
networks can absorb transformer era
design ideas. MobileNet shows
convolutions are still essential for
mobile and edge hardware. And then YOLO
26 shows that CNN style architectures
still matter for real-time deployment.
So that's all I have for today. Thank
you all so much for watching. I'm going
to leave a link to all the papers that I
just like whizzed through down in the
description below so that you can go
through it in more detail. And thank you
all once again and I will see you in the
next one. Bye-bye.