Video summary
In 2015, ResNets established themselves as the standard for deep convolutional neural networks in computer vision, relying on stacked layers with skip connections to ensure that deeper models always outperformed shallower ones. This architecture dominated image-related tasks until the introduction of the Transformer in 2017, which utilized self-attention mechanisms originally designed for language translation. Vision Transformers (ViT) soon adapted this approach by treating images as sequences of patches, and while they initially lagged behind convolutional networks on smaller datasets, they eventually surpassed them at very large scales due to their superior scalability.
To bridge the gap between these two paradigms, researchers developed the CoNeXt family of architectures in 2022, aiming to create a modern convolutional network that incorporates Transformer learnings without sacrificing efficiency. This new design integrates several macro-level changes, such as adjusting the stage ratios to include more blocks and replacing the standard image patching process with a "patchify stem" that uses specific convolution strides to mimic tokenization. Furthermore, the architecture adopts an inverted bottleneck structure similar to Transformers, where dimensions expand before processing and then contract, while also moving computationally expensive depthwise convolutions to earlier layers to optimize floating-point operations.
The CoNeXt architecture further enhances its Transformer-like qualities by utilizing larger kernel sizes, such as 7x7 convolutions, to achieve a global receptive field comparable to the self-attention mechanism, rather than relying on small local kernels. Micro-design adjustments were also implemented, including the replacement of batch normalization with layer normalization and the substitution of ReLU activation functions with GELU. These modifications result in a network block that closely resembles the efficient Swin Transformer while maintaining the structural benefits of convolutional operations, effectively creating a healthy hybrid that performs comparably to top-tier Transformer models on medium-sized datasets.
Although CoNeXt demonstrates remarkable performance by successfully re-architecting convolutional networks with Transformer principles, questions remain regarding its ability to scale to the massive dataset sizes where Transformers currently excel. Future research will need to determine if this hybrid approach can maintain its competitive edge at such large scales, but the current findings suggest that the insights gained from Transformer architectures can indeed be effectively applied to redesign and improve traditional convolutional networks for the modern era.
Read the full video transcript
In 2015, we had the introduction of
ResNets. And these ResNet style
architectures, which are very deep
convolution neural networks with skip
connections, became the de facto for
image related problems in computer
vision. And a lot of these were because
of their stacked convolution,
activation, batch normalization, and
pooling layers. and these skip
connections which basically entailed
that any network is going to be at least
as performant as its shallower
counterpart. So basically this 34 layer
neural network is always going to be
more performant at least as performant
as like a 20 layer network and so that
allowed these you know convolution
networks to go very deep. But around 2
years later in 2017, we had the
introduction of the transformer neural
network. And this architecture worked
mostly on self attention and cross
attention where we had an encoder
decoder architecture. And this
originally was introduced to solve the
problem of language translation but very
quickly spread to different facets of
natural language processing. Now seeing
the success of these transformer
architectures in NLP, vision
transformers tried to take this a step
further to see, hey, what if we use this
transformer architecture to solve
computer vision problems? And so this
could involve trading images as like a
sequence of patches. So you take an
image, cut it up into a bunch of
patches, then shove it into a
transformer architecture and then
perform classification. And
[clears throat] it turned out that this
actually performed quite well. So the
gray region is the performance of
convolution architectures and the
colorful circles are transformers. And
you can see for like small and medium
data set sizes while convolution
networks are kind of more performant
here. It is at these very large scales
where we see even the simple VIT the
vision transformer start to take over.
Now since 2020 there have been many
improvements to the vision transformer
and one such improvement is the swin
transformer that we see over here where
it essentially just tries to make the
multi-headed self attention block a
little bit more efficient and so this
allows different size feature maps or
hierarchy of feature maps to be created.
So this entire architecture just
basically makes it better to detect
objects of varying sizes while also
making the attention operation more
efficient. And this is just like one of
the many breakthroughs that have
continuously pushed the needle into
using transformers in computer vision.
And hence so begged the natural question
of like okay so are convolution networks
now just kind of left in the dust. Well,
this paper was actually a really
interesting paper back in 2022 which
showed that we can actually try to
create a convolution neural network for
this new 2020s era. And the idea here is
we can try to now replace multiple parts
of let's say we take a ResNet
architecture. We can try to use the
learnings of what we know about
transformers in the last like 5 to 10
years. We try to incorporate them here
in order to come up with a new
architecture which is the con next
family of architectures. And this is
actually as performant as like the swin
transformer that we just saw. So the
first thing that we see over here is
that we have this macro design changes
over here. So one of them is stage
ratio. Now if you look at the swin
transformer we see that the ratio of
blocks is 2 is to 2 is to 6 is to 2. And
if we look at our reset architecture
over here you can see that there's like
for stage one we have like six blocks
here maybe seven blocks here there's 12
blocks for the stage three and then
another six blocks for the stage four.
So it's kind of like a one is to one is
to two is to one. But clearly with the
swin transformer there is a lot more of
these center pieces here. So one thing
that we're doing right now is basically
just adding a bunch of more blocks at
this point over here to the ResNet
architecture. Next they have this
patchify stem. And what this basically
means is that in a normal vision
transformer here, the input image is cut
up into a bunch of patches and then
passed into the transformer. And we want
to try to mimic this with convolution
operations. And the way that we do this
with convolutions is by using like 4 + 4
convolutions with a stride of four. So
that effectively will just encode a 4
cross4 region of pixels and then make it
into kind of like a token that we pass
into our convolution neural network. So
we can convolutionify this patching
process. Another thing that we do is we
split up the convolution operation into
two parts like a depthwise convolution
and then a pointwise convolution. And
this is done in order to reduce the
number of floatingpoint operations to
make the entire operation just more
efficient. This idea of splitting a
convolution operation like a standard
convolution into a depthwise convolution
where we just take you know only space
within the same channel we will perform
a convolution and then a pointwise
convolution which is like one cross one
convolutions. This was really
popularized in earlier networks like
mobile net and exception and was used
for you know getting these convolutions
more efficient so that they can be
powerful on like mobile devices and so
we kind of make this convolution far
more efficient and we see a jump in
performance.
Another thing that we do around this is
inverting the bottleneck. So in the
transformer architecture that we see
here, we might have inputs where each
token is like 512 dimensions and the
output here, each vector is also of 512
dimensions. But in the middle, we have
this feed forward network that actually
expands the number of dimensions
momentarily, processes it before
downsampling. And so it creates this
inverted bottleneck. And so we actually
reflect that. You can see in this figure
where we have 96 channels, this also now
becomes 4x to 384 and then it goes back
from 384 back to 96. And so we create an
inverted bottleneck just like we would
see in transformers but only using
convolution operations. Another thing
that we do is now you know because that
this 384 cross 384 we're doing like a
large 3 +3 depthwise convolution. This
generates more floatingpoint operations.
A colonel has to kind of slide through
all of these channels and do this
mapping. And so instead what we do is to
save parameters. We bring this depthwise
convolution to the earlier layer. So
it's performed at this 96 to 96 step.
And so this operation now just becomes
far more efficient because pointwise
convolutions are far easier to play with
when we have such large dimensions.
Next, what we do is play with the large
kernel sizes. So, what's really
important in our standard transformer
neural network is the self attention
piece. With self attention, we have all
of the tokens that can kind of pay
attention to all other tokens. So, there
is this global feature representation
that's being learned. However, in a
convolution operation, it is kind of
more understood that these convolutions
are only very local and whatever
features that are learned are only like
in a local setting. And so to kind of
help solve this, what we can do is
instead of using smaller like 3 + 3
convolutions, we can try to expand this
to let's say 5 + 5 convolutions, 7 + 7,
9 + 9 or 11 + 11 convolutions. And it
turns out the 7 + 7 convolution is the
sweet spot. And so we use it here
because it gives us the best performance
boost.
And then we just make some micro design
changes. This includes, you know, just
tailoring our network to be more like
transformers in general. And so I think
that this block diagram actually shows
the comparison quite well. So for
example, we're using depthwise
convolutions and pointwise convolutions.
We're using, you know, in the depthwise
section, we've, you know, pushed it
towards the front. And we're performing
larger convolutions to increase the
receptive field kind of to make it more
like global just like how attention
mechanism and transformers are. Instead
of the bash normalization, we use layer
normalization. Instead of relu, we use
gel. And we also remove the in you know
the repeated batch normalization
activations in subsequent layers. So
this conext block actually performs more
similar or looks more similar to this
swin transformer block than it does now
to this ResNet. Or at least it's like
some healthy hybrid of the two. And now
making all of these changes to ResNet,
we see that the con next architecture is
now much more performant than ResNet was
and it's even comparable to the Swin
transformer too. One thing to note is
that these are of course on like small
data sets and medium-siz data sets and
what remains to be a fact is that
transformers scale very well. And so
will this convolution architecture
actually hold for much larger data sets?
I think future research still needs to
dictate this. But all of this kind of
does suggest that we can still use the
learnings of a transformer architecture
to kind of rearchitect a convolution
neural network and it still performs
very well. So that's kind of all that I
had today. Thank you all so much for
watching and I will see you in the next
one.