Video summary
The video explains the evolution of convolutional neural networks (CNNs) from simple architectures that merely memorized training data to sophisticated models capable of competing with Transformers. Initially, basic networks without hidden layers achieved perfect accuracy on training sets but failed completely on test data due to overfitting. By introducing hidden layers, activation functions, and pooling operations, the network gained the ability to learn non-linear relationships and extract local features like translation invariance. These foundational concepts led to LeNet-5 in 1998, which significantly improved classification performance but was limited by the lack of data and hardware available at the time, causing CNNs to stagnate until the early 2000s when internet data and GPUs enabled major advancements like AlexNet.
To further improve performance on complex datasets like ImageNet, researchers focused on making networks deeper through two primary strategies: increasing the number of layers or simulating sparse brain-like connections. One approach involved converting large convolutions into sequences of smaller ones, as seen in VGG Net, which increased non-linearities and discriminative power to reach 93% accuracy. Another method, utilized by Inception, stacked dense tensors to allow for deeper architectures while reducing the number of learnable parameters. However, simply adding layers often led to performance degradation because deep networks struggled to learn pass-through functions, causing accuracy to drop as depth increased beyond a certain point.
The breakthrough that resolved this issue was the introduction of skip connections, popularized by ResNet, which allowed information to flow directly through the network and prevented performance loss in very deep models. With these connections, a 34-layer network achieved 95% accuracy, and increasing the depth to 54 layers further improved results to 96%, proving that deeper networks could outperform shallower ones without degradation. This architecture not only stabilized training but also became a critical component used even in modern Transformers for scalability. Ultimately, these iterative improvements transformed CNNs into highly performant systems that effectively challenge the dominance of Transformer models in various computer vision tasks.
Read the full video transcript
Let's talk about how convolutional
neural networks evolve to large and
performant networks that give the
Transformer a run for its money. Hey,
I'm a network with no hidden layers.
Hey there, let's train you.
This is a six, this is a five, this is a
three, da da da.
100% performance.
That's awesome. How are you on this test
set though?
Oh, I'll be great. That's an A, that's a
nine, and then that's an eight.
I'm at 68%.
Hmm, why is this happening?
This is because your architecture is too
simplistic. So, it just memorizes
training data.
Let's now add a hidden layer and train.
This is a two, this is a one, this is a
six, da da.
100% performance.
Nice. Let's now evaluate on the test
set.
I'll get it right this time. That's a
six, that's a one, that's a five, hmm,
no, 78%.
Okay, that looks a lot better. Yay.
But wait, can you train yourself again?
Same network? Same network. Same data?
Same data. But that shouldn't change
anything, right?
Just do it, please. All right, training
now.
100% training, now testing, da da
Oh, now we have 67% performance. That's
weird. That is weird indeed. Let me try
again. Training, da da da da da,
100% now testing, da da da da da da,
oh, it's now 76%.
Looks like I have a high variance in
performance, but why is this the case?
That's because you have a large number
of learnable parameters. There are
multiple network parameters that lead to
a perfect training performance.
Different networks means different
predictions, which means different
performances.
How do I deal with this?
Well, we'll have to modify you with
three concepts. The first one is
convolution. Remember, the input is an
image. Take advantage of this. Images
have pixels that correlate to pixels
around it. The convolution operation is
great at extracting local features.
Also, a seven here is the same as a
seven here or here. This is translation
invariance, and the convolution
operation enables this.
The second concept is activation
functions. They add non-linearity after
each neuron operation, and they enable
the network to learn non-linear
relationships between the inputs and the
outputs. It's good for handling
real-world images. Now, the third
concept is pooling or subsampling.
Pooling is the downsampling of inputs
summarizing their effects, and this
helps drastically reduce the number of
learnable parameters.
And so, using these concepts together,
we are now able to revamp our network.
This is LeNet-5, introduced in 1998.
It had these convolution, activation,
and pooling layers along with fully
connected layers to perform
classification, and it was trained with
backpropagation.
All right, network, how do you work now?
Uh let me test. Let's classify this.
That.
I'm at 98%.
That That is amazing.
Now, can you tell me what this is?
No?
Okay, that's okay. What about this one?
Ah, no again?
Is that a me problem? No, it's not you.
It's just that the '90s doesn't have
enough data nor hardware to make
training feasible.
So, convolutional neural networks
actually died for a while around this
time. In the early 2000s, we had two
main changes. There was more data on the
internet, which could be used for
training models, and there was also the
rise of GPUs, which could help models
process this data. And these helped
modify the LeNet architecture to give us
AlexNet in 2012.
I am so back. Yes, you are. How do you
perform on ImageNet data now? Ooh,
trickier data. Let me see.
That's a squirrel monkey. That's a
cherry. Da da da hmm hmm
I'm at 65%
accuracy.
All right, that's not bad at all,
especially for these like really tough
images.
Nice. So, how do I do better?
Well, well, you can do better by getting
deeper. And there are two ways to become
deeper.
For one, you could convert your large
convolutions into a sequence of three
cross three convolutions. And this is
what VGG net did.
I feel awesome.
Glad. So, how are you performing on
ImageNet?
I'm at 93%
Ah Thought so. Increasing layers
increases non-linearities, which
increases the discriminative power of
the network. So, better performance.
There is a second way to go deeper, too.
The network can simulate sparse
connections like we have in the brain.
And this is what Inception did.
So, how exactly do I do that? You create
multiple dense tensors from
convolutional operations, then stack
them. The dense tensors allow us to make
use of GPUs, but different stacks don't
mix to simulate the sparse connectivity.
This reduces learnable parameters while
making the network deeper. So,
performance should improve.
Hmm, let me check actually.
Oh, I'm at 93%.
Yep, that's amazing. This is great. Now,
let me try to get deeper.
I bet my performance is so good now. Let
me check.
Classify, classify.
Ooh, looks like my performance is 85%
now. Why am I worse?
This is performance degradation. It
happens because it is hard for a deep
network to learn a pass-through
function, and we can remedy this using
skip connections. This is what ResNet
did. All righty, I'm 34 layers, so let
me try now.
Classify, classify, classify. Looks like
my performance is 95%. Woohoo!
Let's increase this to now 54 layers.
Classify, classify.
Ooh, performance is now 96%.
With skip connections, a deeper network
is as performant as its shallower
counterpart. This critical component is
used even in transformers today to make
them scalable. These changes created
modern convolutional neural networks
that even gives transformers a run for
their money.
And that's all that we got for this
video. If you think I deserve it, please
do consider giving this video a like.
Subscribe for more, and I'll see you in
the next one. Bye-bye.