Video summary
Image embeddings are vector representations that encode the semantic meaning of an input image, allowing models to group similar concepts closely together while separating dissimilar ones in a high-dimensional space. The importance of these vectors became particularly pronounced after 2017 with the advent of transformers and the rise of foundation models like CLIP and DINO V2. Unlike earlier convolutional neural networks trained on narrow objectives such as specific image classification tasks, which produced embeddings that were difficult to reuse for other purposes, modern foundation models are pre-trained on vast datasets using broad objectives. This shift enabled the creation of highly reusable embeddings that can effectively serve various downstream applications beyond their original training scope.
The video illustrates how CLIP utilizes a dual-encoder architecture consisting of both text and image encoders to align these two modalities within a shared vector space. By pushing together vectors representing images and texts with matching meanings while separating those without alignment, CLIP creates an embedding space ideal for tasks like visual search on e-commerce platforms. In this scenario, product catalog images are converted into embeddings stored in a database; when a user searches using text such as "leather boots," the system generates a corresponding text embedding to find the nearest neighbors in the vector space, effectively retrieving relevant products without needing explicit image tags or categories.
DINO V2 offers another approach by focusing purely on visual understanding through a student-teacher framework that analyzes different crops of an image to recognize holistic parts like ears or noses as components of a whole object. This method employs specific loss functions and regularizers to ensure embeddings maintain healthy distances from one another, making them suitable for foundation model applications in visual search. When applied similarly to CLIP, DINO V2 allows users to upload an image query rather than text; the system generates an embedding for this query and identifies visually similar items as top results, demonstrating how these models facilitate advanced retrieval systems based on content similarity rather than keyword matching.
In summary, image embeddings have become central to modern artificial intelligence by transforming visual data into reusable mathematical representations that power diverse functions such as clustering, classification, recommendation, and multimodal AI tasks. The transition from specialized task-specific networks in the 2010s to general-purpose foundation models in the 2020s marked a pivotal evolution where passing an image through a pre-trained model yields versatile vectors applicable across numerous domains. Ultimately, these embeddings bridge the gap between raw visual input and complex computational tasks, enabling systems like visual search engines to understand and process images with human-like semantic comprehension.
Read the full video transcript
Greetings fellow learners. In this
video, we are going to talk about image
embeddings. The idea here is we have a
model that can take an image and it can
output a vector. And this vector is
going to somehow encode the meaning of
this image. And if it does, then this
vector becomes known as an embedding.
Embeddings that represent very similar
concepts are grouped somehow near to
each other and those that are very
different in concept are a little bit
further apart. Why are image embeddings
important and why do we even have them
to begin with? So, in 2012, AlexNet was
introduced to start the revival of
convolution neural networks. And since
then, from like 2013 onwards, we saw a
slew of convolution neural networks
taking over different domains in
computer vision. Now, these models could
technically produce embeddings, but
their narrow training objective makes
them less reusable. And to further kind
of expound on that, we have AlexNet from
2012 where we have an image passed
through a convolution network and it's
trained on image classification.
Now, we could take this layer with 4,096
dimensions as a vector that's called an
embedding that represents this image.
While this is okay, the problem here is
that like this network was trained on
this narrow objective of image
classification and so the vector that is
encoded really doesn't have a great
representation of this image that can be
reused across different tasks outside of
image classification. And so, that's
kind of the issue here. They don't
really generate reusable embeddings.
Now, in 2017, we had the creation of
transformers. And since then, as they
were originally introduced in natural
language, but eventually made their way
into computer vision. In the 2020s, we
saw a slew of transformer neural
networks kind of taking over all parts
of computer vision. And what we also saw
that was very interesting was that this
pre-training/fine-tuning
paradigm became popularized. We even had
models like CLIP and DINO V2 that were
trained on broader objectives on much
larger data sets. And so the embeddings
that they created became much more
reusable across downstream tasks. And
these networks eventually became known
as foundation models.
And so to add some more clarity here,
what we mean is that if we were to take
an image, we pass it through a
foundation model, we will get a vector.
And this vector, we'll call it
embedding. And the cool thing is that
this embedding is actually quite
reusable across different tasks in
computer vision outside of what the
foundation model was trained on. Let's
actually see how they're created and
used with the two foundation models that
we just talked about, CLIP and DINO V2.
So CLIP contains two main components, a
text encoder and an image encoder. The
text encoder will take some English text
and it will translate that into a vector
representation, which is an text
embedding.
Then the image encoder is going to do
the same for images where it creates an
image embedding. We want these two, if
they have the same meaning, to be pushed
together as close to each other as
possible. And then those that disagree,
the text that doesn't align with the
images, we want them to be pushed away
from each other as much as possible. And
when you do this, it actually creates an
embedding space with these vector
representations that can be reused
effectively on other tasks. Now one such
task that it can be reused for is text
to image search. So, let's say now that
we have an e-commerce platform where we
have a bunch of products in our store
and we want to create search for it. So,
let's say we have this pre-trained clip.
We'll pass all of our product catalog
images to clip to generate embeddings.
Now, all of these embeddings can thus
exist in some vector database. Now, what
we can do is whenever a user types some
search, let's say they search for
leather boots, we pass it to clip,
generate a text embedding for clip, and
then what we can do is do a nearest
neighbor look up within this embedding
space. And what we get is leather boots
that are shown.
So, this kind of shows how clip itself
can be used for all kinds of
applications where images and text kind
of need to align with each other. The
next thing I want to talk about is
DINOv2. DINO is kind of just going to be
working with images with different
crops. So, if X is an image, X1 is like
a set of images where we're just
cropping different parts of the image
that we pass into a vision transformer
student.
And then X2 is going to be much larger
like parts of the dog where you can kind
of see most of the image and it's passed
to the teacher. And both of these are
the exact same network and the idea here
is that we want the student when even
though it looks at just like the ear of
the dog or the nose of the dog, it
should still recognize that as a part of
the holistic dog. And that's what is
within this loss. It tries to make sure
that whatever the student sees, it is
matching the teacher as much as
possible. And then there's also this
regularizer over here. And this is going
to ensure that we whenever we generate
embeddings with DINO, that they are at a
healthy distance from each other and
this allows DINO to be used as a
foundation model and those embeddings to
be hence reused for downstream
applications. Let's now talk about one
of those downstream applications, which
is visual search. So, first is the same
that we did with CLIP. We take DINOv2
and embed all of our images in our
catalog. And then, when we have some
image now that comes in for searching,
let's say that this is now passed into
DINOv2.
We create an embedding for it. And now,
what we can do is do a visual, you know,
similarity here. We can now look for
nearest neighbors for this. The nearest
neighbor is like the boot, and this
could appear as like a top results of
search.
So, you can kind of see here how both
CLIP and DINO can thus be used as
foundation models in downstream tasks.
And at the heart of it all are these
embeddings.
So, if I were to now like summarize what
we've just talked about in this video,
image embeddings are vectors that
represent the meaning of an image. They
became especially important in the 2020s
as vision models shifted from
task-specific systems to more
general-purpose reusable foundation
models. And in this new paradigm, we can
pass an image through a pre-trained
model,
convert it into an embedding, and then
use that embedding for downstream tasks
like search, clustering, classification,
retrieval, or recommendation.
And that is why image embeddings are
central to modern visual search,
retrieval, and multimodal artificial
intelligence.
So, I hope this video makes it clear of
what, why, and how of image embeddings.
And if you like this video, please do
consider giving it a like, subscribe for
more awesome videos, and I will see you
in the next one. Bye-bye.