Video summary
The video introduces KubeElasti, an open-source tool developed by True Foundry and recently accepted into the CNCF sandbox to address a critical inefficiency in Kubernetes clusters: massive resource underutilization. The speaker highlights that many organizations pay for significant computing capacity while their actual CPU usage rarely exceeds 10-15%, often hovering below 2%. To solve this, KubeElasti implements a "scale-to-zero" capability similar to serverless platforms but without the high barriers to adoption found in existing solutions like Knative or OpenFaaS. Unlike those alternatives which require complex migrations or introduce persistent latency through always-on proxies, KubeElasti allows teams to run standard Kubernetes deployments with zero code changes, ensuring that services only consume resources when traffic is actually present.
The core functionality of the tool relies on a unique architecture consisting of two main components: an Elastic resolver and a controller. When no traffic reaches a service, the system scales it down to zero replicas to save costs. If incoming requests arrive while the pods are inactive, the Elastic resolver intercepts them, queues the data in memory, and signals the controller to spin up new pod instances immediately. Once these pods become ready, the queued requests are forwarded successfully with no downtime for the end user. Crucially, after handling a batch of traffic, the proxy steps out of the request path entirely, meaning that subsequent legitimate traffic flows directly between the service and the pods without any additional latency or reliability concerns beyond the initial cold start period.
Practical applications demonstrate significant cost savings, particularly in development environments where multiple preview services for pull requests often sit idle waiting for review. By scaling these unused previews to zero, organizations can eliminate costs associated with running five or more unnecessary services simultaneously while still maintaining instant availability when a specific PR is actively being reviewed. The speaker notes that even though there are limitations regarding GPU workloads due to longer node provisioning times and on-premises clusters lacking dynamic infrastructure flexibility, the tool remains highly effective for CPU-based workloads in cloud-native settings where traffic patterns fluctuate or services have low baseline demand.
In conclusion, KubeElasti offers a pragmatic middle ground between traditional auto-scaling and full serverless adoption by focusing specifically on scaling from zero to one replica rather than managing large clusters of idle nodes. While vertical scaling for memory optimization is acknowledged as a separate challenge since even single pods consume substantial RAM, the horizontal ability to drop resources entirely provides superior cost efficiency compared to keeping minimal replicas running 24/7. The project invites community contributions and aims to provide enterprises with a flexible solution that integrates seamlessly with existing tools like HPA or KEDA, allowing them to reclaim wasted cloud spend without compromising on service availability for low-traffic applications.
Read the full video transcript
Yeah.
Okay, hi. I'm Shubham. I am from True
Foundry. I have been
working in this field of cloud native
engineering for the last 6 years. In
total, I've worked for more than 9 years
now.
Yeah.
Yeah, so that's basically about me.
Before True Foundry, I was working in
Gojek for around 2 years. Uh right now,
so True Foundry is a
ML platform. Uh
right now, we also build AI gateways,
but that's a different thing.
Uh
so, Kubelastis is basically a tool that
we have built. It's
louder.
KubeLastis is an open source tool that
we have released uh for the community.
It has been selected within CNCF sandbox
recently.
Uh
I'll go through what that tool is, what
that tool does, and the overall agenda.
So, the agenda is basically what the
problem was, what we were trying to
solve, how we arrived at the problem
statement of scale to zero,
how we decided to solve it, what other
options are available, and then we'll go
through a demo.
Okay. So, the problem is actually very
simple. Uh this is typically the usage
of a typical cluster a Kubernetes
cluster that we encounter. So, it might
not be visible. The usage is around 2.6,
and the allocatable is around 167.
So, this is less than uh 1.5% of
utilization of what you're actually
paying for.
So, this is a massive problem in the
industry.
Uh like maximum people hit like 10 to
15% in many cases
uh of actual CPU utilization. You can
see some spikes over there, but those
are very few.
So, uh this is the problem that we uh
were trying to tackle.
So, there are two halves to this
problem. One is
how do you actually right size the
workload? How do you actually right size
the pods that you're running? And second
is how many pods are you running?
So, we decided to tackle the second
question first,
which is basically can we actually
switch off services which are actually
not receiving any traffic at all?
So, that's scale to zero.
Scale to zero is similar to any
serverless platform that you might have
heard of, and we wanted to implement
that in
uh any Kubernetes cluster without using
a lot of external dependencies.
So, these were our requirements. We want
to scale to zero on zero traffic.
We want to resume service if traffic has
come back.
We are not okay to accept any dropped
traffic.
We want to use native Kubernetes
resources only, and we don't want any
additional latency. We also wanted it so
that we don't have to rely on a
particular cloud provider or an ingress
provider.
So, for example, by ingress provider,
what I mean is Nginx, Istio, Kong,
anything. All of that should work.
Right. So, it started with a search. So,
these are the few options that we found
in open source. So, that is Knative,
Keda HTTP Add-on, and OpenFaaS.
In case of Knative and OpenFaaS, the
problem was the barrier to adoption was
really really high. Because in case of
Knative, you have to actually uh migrate
yourself completely to a different to uh
their own CRD, their own Knative
service.
In case of OpenFaaS, you have to model
everything as a function.
And in case of And there's a project
under Keda. Keda is
event-driven auto scaling in uh
Kubernetes' context. Uh
they have a project called HTTP Add-on,
but the problem with that
uh project is that it adds a proxy which
remains in the hot path
which remains in the request path for
the complete time. So, it adds external
latency, additional latency,
and some reliability uh hit.
So, yeah. So, basically, we set about
building Kubelessity.
And
so, these are the things that
Kubelessity does. It scales down to zero
when idle, scales to one on traffic. It
uses a smart proxy that steps out after
scale up.
It integrates with HPA or Keda. So, if
you're already using HPA or Keda, you
can basically it hands over the
uh part of one to end scaling to these
other tools?
It works with an existing plain
deployment plus service setup. It does
not need any kind of change to most of
the Kubernetes uh
setups out there. It has no barrier to
adoption, and it is open source.
Right. So, how it works So, how it works
is actually very simple.
If you don't have KubeElastic, you have
a service which is uh forwarding traffic
to pods. Traffic comes in, request fails
because the pod is not there. If you
have If you have actually scaled down to
zero.
So,
how it How it is solved in some of the
cases is you have uh
the traffic to service is actually
metered. It's actually like you get
those metrics, and you scale it up, but
still like the initial few requests are
going to fail.
In case of Elastic, what we do is
Elastic proxy basically injects itself
in this path, and any traffic coming to
the service actually goes to Elastic
proxy, and it queues those requests. It
queues those requests, and on the side
it uh
uh brings up pods for that deployment.
So, once the pods have come up, it will
basically forward that request so that
that request ends up getting served. So,
you get zero downtime as a result of
this.
How this is actually built is built is
it has two components, a resolver and a
controller. I hope that is visible.
Within the yellow box, there is a
resolver and a controller.
So, the resolver is basically the proxy
that holds the request in memory, and
controller is the one that takes care of
actually scaling the replicas as needed.
Uh not going into architecture too much,
but broadly
there are two modes to it. There is the
serve mode and proxy mode. Serve mode is
what you see right here.
A request comes in, hits the service.
We are in serve mode, it goes to pods.
Our Elastic is running out there. It is
talking to Prometheus, basically trying
to figure out Should I scale this
service down to zero or not?
It decides that this service can be
scaled down to zero since there is zero
traffic. It scales it down to zero, and
what happens is Elastic resolver
actually injects itself in the path of
that service so that all
any further traffic if it comes in, it
actually hits the resolver.
Right. So, when it when traffic actually
comes in, Elastic Resolver gets it, it
queues it, it informs Elastic Controller
that the pods need to be brought up, it
brings up that pod, and then resolver
resolver forwards that traffic to the
pod, and then it takes itself out of the
path. So, any future request actually
goes directly from service to the pod
without having resolver in between. So,
there's no additional latency or
reliability concern beyond the initial
few requests.
Right.
So, this is one of the use cases that we
have found to be where this was
extremely useful. So, this is a dev
environment. In dev environment,
basically what we had is
we have two services, three services in
fact. We have a front-end service and
two back-end services.
For front-end services, we wanted a PR
preview to be available for every PR
that is raised on that repo.
So, we have three different PRs raised.
In normal condition, what would happen
is if you want a PR preview to be
available for every PR, this is
something that you would have used
external
tools for within your own companies, but
this is something that we built
in-house.
So, you have three different PR previews
available to it, and all those PR
previews have to be up
for this to work.
So, without KubeElastic, there's no
traffic in the system, no PRs are
undergoing review, and we still have
five services
being up.
The traffic is flowing through front-end
PR one, but still like we are continuing
with five
services. So, the obvious saving over
here is none of this has to be up
if the if there is no traffic flowing in
the system.
If traffic does flow through the system,
that is the one that needs to get
activated, the other front-end PR
services can remain at zero replicas.
So, if you just calculate through this
cost, what it basically comes down to is
let's say 20 open PRs, two PRs actively
under review, and each PR taking 0.5
VCPU and 1 GI memory. So, that comes to
around 315 US dollar per month per
service.
Right.
So, coming to how it actually how it is
actually used. So, this is a typical
Elasti service that you would use to
get Elasti intercept get Elasti to
intercept your requests.
So, this is an Elasti service. You can
see three parts to this that is actually
an input. So, there's the service which
is target service. So, this is the
service that you want to intercept.
Then, there's the scale target ref which
is a target deployment. So, this is the
part that you actually want to scale.
This is similar to how HPA is designed.
And then there is triggers. So, in
triggers basically what we have is what
is the query that Elasti needs to make
in order to figure out
whether it needs to scale down or scale
up the requests.
Right. So, this is a small demo. Two
minutes. So, I'll go through this. The
guy demoing over here is Raman. He has
written most of the code in this.
Yeah.
Okay, this might not be
Is this audible?
>> So limitations wise, you can
like basically the the limitations are
similar to any serverless platform.
Anything with high cold start time, like
GPU workloads.
They do suffer from a large time of time
for provisioning because of cloud
provider delays.
So that is one case where this will not
work.
Any setup where the traffic does not go
through a Kubernetes service because we
intercept Kubernetes services directly.
There are some cases where load
balancers are performing health check on
services. So that is something which we
are in the process of figuring out.
And anywhere where the infrastructure
itself is not flexible. So if you have
an on-prem Kubernetes cluster where
nodes are not provisioned dynamically
like in carpenter.
That is where you will face some problem
with
any serverless platform. Like basically
you won't realize any cost gains over
there.
Yeah, so that's basically about
Kubelessity.
This QR code will take you to the
website for Kubelessity.
We are actively looking for
contributors, maintainers, anyone who
could get connected. So yeah, please
feel free to jump in.
Any questions?
Yes.
So
the question is like can this actually
be run on production because this is
this looks like something that would be
good for dev because there is low
traffic, right?
Yes.
On a low traffic, low reliability
requirement servers.
But in case of prod basically
if you have a service which has low
traffic or even if
it's similar to what
the previous like what we talked about
in spot, if the time for a service to
come up is really small, then it
actually it's actually fine. If that
service is going down to zero traffic,
then it's fine. You can like
run that service behind Kubelessity. It
will come up in let's say 30 seconds.
That's fine if you can tolerate that.
Yeah.
Right.
That is true.
We have not encountered such a
situation. That can obviously happen.
But yeah, the way to solve it would be
to to vertically scale the resolver. We
like that is horizontally scalable,
completely stateless system. Like you
can solve it using scaling.
Right?
>> Right?
Order of requests coming in
would not be preserved, I think. Yeah.
Because all of that is in separate go
routines, which have their own uh setup.
Yeah.
Uh
I mean, this uh basically makes the
resolver uh like stateful to some
extent. So, yeah, that could be a good
idea. We could write to disk and then
basically load it up from there. Uh like
we could make it stateless, but then
like we lose the
uh HPA like basically uh being able to
horizontally scale it to infinite.
So, yeah, there's a trade-off there.
Yeah, you had a question.
Yes.
Right.
Yeah.
Yeah, in that case uh basically like if
you if if you if the request like the
token that is coming with the request,
if it is that close to the expiration
window, then there's always a chance
that it goes over.
Yes.
By the way, the question was what if uh
the request had uh an any authentication
information which might end up getting
expired
uh
by the time the pod pod actually comes
up.
Right.
Right. So, uh zero to one is something
that uh
So, zero is by definition low traffic.
So, if you So, zero to one is what
Kubelast handles. Everything after one
to n is handled by other uh auto
scalers. So, you can have you HPA, you
can use KEDA.
So,
yeah.
Right.
So, uh, elastic service can be
configured in that way.
So,
if I can show that.
Yes, so this meant target replicas. You
could start with a large number of
replicas.
If that would help.
Yeah.
Yes.
Like the middle So, the question is why
go to zero? Is there no middle ground?
So,
the middle ground is basically making
your requests smaller.
Right.
So, that is something that VPA does in
case of,
uh, Kubernetes.
But,
at the end of the day, like this is,
this is a problem that we face because a
lot of people had a lot of services
running just in case it receives some
kind of traffic.
So, that is the case that we started it
with, but,
uh, what you said we can like you can
save a lot by actually sizing it very
very low. Sizing a particular pod. But,
even in that case, a single pod can take
up a lot of resources.
So, yeah.
Like zero is better than one, always,
yeah.
Uh, the vertical auto scaling part.
That's a separate project. Uh, we have
some plans around that.
Okay, if that's it.
Thank you. Uh,
he has a question.
Yeah.
Right.
Yeah.
Right. So, the question is why not make
it sort of like a window? Like in in a
certain kind of in certain window I want
more resources in a certain window I
want less resources. Is that
Right.
Yeah.
You could do that, but then like again,
you could do that but zero.
Zero is less than anything like that.
Like if you can get to zero, you're
better get to zero. Otherwise, yeah.
Obviously, like you can do some other
things around it.
Another point there is if you actually
like memory is not something that can
easily be reduced a lot. Even a single
replica, even with very low traffic,
you're not going to use up a lot a lot
less memory. CPU it is true for CPU, but
not for memory.
So, you will end up paying for memory in
most cases.
Yeah.
So, how much is the how much time does
it take to go from zero to one? So, it
depends on the actual pod. We wait for
the readiness probe for of the pod to
pass. If the readiness probe has passed,
we forward traffic. It can be as quick
as 1 second. Like some
like less than 1 second, but it depends
on how long does it take for the node to
come up. How like if you have flexible
infrastructure and how long does it take
for the pod to actually get scheduled
and start running and be ready? So,
that's why I mentioned one of the in one
of the limitations GPU with GPU
workloads, it does take more time
because the node provisioning actually
takes more time. But if you can uh
if you can tolerate that initial latency
spike, then you can save a lot by just
uh scaling GPUs uh GPU workloads down to
zero.
Yeah.
Sure.
Right.
Right. I mean, so uh
for Deliveroo, that might be true. You
guys get a lot of traffic.
But yeah, for
Yeah, yeah, for low traffic scenarios,
services with low traffic, you can use
it. If the number of services is large,
you will get more savings.
Thank you.