How Confluent deploys 1,000 microservices across 5,000 Kubernetes clusters

Okay. Hi. Um hi everyone. Uh thank you for joining me today in the talk about scaling microservices with Halyard at Confluent. Uh yeah, before I get started Sorry. You getting something? Okay, yeah. Uh yeah, I'll introduce myself here. Sorry. Louder? Okay. Uh I'm Mustajib. I'm a senior software developer here at Confluent. And today I'll be talking about scaling microservices with Halyard at Confluent. And before we get started, uh I would like to know how many of you here have heard about Confluent? Uh can you can I get a raise of hands? Okay, that's a few people. How many of you have heard about uh Apache Kafka? Okay, that is a few more. Uh how many of you have heard about Apache Flink? Okay, that is a little less number. Okay, uh for those of you who don't know about Confluent it's a data streaming platform that empowers organizations to make real-time decisions by using uh using data streams. Right? And uh Confluent provides uh Apache Flink and Apache Kafka-based solutions which help uh con- help companies take AI-driven decisions with uh real-time data. I mean, AI requires real-time data for it to take the best decision, right? So, uh Confluent comes in handy there. And you also have event-driven architectures which come which happen because of this. And we have shift-left data which comes into picture because as uh which basically means that you move to the closer to the source and process the data there as much as possible so that the downstream systems end up having the data in the right format and they work as expected. Uh yeah, and many other things that Confluent helps you Uh we discussed about the the products that Confluent has to offer Apache Flink, Apache Kafka and there are many more like table flow and then a few more that it does connect and things like that. So in order to make all of these happen Confluent has like two offerings that it has. One is the cloud based offering and one is the on-premise based offering. For the purpose of this talk we will be limiting it to the cloud based offering that Confluent has to offer and the deployments that go through in in the cloud based offering. Yeah and we are deployed across like across all the public clouds that are there. So we have like 90 plus regions where we are deployed and this basically helps us meet the customer where they are and and fit their needs when required. Yeah there is another distinction in the deployments that we have. We have like we have services which get deployed in the control plane and we also have services which get deployed in the data plane. Now most of the services that get deployed in the control plane are helm based services like most of them. But in data plane we also have helm as well as non-helm based services and we and this deployment platform that we're going to talk about today takes care of both of them and we'll look into it how. Now we discussed about the landscape but what is the scale that we handle at? Right? So to talk about the scale Confluent has like 1,400 microservices running. Trust me these are needed but not that we have made up these services or just doing it like that. Yeah and we have we have around 5,000 clusters which are present and all these all these services are running on on a bunch of these clusters and given this given we have 5,000 more than 5,000 service clusters, and we have these many microservices present, you might expect that the peak load that we see is a multiple of these two. Like, that's the combination, right? But that's not actually what we see. It The peak load that we see is around 10 ^ 5 per day. Because not all all services that get deployed, they're not deployed to all clusters. They might be limited to just the control plane or just the data plane. And even in data plane, they might just be limited to a certain set, like maybe Azure clusters or maybe GCP clusters or things like that. Right? And uh uh yeah. Uh before we get into the crux of the uh crux of the uh platform that I'll be discussing, I also wanted to talk about what uh are we trying to achieve? So, basically, continuous deployment, right? And what does it mean? I mean, what Most of you might know about it, but just to jog your memory, uh it is a practice where you automatically validated code uh change gets directly sent to the customers uh without the without the involvement of developers a lot. I mean, it's a dream scenario, right? Well, customers where developer just clicks a button, and then uh the code change gets validated, it it makes sure that nothing is breaking, and then after that, it gets promoted to the next environment, and and things like that. And and ultimately, the customer receives the uh receives the code and the feature which had to be released gets released as easily as possible. Now, uh introducing our unified deployment platform or Halyard, which uh which we used to call it previously, uh which solved the uh solved the deployment problem at Halyard at Confluent. Uh so, Con- uh Unified Deployment Platform is a fully integrated, uh continuous deployment platform. Like I mentioned, it was present across We need to be present across environments and need to make sure that everything all the customer workloads are working as expected. And we need to manage both Helm as well as non-Helm based uh based internal microservices in Confluent Cloud. Yeah, and there are a few These are some of the features that we that we have in this platform. Uh I'll be going through a few of them as as a part of this talk and in in the in future. Yeah. Uh so, we have a bunch of technologies which which make which we used to make this happen and these are a few that I've I've listed here. Now, uh to crux the crux of any deployment platform is the version that gets created, right? The version or uh you know, uh the version basically what gets deployed and where it gets deployed. So, that is what we'll be talking about in this. So, the version is in Confluent is basically uh a YAML spec that uh uses uh uses the developers define in their own repository. So, we have like uh 1,000 uh services. So, we have 1,004 YAML 1,000 YAML files that are present in their own repositories across GitHub. And the complete ownership of that is present with the service owner. Now, for example here, we have service as the as the ultimate uh object which which has the metadata about the service and it also has information about what uh about who can deploy and and many other R back details like that. And you have multiple cluster groupings present within a service. So, this cluster grouping is basically a target of clusters that that is defined for this particular service. So, for a particular service you can have like devil development cluster groups which are present and you can have like fraud develop fraud cluster group which is present. And the configuration for each of them is I mean the target is defined in this YAML file again. We'll discuss about how the cluster grouping happens and and and in in the future in the next slide. The version The version the the next next object in this is the version, right? So, the version is basically a snapshot of all the configurations which which are which the service will use in order to come up and connect to whatever the relevant microservices or whatever it needs it needs to do. Now, how does this change in GitHub? I mentioned that this was present in GitHub, right? How does this change in GitHub get come to our system the unified deployment platform? And I mean, that happens with a merge to master when there's a merge to master in the GitHub which happens. Um the YAML spec, if there's an update to the service service level details that we have, uh we would update that in our UDP in our platform. And this in a similar fashion, if there's any update to the cluster group, it would again contain again the cluster would be either updated or created or depending on what happens, it would be like it would it would be updated. And we also have the version which gets created. The version is basically like a number which is a number or a tag which is given to the service owner which they can use to deploy that particular configuration on their clusters whichever they intend to. And uh Uh, within a service I have mentioned this before, but within a service there are multiple cluster groups and each cluster group will have one version. Uh, like multiple versions associated over a period of time, right? But at one point of time, like it'll it's a snapshot. The version. Now, coming to the cluster targeting, I mentioned that we have some services running on just a few clusters, while some of the services might be running across clusters, right? For example, if there is a a telemetry service or maybe the logging service, which uh, which is needed for collecting the logs and metrics, it would be it should be present in all the clusters, right? You know, you want to understand what is happening in all the clusters. Uh, so we want to provide the users the flexibility to to define where they want their service to be installed in and with what configuration, if it's possible. And to do this, we have the cluster grouping uh, that we that we discussed previously. The cluster group So, here service A has two two groups. Uh, group A service A will of service A will target just the cluster one, but group B of service uh, A would target cluster three. It can so happen that all of these clusters are again targeted by just one group, but that's not the case here. And in in service B, we can see that there are uh, multiple uh, again cluster groups present. And this is all these cluster groups are can be defined by the owner uh, by the service owner itself. We don't uh, mandate that this cluster one and cluster two are supposed to be present in just the the group A or group B. They They give that flexibility to the to the service owner. And how do we do this? Like To get into the details of this, we have a cluster which has some metadata present with it. So, it has a bunch of regions, tiers, labels, which are present. And these labels are then used by the the targeting defined in the group to like target the specific cluster. So, for for example, in this case, it is a simple example that I've taken. Cluster contains label A. If the cluster contains label A, target that. That's what That's where we end up deploying a group versions related to group B on that cluster. But But this can be way more complicated, right? It can It can be just that the cluster the cluster It can be like cluster contains label A, but label not label B. It can be a multitude of things which come in. And this label that we define, it can be very fine-grained. It can be just the cluster ID, which is unique to every cluster. So, it can be like target cluster two. Or it can also be like a broad thing, like target all the clusters in the prod environment. Or target all the clusters in the staging environment. Something like that. It can be It can be way broader. It can be like GCP cluster All the GCP clusters. So, it's it's All of that is is a possibility and flexibility that we provide to the to the service owners. Now, uh we spoke about how the version gets created and how the targeting happens. We we uh we'll be looking at how the version gets deployed, right? So, in order for the version to get deployed, we have uh the service owner would need to trigger uh installation request on on the CI or through the UI. We We provide a bunch of options like this, uh where they specify which service, which version, which deployment, which cluster grouping to that they want to deploy to. And they also need to specify the In case they need, specify the clusters that it needs to be deployed to. Uh After this comes the orchestration layer, uh which takes care of batching these batching the eligible clusters or maybe even selecting which are the eligible clusters. This is needed because in case the cluster mentioned by the by the user is not a valid cluster for that particular service for that particular cluster group, then we need to then we need to make sure we need to cross-check that, right? So, this basically does that. We can't be completely dependent on the user's input because we might end up installing wrong services where where we don't intend it to. And after the after this batching is done, in the orchestration layer, we also provide a way to like do progressive rollouts. So, this is very important in case you want to to reduce the blast radius. So, you want to first release the change to like 5% of the cluster, then then the rest five, and then so on proceed like that, right? So, the orchestration layer also helps you batch it. It It gets the list of all the clusters, then it creates batches in the way that is needed, and then it it issues install command on all those clusters to our platform. And this uh this install request which comes in, it it it involves like rendering the manifest and and getting the exact details of what need to be deployed. Like we have multiple templates which are present, those need to be like resolved. Like we need to understand what what are the cluster groups and all of that happens in based on the cluster. There might be some parameter some metadata from the cluster used in the charts, use that using that using the data again, we need we render the charts and then once we have the chart rendered, we we have something called as an operator present on all the clusters. So, we had like 5,000 odd clusters, all of these have like an operator which is shown here. So, the first object that you see on all of these clusters is the operator and it's present on all the clusters. Now, when when once the rendered manifest is generated and stored with the platform, the operator keeps continuously polling our platform and given this platform is a it's acts like a core service and then there are multiple agents which are present on on on all the clusters that we have. Now, with this with this polling which happens, the as soon as there's a change to the rendered manifest, what ends up happening is the the operator picks this change up and and then starts applying it. It might we currently use Kubernetes manifest apply and then we are moving slowly towards Helm based application and the operator applies it. Now, you might also notice that there's an other operator for non-Helm based based things, right? So, this is basically required for non-Helm the operator this operator is deployed by the base operator that you have and that takes care of all the scaling of pod pod that needs to be present, managing stateful sets, all of that in the non-Helm based based products that we have in place. Now, yeah, this is about the install and coming to the next part, yeah. There might be some cases where you don't want any deployment to go through like like we recently had Christmas holidays and that tends to be a lean period where people don't want to deploy anything or many people are on leave. Like they don't want to work. And in that case, you don't want someone deploying a change where it it breaks something and then everybody has to come and work and figure out what is happening. And in order to solve for this, we have something called as maintenance block which we which is applied. And this is again very configurable to uh you know, uh to to have a certain number of clusters or it can have certain number of services and once the service or the cluster matches, the installations wouldn't happen. It's basically blocked from happening and yeah, uh that's that's about the version deployment flow. Now, uh I spoke about continuous deployment and I mentioned to you that it it's it's very important that continuous deployment happens where there is real-time monitoring which is taking place and and it happens in phases basically, right? From one one environment to another and in between there are big there is big time and then there is there is you know, monitoring which happens on all the required metrics making sure that everything is fine. Now, in case of progressive rollout, we have this case that we support both sequential rollouts which is basically uh given a phase like the customer is free to define I mean the service owner is free to define whatever cluster set as the first phase that they want to. So, they can have first re uh first set of clusters defined. In this case, it's based on region. So, there are multiple regions which which the service is getting deployed on and that is happening in waves. Uh now, uh with serial, we allow it's it's pretty simple. We allow the deployment happens one after the other. And after each wave is complete, we can we have some big time. And then we have post-deployment and pre-deployment checks which run in order to make sure everything is fine. And if if something has gone wrong, then we roll back the change or we make it stop there. Like depending on how the customer I mean, the service owner has configured it. It can It can easily be that uh they just want to roll back or the other way around. And uh in this case, it was successful, so it proceeded to the next wave. And sequentially, this happens. Uh now, this is a very uh simple thing that I've shown. We uh the the service owners are free to create more customized and more complex workflows if they need to. Now, uh coming to the parallel workflow that the progressive workflow allows, I mean, in in within a wave, you can have like all the regions targeted just at once. Uh and again, you'll have uh big time after everything is successful, and then deployment, making sure that everything is fine. Uh Now, this was This is an other requirement that you had where we had like three four services which need to go in together. Like Kafka goes in, then there's a R back change along with that which needs to go in. And then there is an other uh different service which is also needed, like something like networking for the Kafka change to work as expected. In in those cases, uh again, uh we support that uh we support that functionality where users are able to specify multiple services that need to happen. And it can either be sequential, parallel, however they intend it to, like. And the deployment checks are pre- deployment checks that would happen. It would happen either based on the configuration of the user. Where we basically go look at the monitors uh that they would have given to given us. So, uh these monitors are basically uh metrics based on metrics where uh you where the services emit metrics, right? So, we use that and have defined some monitors, and those monitors get used to understand what the state of the uh of the services and then proceed. Now, uh in all of the cases that I discussed till now, uh the deployment the installation had happened through uh a user triggering it. Uh like user had I mean, the service owner had triggered it. Uh this was this this would go through a CI or maybe a UI, and then uh and then it would trigger the unified the deployment platform, and then that would trigger an installation, right? But then, there is uh uh there is also a case where called case of cluster bootstrapping where the customer Confluent customer is is takes an action due to which uh it triggers a cluster creation. Now, in this case, uh we encounter a chicken-and-egg problem. I'll talk about that. So, basically, once uh the customer has triggered and cluster creation due to some reason, it can either be that they've asked for a dedicated cluster or the multi-tenant cluster has run out of its capacity, and then we need to new use a new cluster or something like that. Uh in in such cases the provisioner we have a provisioner service. This this is a thing different from the deployment platform, but this basically sets up the infrastructure at uh at uh of the cluster that is needed. It makes sure that all the basic infra which is needed, like uh networking and uh pods that need to come up, auto scaler, and all of that are present on the cluster, and and uh and those things nodes that need to be assigned. All of that is taken by care by the provisional service. Now, uh but but the problem is we don't have the operator which does the deployment of the services on this cluster. Now, if this is a Kafka cluster, we want Kafka uh related services to be deployed on it, but we don't have the we don't have the operator here. Now, who deploys the operator? So, like it's it's a chicken and egg problem that we end up at. And here uh the the provisioner solves this solves this by going to the deployment platform, understanding what getting the operator manifest, and then manually applying it on the cluster, and then creating the operator for for us. Now, once this operator is up, it uh goes and fetches all the default versions that need to be installed on the cluster. Now, these default versions uh if they were present in a GitHub file, we wouldn't be able to uh like do it. And this is why we have had to move away from the GitOps model in in our deployment because the default version that is present, it it needs to be present in our DB. And and our deployment platform needs to be aware of what the default version is at a given point of point in time. And this uh this is why we store uh the default version and version information in our DB. And then after the operator fetches all the default version, it gets stored in uh it you it then brings up all the services that are needed, and a bunch of services that are needed on the cluster would come up. And that's how this whole flow would complete. Uh yeah. Uh now coming to the current and future developments that we have at here. I mentioned that we are currently using the uh highly uh custom CRD that that gets that is the operator that is there, which is a custom custom controller which which is there which which I've written and this applies all the manifests on the clusters and makes sure that all the resources or whatever is needed comes up. Now we are moving to a open source open source controller there which is flux CD which would support more cloud native features and and things like that. And we are also looking at possibility of getting rid of the operator all together. That's the agent that I'll be talking about here and manage the deployments completely centrally. This has some scale issues but yeah, we are exploring how we can fix that and get around it. And we are also looking we have CLI tooling which is present so that users can interact with our platform, understand what is deployed, how what are the targeted clusters, what what are the service details that are currently present on in the DB. Interaction of all of that is taken care by our CLI and that CLI we are looking at integrating it with our LLM tools that we have in house to help make investigations faster. Cool. I ranted about this for about half an hour but you might be wondering why do I why do we need this? Can't we use an open source solution or a third third party solution that we have in place? Uh and and why why do I have a job around this and why does my team work on this? But the the the crux of the the thing was when we did the evaluation there were not a lot of products which were cloud native and finished and had all the and had all the features that we requested. And it it did not either have the uh the integration with the Confluent ecosystem which was present, like we had a bunch of Confluent specific lifecycle tools which with which the integration was needed. And for that, we had we needed uh we we had to build our own uh system. And uh and there were a bunch of other security requirements which which was uh which which was uh which was to be taken care as well. Like the RBA that I mentioned, like we had to control who could deploy a particular service and that had to be present with the service owner. They had to take the decision on who could control the deployments and things like that. All of that was not possible at that point of time. Uh so, yeah, we had built this. And and we're actively, like I mentioned that we are moving towards a Flux CD controller, right? Which takes care of the deployment on the cluster. The the operator is now moving to a Flux CD controller. And that uh things like that are we are trying to integrate uh open source tools as as and where possible and maybe even third-party tools. So, yeah, this evaluation was done 5 day 5 years back and I think that's it. Thank you. If you have any questions, please. Sure. Uh you Flux controller or Argo controller? Flux controller. Uh we had done a comparative study on that. Uh I don't remember the exact uh reasons. Maybe I'll get back to you on that. Yeah. Uh so, the pro- the question was what is the reason why we picked Argo CD, why we picked Flux control Flux CD controllers instead of Argo CD controllers. I think one reason that I remember was uh, we had uh, already used Flux uh, controllers in one of the other projects that we were building, and we we knew what to expect uh, from that, and that we decided to use it here as well. So, yeah. Yeah. Yeah. This decision predates me, but uh, from what I understand, this this was considered. Argo CD was considered uh, as one of the options that we could have taken, but the we uh, like I mentioned, right? There were a few missing features like uh, that were present, and integration with the other tools that were needed was not was not there at the Sorry. Yeah. Yeah, good. Correct. So, uh, in the GitOps model, everything that we have would would stay uh, would reflect what is present in the in the Git, right? Git repository. But the problem with that was uh, when there's an actual deployment, it makes sense. Uh, when the deployment is triggered by user, then it makes sense. But then when there's when there's a new cluster which comes up which was not previously targeted by any of these clusters, we still need to figure out where this cluster fits in, right? So, that information we'll have to again either go back to GitHub and then fetch it from there and then do it. That we found it a little tedious and we instead chose to do it in the DB where we have all the information present and then use that. Yeah. So. Correct. It It It would require, but we're still evaluating that, yeah. I mean, that's that's that's one of the possibilities that you mentioned. You can do it that way. Yeah, that that can be. Any other questions? Thank you. Uh