The Evolution of GitOps in Platform Engineering - Artem Lajko, iits

Hello everyone. I'm really happy to be here and also really proud to be on the stage and present to you the evolution of GitHubs in platform engineering. I try to put between seven and 9 years in 30 minutes. Let's see how it works out. So, but before we start first who am I? My name is Atan Leo. I'm heading for consulting. It's a company from Germany. I'm also CNCF certified kiponote ambassador for cut.org and published co-author of the book implementing githops with kubernetus. At the moment I'm working on a second edition of the book and I'm also part of a team where we built vera.io. It's an open source framework where we are having created for example a lot of teams wasting a lot [snorts] of time to find the right tools and this helps you to build reusable uh platforms from scratch and we're using the same framework to address the challenge for 15,000 plus cluster to manage it worldwide on edge we combine the both states this is what I'm working to see here on and our mission today will be to understand the future first we need to understand the past and understand where all the things started also to see what is possible today and also to understand where we did the wrong term. So I like to having a common definition or like a common understanding I'm not saying that the terms are true by definition but it helps us to understand this talk. So just two three terms the first one is declarative. What I mean by declarative today is for example if I have a client in as a user I'm just saying for example I want this deployment I don't care how it will be deployed it's just fulfilled by an agent by an operator or something like this if I'm speaking to the catalog I'm speaking in the most cases about terform models umbrella charts uh custom resource definition bash script everything you need to provide a self service for your platform if I'm speaking about push versus pull this confuse a lot of people push in the most cases meaning for example you have CI/CD pipeline you have credentials to push it to your cluster and pull is meaning you have an operator running on the cluster maybe just only read only and pulling the manifest from OCI or git. So how it all began? I don't know if you're familiar with this space. This is Alexis Richardson, the CEO of Config and he in 2017 he gave it a name. He called it GitOps and what what problem we wanted to solve in 2017 or 2016. It was like this is a classic enterprise in Europe how it looks like uh about 10 years ago. You can see for example we have a five load balancer HA setup to Apache web server and then application server. So what we did we jump from VM to VM via ADP sessions [clears throat] or SSH and to configure some HTTP configs or the if you're running on Tom some XML files this part doesn't scale maybe for 10 VMs maybe for 50 but if you are reaching out about 300 it doesn't scale so we need something new and this part lead us to infrastructure as code auto configuration as code with tools like enzel terraform soul stage chat carpet and another tools and everything becomes as code. So we are able to define infrastructure as code. We were able to define policy as code, network as code and everything as code. So but with new shiny tooling we still have the same challenge because um it's anal projects where I was involved we had like I call it here local ops. So different engineers running it in local machines is they just living on their machines. it doesn't scale and also create a lot of race condition. Then we moved from local ops to pipe ops where we let execute the pipeline everything also saves the states but tools doesn't change the mindset. So we needed something different. We needed to go away from local ops, pipe ops to an agent based. It's not AI agent. We are not speaking about AI agents today. So what agent-based approach means? Agent keeps the environments in zinc. I'm as a user declare my intent to get or OCI as a contract. The agent is watching it and fulfilling it. So every change I want to make, it's just to the contract. Very simple. And the agent keep it in zinc. This was the idea to going away from local ops pipe ops to an agent based approach with githops and this approach was based on four on basically principles uh on the four core principle of open githops they are really easy to understand it's declarative you define a state then it's version and immutable you need to make sure that you can change it and after you locked it it's also pull based the state will be pulled and after everything it's working the agent or the controller doing continuous regulation reconcile and looking at feedback loop. If you have a drift he will correct it. These four constraints make GitHubs predictable and scalable. This becoming very important for later of the talk and just remember for principle no more no less and this for principle allows also with GitHubs for example not just managing a single chip or single cluster we are now able to managing a whole fleet like 10 100 cluster for example by creating a catalog a catalog is meaning putting all everything you needed then for example having different githops engine like you can see here Argo CD OS add-on controller putting one label on one cluster putting 10 labels or on one cluster or just putting on 100 cluster label like search manager and the GitHubs engine will take care of it is how you can design it and by also you need of course combine different approaches like if you are on the Argo side a lot of engineers combining app of apps pattern also a lot of engineers combining application sets with tools like cluster generator which works on labels based if you're on the fluxity side you're maybe more familiar with customize so you have like base values where you define it and you have for every stage or for every however you group it specific values and there are different approaches but the idea was the same. We don't want it to repeat ourself. We wanted to compose it and to provide this as a self-service. This is just a simple uh in the most cases a setup uh simple example if you're using for example set manager in the most cases we are creating an umbrella helm chart or rapper chart we are adding some resources they're missing and then we allow to define for every cluster for every tool dedicated overlays and make it flexible but having like a different configuration and what we did thinking back to the four principles now we allowing the GitHubs engine to execute code. So now we are putting the complexity away and let this hand template parts and so on doing to the GitHubs engine. It was never me made for that and this also becoming later more important for this part. Just remember it what we are doing now because it's possible and we're also doing this is just still a simple example and this scales to managing 1,00 clusters or like you saw at the beginning 15,000 cluster create multiple challenges because we just not only managing one tool uh this leads us to the config sprawl because the GitHubs engine like Argo CD just don't apply the plain manifest it's doing something like hand template minus third manager putting some parameters there giving some paths and so on. So it's becoming if you're managing I don't know maybe 75 75,000 applications of different tools it's becoming very difficult to keep to understand what's happening in your environment especially at scale and we also realized in 2022 that we killed a lot of APIs crossplane for example it's a tool to managing infrastructure cloud infrastructure but also Kubernetes resources to compose it um over the Kubernetes API server at the beginning now it's more stable but not just only the Kubernetes API We also saw a lot of traffic because now with crossplane we were allowed over GitHubs like to create 1,000 or 10,000 databases and it's also the G the GCP API struggled a lot and you can also see if you're going on scale that's git as a protocol also struggling if you have like thousand clusters and thousand agents running zinc different tools from git if you're going at scale and we lost the visibility also with this part because we a lot of overlays and we also understood that we need rendering now like it started two or three years ago. We have tools like Kumi Robin Leap yesterday uh uh talk about it and presented with Michael what Kumi is doing and why we need it then because of course another parts like config car created like created by Alexis Richardson they understand now if you're going at scale you need tools so they created config car to manage it or to get it back in control then we have also some project from ago like the source hydrator or argo city diff preview but also another project is this I don't list all of them and we also understood uh git was never the single source of true I teach it a lot and sorry for that but if you're looking here for example in the most cases we declare the client intent there so the images are outside the hamm charts from the provider outside we're just creating hem umbrella charts so it was never the source of true this is what we realize now in platform engineering and we call it now or we try to move it a little bit away we are saying now gci is the source of intent. It's a complex diagram but we are not going to all the details. We just taking a look for example at number one if we using git as source of intent we have the yl manifest we have the h umbrella charts but we pulling in from the outside like the provider h charts like the images and another parts if we are going to OCI we are able to create sbooms we are able to put images there everything package them and then let the githops engine deploy it but uh here at point 2 you need also to be aware this is just an intent because in the most cases platform engineers we have Some controllers running in the cluster like Kibano or Caberno make sure that the labels are set in another stuff so they manipulate parts. So you will never have the real intent defined in Git OCI um because they manipulated in the real store in the ATCD store of the Kubernetes and this is what we learned about it uh especially if you're going out scale. So if you just in short compare git versus OCI, Git is great for human collaboration. If you're using a lot of agents, not AI agents but also AI agents showing us it's great for declar OCI is built for immutable artifacts and also for real scale as distribution layer and gitless githops was also because git has some limitations. The flux community realized it really really fast. So gitless githubs with OCI uh arise. I can highly recommend the talk from Stefan Podan uh from 2022. So you can see you still have UCI pipelines and you hydrate your manifest but now you put it as artifacts to OCI and your GitHubs engine will pull it. This is the idea behind it behind gitless githubs and we also started to think about depending if you're managing multiple cluster what is the best fitting architecture for my company or for our customers I'm working for a consulting comp company so we have different customers different requirements and we start to implement different architectures like this is one of the most famous hub and spoke where you have a central hub and you have different spokes so you're managing it from a central point of view you're getting a single pane of s and you also have the visibility but it's not work on all environment especially in the regulated because it's becoming dangerous if you have access to the hub in the most cases you have high privilege access to the spokes then we also have dedicated instance per cluster where you deploy every controller dedicated on a on his own cluster this is also a setup I'm seeing a lot at banks insurance companies in Europe but also in the US then we also combined we are able to combine both architectures for example putting hub and spoke and deploying through the hub uh githops engines like argod and I worked in a setup and it's working really really well for the developer with the open airbark with their own permission management to provide a self-service and then also if you are thinking about scale you need also to understand that every controller or every project at tool we are speaking about charting so they can also chart you can say for example for flux or argod one controller is able to manage 10 clusters you can group them by we are speaking then about sharding this where you think about it if you're going at scale and also if uh things like agentbased architecture evolved it's still work in progress for a lot of stuffs is one of the most um equity has a platform it's also works like uh really really well but sws is also one of the most projects I'm familiar with which is already this approach implemented so you have like a hub you have lightweight agents running on in a this is a agent pushbased approach. So the agent are telling like hey something is changed and then the hub will reconcile or correct the drift but this this is the most part I like uh especially if you think on security on semi-air environments you have a pull-based approach so you're deploying the agent and the agent pulls the manifest so you don't need having inbound access no credentials inside this is why we like githops because we don't push it anymore we pull it and you can also be behind the firewall because you don't need inbound access there. the most um architecture I'm familiar with or we are seeing and coming to another part was what gives us also allows us we are now able to manage everything as code like secrets dashboards budget rules policies infrastructure and this becomes very interesting because if you provide a platform you are not just providing a platform for your developer as software developer you platform user are also fin product owner marketing everything need to having like a dashboard or control and this how you can easily provide a self service by having everything declarative and everything as code in the this was the GitHubs part we focused mostly in the meantime also the platform engineering part involved so now we are going uh focusing a little more on platform engineering and how githubs fit in so what's happening in the meantime in the platform engineering area defos was declared as uh about 2015 2018 in Europe but also in the Yes, from my point of view, the culture was real. It was nice. But this infinity loop of tools and creating roles of it and hiring people for an infinity loop, it's becoming very difficult in Europe. We had a lot of challenges because I mean you need to know everything and how do you want to fit a profile if you have a really a lot of tools. So what's happening platform engineering rises the DevOps tool chain expands more in an internal platform ecosystem. So we moved from this infinity from uh dedicated tools in the infinity loops like to a more organized planesbased architecture. At the moment the existing five if you have the five or if you cover them you're covering the most needs from your platform. For example you can see observability plane every one of us need observability or also resource plane where we creating the compute resources databases kubernetus cluster but also the developer plane is part of it. how you provide a safe service to your developer through AI agents through a portal and different other stuff and of course security plan to be compliant and also um enforce some rules depending on your area where you are and central teams are created sorry we mean relabeled if you're going on LinkedIn and looking on the most profiles you can see that the people are renamed or reabelled from system engineer to cloud engineer to devops engineer to platform engineer but from what they're doing it's still the same work they just changed the teacher. If you if you are happy and you are not just only relate but getting also a higher salary congrats um I know some colleagues of mines it's happened so I'm really happy for them but you still from the mind of a system engineer it is becoming a little bit difficult and dangerous if you try to provide a safe service I will show you some numbers um but coming back to here we after this labling part the same team same mindset have new mission they need now to increase developer experience through platform engineering and they tried to provide a safe service uh choose the right abstraction layer and the outcome was about the last two years they failed the adoption rate was really really low I did a lot of service personally I speak with a lot of product provider and here's for example new numbers from heckons from 2026 you can see for example to build a platform it will cost you about 4.2 €2 million and 18 months of work. But you need also to keep in mind to build it in 18 months of work, you need having engineering resources with experience about five and 10 years to build it in 18 months. This is what a lot of companies don't think about it. And even if you build it, you can see 64% of the engineers bypass your platform because system engineers building with a infrastructure mindset for infrastructure people. Now you have different developer of your platform but there also another um KPIs where the company's failing but this is one of the I found it out through different surveys and yeah with this mindset it's become very difficult to provide a self-service because if you're a software developer you're already familiar with features and customers creating requests but system engineers in the most cases not this is what we found out but not only we if you search for that and we also realized deploying backstage isn't equally to building an internal developer platform. So I heard a lot of statement lots like let's just use backstage it's production ready right no it's a framework to build your portal you need to integrate everything or something like just deploy it it's a hamchart no not of the box it doesn't even exist a official hamchart I guess a community hamchart exists or also things like system engineer as uh started to ask me why we need to learn react and writing javascript because it's was built by developer for software developer out of the context backstage is great and open Korea for example adopted it and another companies if you understand why backstage built it or and for which purpose and we built a platform to improve developer experience and in the most cases we also found it as a service we never ask the developers what they actually want for example it's also was one of my faults I was going to the conferences a lot of shiny new tools we are coming back integrate in the platform and just put it on the developer and say just use and the are like but what you want to solve. So we realized um we talked in the most case about technology but not about the daily problems to understand them and then we starting to talk about culture because githubs is about culture and immutable infrastructure it's not so easy to understand and if you're looking here for example we needed to educate the developer and saying hey you don't doing ad hoc changes in the environment everything is based on a contract so the githubs can or the githubs engine can fulfill it. So don't manipulate resources manually. You need to go to uh the loop. So Argo or another GitHubs engine can fulfill it. And we also tried to explain the developers over the time that branches or in GitHubs are antiattern and this was how the developer living. They have long branches dev QA staging pro. Instant we tell the developers trunkbased development information over folder is a new thing you need to working with short living branches everything is on master main or however you call it and then you have for example in the folder uh default configuration and for every stage you overwrite it by just moving the files. This is what we tried to teach the developer and then we realized branches and githubs make actually sense without human interaction. So tools like cargo emerge from equity and this tools allows us promotion through click ops but behind uh the scenes it's creating branches. So for the developer it feels like if he push something to main and change something in the UI it's like click ops but behind the scenes cargo for example uh using a warehouse defined by the platform engineers and creating branches and integrated Argo CD and Argo pick it up. So it's not an official term but I call it personally directional githops because not just only humans now fulfilling operation but also githops engine or another agents or controller making changes and pushing back back to git or OCI and we also learned one thing adopting githops correctly everything becomes else possible so continuous delivery it's not a problem progressive delivery and also zero downtime if you implement is the right thing so maybe you asking wait what is progressive delivery I keep it short we are it's umbrella term it's overreaching the philosophy of modern software releases we are all familiar with rolling update if you're using kubernetus so if a new version if the new version is healthy it will replace pot by pot and but we have also another approaches like blue green deployments or canary deployment where we want to test a subset of new features to uh features as a subset to just a few users but for this part you will rely on tools like argo rollouts of leger but they also integrated with the GitHubs engine you are using and you can keep it declarative and this is how for example progressive delivery can looks like and we're seeing on different projects not just progressive delivery but also um GitHub users to build more resilient platforms so disaster recovery becomes not a problem for state cluster little disclaimer so you can just point from one cluster if you're doing it right we had the philosophy from one customer we said if we need more than 30 minutes to troubleshoot a cluster, we just throw it away, spin a new one, repointing the pointer to the new one and everything, the stake of our platforms, the applications developer will be there. Of course, depends stateless cluster. If you're managing databases and state in your cluster, you need having a disaster recovery strategy also possible with GitHubs, but this is the easiest way. And GitHubs also becomes based on the service from platform engine.org, one of the top capabilities the third one of the platforms of the most platforms as the continuous delivery part of the um delivery plane. So and GitHubs becomes also the heart of modern platforms and soon it's starting something like faction started to form. I don't know if you ever played World of Warcraft but I'm giving some book signing on some conferences and in the most case of curiosity I'm asking the people what are you using Argo City or Flux City because the most are not familiar with status and they are like we are on the right side. I say okay [clears throat] I I I don't comment it. I'm just listening and not judging. But you can see different communities, different mindset. It just remind me a little bit on World of Warcraft. And as a consultant, I'm getting a lot of questions like which GitHubs should we choose and you will get an answer of course depends on uh otherwise I will be not a consultant. This is not a comparation of the GitHubs engine. I just wanted to highlight some of the GitHubs engine exists on the market. Argo City is the most adopted based on surveys on different surveys. Flux CD has also a strong community and Swam controller is a new uh GitHubs engine coming made for scale and if you scan the QR code we created uh some prediction before we starting with this 15,000 cluster challenge to see how we can manage it and we are able to deploy something about 5,000 application or 1,000 cluster in like under two hours and then I was running out of budget like 200k so um it's made for scale for addon management and now after choosing the githops engine like aros city flux city osto you can also combine them like you signs up and spoke architecture we starting to build real platforms and then you need to understand um for example platformless if you don't have engineering resources you can outsour this to solutions like korea or hioko where they have a platform team if you don't have it or if you for example we already took a look at backstage if you want to build a portal and integrate all these services you can use a framework like backstage or for example if you want to go an orchestrator and operator driven approach where you also need to integrate parts so you can create your custom resource definitions to provide the self-s service you can go with different solutions like open career it's a CNCF projects or cubrix moenius humanitic there a lot of different um tools existing and after years of failure this makes me a little bit happy on one conferences I had the honor to represent the new CNCF project open kore IO and developers starting asking me on the conference like where I can connect my AI agents. The question was like I'm have my workflow and I don't want to open my portal I just want to connect it through my workflow and it was like yeah open Korea provides you everything you need to communicate to understand the resource how it's built and so on. So the developer just can create a component if it doesn't exist. This is what I'm hoping how they will work. For example, the developer can for example create a request directly with AI creating the custom resource definition. The platform team can review it can apply it and the developer can choosing the new component. A component is like a web application API everything a team needs to deploy and I think I don't know if it's the right direction but I have the hope that it will increase the adoption rate about um platform engineering and because we at the moment we are moving away from internal developer platform to agentic developer platform. So yeah and also meanwhile what happening AI is becoming citizens of your platform. Running AI on Kubernetes is easy. If you have a catalog just put in it and provide a inferences service but integrating it it in your platform because GPU sharing is very difficult compared to CPU sharing you need to think about especially on small GPUs about time slicing because most cases the GPU you're getting the whole context and it's not so easy to integrate in your platform because you need to think about multi-tenency but it's now also part of it but also we have AI SIE agent like home GPT they don't use your platform they operate And the idea here is they're running or using it as a developer. So you need to create capabilities, you need to create security rules for your agent. I don't want to point out to agents today. I just wanted to pointing out that also citizen of your platforms. And what the agents are able to do they really able to do uh AI troubleshooting because they have the rich context. They have logs matrix traces everything in plain data. So they can troubleshoot it. And [snorts] remember this one, if you are using AI to generate parts of your platform with a lot of overlays, this is still a simple example. It's becoming very very difficult because the AI will give you gener generic uh answers to you to Helm, but it doesn't understand what you want to deploy. So this is what we why we need AI or AI need plain data, configurations data. This how it's cold and wet. Write everything twice. So what you see is what you get. If you change something you can directly see what change and configuration code will be you provide a template don't repeat yourself putting values and then you template the real manifest of it this is what the I needs to operate the best and also not just only I we have a lot of regulation or Europe created some of them one is cyber resilience act I don't want to speak about it I like the idea they are saying if you provide us software or a platform you need to understand your whole chain so if you hydrate the money manifest you we're getting access to everything and yeah we still have some unsoft challenges I just want to point out for example to the most familiar with for example if you're using ham umbrella charts we don't have a part where we can for example if something is missing how we can patch things because h doesn't allow it really so we using customize putting another layer on it and also promotion still a challenge and of course we have another challenges if you're going at scale but I just do wanted to pointing out to uh post on LinkedIn from Michael. So hopefully you're having now a plenty of questions, a lot of questions and I'm happy to answer all of them uh after the talk because this was a lot. So thank you very much for your attention and I will be really really happy if you can provide me a feedback. How do you like the session, the content and so on. Thank you very much for your time.