OS as a Service at Meta Platforms

Hi. Uh my name is Sergey Droski Maraba and we work for Meta Platforms where we are responsible for running a Linux as a platform. Um we are sorry our stuff that we are going to talk about isn't as cool as a most of the stuff you've heard during today's routine work. uh still we thought that it might be interesting if we talk to to you and present problems that we face and how we solve them. First of all why uh OS is important uh nowadays uh everybody talks about AI buzzword is isn't Linux anymore but the truth is that under uh any AI there is still Linux machine that needs to run reliably. Um what is the problem with uh OS management that we challenge uh we we have right what are the challenges first of all uh meta doesn't own operation system we run Santos which mean it's completely external to us and we have to deal with that we uh don't control cadence how often it updates go there we um at the same time we have to make sure that machines and we are talking about millions of cost that we manage gets updated on regular basis because there is security requirements, there is uh policies and so on and so on and so on and we need to uh go to the latest uh releases as not as soon but with CentOS streams the life cycle for versions is much shorter than for Red Hat for example. CentOS9 will be end of line next year. So what strategies? How do you solve those problems? um standard uh image based to us uh automate everything uh uh can constrain or flight life cycle meaning that to and uh try to manage OS as as a service and to do that we developed several tools um to some extent some of them are open source if you know how to run buck because uh everything at ma is uh bug targeted and bug bill and we are going to talk about uh those tools uh little bit right first of all uh OS snapshot and the tools that we we used to create them is called rolling OS updates uh why do we need them again because OS is uh external to us and we need to combine uh Santos repositories DNF repositories with uh some vendors repositories like Nvidia and HP Microsoft with a hypers scale repos that David was talking about a couple of hours before and uh present it as a uh dated snapshot to two uh parts of of uh uh our infrastructure. one uh to a host as a set of DNF repos through the standard DNF uh protocol and another part to uh image build system uh as a buckified list of targets where every repo and every uh every package is can can be addressed as a bug target and built as a buck target and you can say that uh I want that package dated and and and and predefined Um so to do that uh we create we defined uh major release like Santos 8, Santos 9, CentOS 10 as a set of DNF repositories with their URLs. Then uh every couple of weeks we uh do sync up operation from inter external to internal. And when we do that again as I said we uh upload everything into our internal infrastructure and make them uh presentable as a standard DNF repos and as a list of uh bucket uh buck targets. Uh we verify RPM signatures. We build uh new images and images uh for services and for operation system bootable images. We test them before going rolling them out into production. We have to test to testing means uh re-image rebuild um certain set of machines uh into that new image and make sure that at least they boot and uh ideally they can run production workload. Then uh those snapshots gets released uh to image build and to chef meaning um to to production fleet. Uh another tool that we built uh internally is image builder that's called antler. Um as you know there is many many many uh image building tool. Why did we need a a new one? Uh basically again because we have very specific requirements. It has to be integrated with buck. It has to scale up. It has to be able to build uh format that uh compatible with our infrastructure. It has to work fast. And that's why we created antler. Creator is here. Vinnie Marra. If you have any questions, you can ask him. And he did presentation couple of years ago on that. How does it look like image definition in uh in antler? It's basically basel code uh starlark code in in buck uh we define uh certain features uh where users can say that build my image use uh such and such image as a parent layer and then add a set of features. You can install RPM you can uh enable systemdunit if you want to. You can create directories. You can do lots of different stuff uh building and then you do bug build and you get the image either um on your local machine or uh build and upload it into our infra. So you can rebuild the machine uh server test that image or install or it can be integrated into CI/CD and then it it builds and goes to prod if we are talking about uh uh service images for example. Um the main thing about this the good thing about that that all builds are reproducible and deterministics because uh we convert all those DNF repositories and RPMs into bug targets and check them in into our source control right uh you can check out uh our source control code on any date or any hash and you can build image and you are guaranteed to to get exactly that image how it was built months ago or year ago and it it's it's again was a real uh hard requirement for us that we have to have that feature uh images uh they are uh OS agnostics meaning that when you create an a base image you don't have to specify what version of Santos uh you want to get there 9 10 eight and only in the last one. You see here uh we have like image layer and then um parent layer and and and then we default uh in in the very last one we say that it needs to be built as a send to S9 or send to S10 and we get it uh as as as requested and using all those uh nice tools we were able to create a product that called metal OS which isn't distra but the way how to rerun Santo to s at at meta how we package and how to we deliver it into production and uh Marello is going to talk about that. >> Thank you Sergey. So uh what is Metal OS? We have this northstar that is what we want to build is what we test and that we run in prod. And uh in order to do this we need an elaborate and uh extensive image building subsystem such as the one that Sergey presented. And this CentOS flavor is assembled from Quir pieces such as UKI bootloadader. And some of the work that we recently did ended up into systemd 257 such as the uh command line uh the sorry the micro microode adons because we needed to load microode very early in the boot process before other hardware was initialized. Then this boot loader uh is able to open up the control partition that contains it's a single battery first system that contains the all the rest of the the pieces of the OS. The bootloader then reads this well-defined host blueprint. We call the the host config that contains different parts uh things such as the host identity, the host name, the IP address or the root root password hash. Things that can change only when the machine is reimaged. Then we have the boot settings such as the kernel that the the machine should run or runtime settings that can instead be changed uh while the workload is running. Boot settings instead required at least a K exact. The goal of this is to have an OS that is as read only as stateless as possible. So the machine boots the UKI bootloadader is initialized open the controls partition reads the host config and finds out which kernel should I boot now then it k exacts into the target kernel that opens up the initi that generates the configuration of the running system out of the host config itself and switches root into the root fs. So you can change these pieces with either uh reloads or reboots or reimages and you have a unified and coherent life cycle that allows you to minimize as much as possible the on host updates. Eventually the last piece that as well has a config generation mechanism as well out of the host config uh is is able to run service containers that are standard systemd exec uh containers with sandbox sandboxes defined into the systemd units and um uh these service containers uh are a step ahead with regard to what we were doing previously because we used to run instead add say log shippers or telemetry agents or other agents that we run on all machines all together into the same uh host uh OS that that is not great for isolation. Instead, we are we're going into a very uh long running effort to isolate all these pieces so that they don't stop on each other. However, this is all fine and good, but we can't migrate to this scheme overnight. So enter M classic that in it's a metal frame for a classic OS that is is assembled from pieces defined in the host blueprint. We use the host blueprint to assemble the OS. We use changes to this blueprint to say change the root FS that contains a new CentOS release but it's still read write and stateful. So it allows the old chef recipes to install software right configuration and it's our migration strategy because it's a big fleet with lots of teams and a lot of chef recipes around and things that have been done this way for uh more than a decade. So it's difficult and I mean it's a it's a big effort to migrate. So how we update this? Because if we say we have an OS that is mostly mutable, we need to have a way to issue maintenance windows, drain the workload and um reboot the host to install new versions or if not reboot KXe. And like Santa visits kids with presents, we have trains that visits host with box cars fulls of upgrades. And every host is visited every 45 days. We do this in batches. More or less each shared fate zone is visited at the same time. and we deliver fresh pieces. This is a way to deliver safely non-critical fixes. Say we have an incident, we put a mitigation, then we want a permanent fix, but we don't want to just SSH to the entire fleet and deliver the flicks to the the fix to all machines because that would be unsafe. Instead, we say okay, we're going to put this fix in the next version of the OS image and we're going to deliver it. Uh because we have an established 45day cycle or rather high impact upgrades say systemd versions we used to update systemd online while the work was running that is unsafe and we decided to uh instead do that with maintenance strengths. we contain failure and if upgrades don't go as planned there are automatic handbreak mechanism so that they detect if the success rate is below a certain relative or absolute threshold then the uh upgrades are automatically stopped. Still we require a way uh to update hosts in place because you can't just tell security that hey that fix for the zero day is going to be released is going to be rolled out in 45 days. So we still have chef but we are moving from a design in which we had chef to do everything that is first boot initial provisioning configuration software management and whatn not to have two different agents one is called host agent that is mostly workload oriented so it takes care of what the products need and metal d that is more oscentric and they both enforce life cycles and conr to ensure safe and coherent upgrades. Further, these agents expose APIs to the control planes that roll out changes instructing uh these two these two agents. Now what where we headed? We want to move more software and config management out of Chef, leverage these two agents life cycle and constraints and reduce the scope of in place host updates and possibly stop the arrow use that Sergey described because we were in this I mean scenario in which we build the OS image, we test image, we roll it out and then chef changes the OS image and we I mean all our tests are less relevant. uh new measure releases as well. They occur every couple of years. They require some preparation for the base layers that build on the new uh measure version. But the continuous provisioning cycles that we've established in reimaging the fleet periodically made these migrations safer and easier because it took 15 months to move from CentOS 7 to CentOS 8. But then from 8 to 9 we took just 3 months that is just two uh release cycles controlled rollouts with incremental switch over and baking. So we install a new thing and wait does did everything break? Did something break or is everything okay? Okay, let's move on and have a roll back mechanisms always in place. Reaching this 45 days goal required a lot of coordination. I was personally responsible for it and my strategy was to first make everyone aware of the benefits and ensure the visibility that means do project manager work and prepare dashboards with fancy graphs and things so to convince people hey this is the right way to do we work together on regression and we gently pushed back on requests to delay the cycle proposing alternatives because when people say I am scared of this uh frequent reimaging. Let's do it instead back to every 90 days. We work together to make the updates safer and ensure that the requirements were satisfied. Of course, you need a buffer of machines that you have to use when you take machines out of production. you have to move that workload to machines that are idle and waiting for um uh I mean as as a buffer and if these buffers are become thinner because of other incidents then your cycle is going to be impacted and as well when you require technicians to go into data centers and you have to schedule manual work in data centers that also is going to impact your cycle because that takes an higher priority because scheduling humans is something different than scheduling upgrades on machines. So there are many things to think when you have to work at at this scale. Anyway, we were able in about two years to change the OS on 97% of this multi-million server freight. We are able now to visit and reinstall hosts every five every 45 days. These images go under an extensive testing uh on our continuous integration pipeline that provisions real hosts run real workloads on hosts and verifies that everything is fine before releasing the machines to the maintenance trains and basically all senate to and to centtos 9 migration was automated the code modifications. Now we are planning to migrate to Cent 7 at this scale and hopefully we'll do that in three months. Thank you, questions, comments. Yes. um perhaps slightly OT but can you talk a little bit about what you do in the hyperscaler SIG and uh after all this work if you are still happy with going with CentOS or if you consider rolling something of your own >> um hyper scale and seek like David said it's a version of software technically upstream software that uh we and our core partners built ahead of scale schedule there is stuff like systemd for example right we have our own systemd team which uh develops features and and sometimes they find bugs or adding new features that cannot go upstream right away so they build it and publish to hypers scale so we can install it if that makes sense eventually all of that will go to uh upstream David wants to add something >> uh to answer your Uh I think the IPL work is kind of orthogonal to this and the idea with IPL is to try to do as much of the work we can as possible upstream and I don't see that changing at any point in the future. I think we'll always have things like for example the system work we do maintain there. Uh as for using something else outside of Santos um there isn't really a reason to. I think if we were if we had nothing and we were starting the company now and we were discussing which OS to use, we would have a conversation and that could go in many different directions. But I think even in a world where we have now where we have millions of machines running central stream, we have established relationship with that community, we have processes and tooling and work tied in that space. I I don't see a compelling reason to consider changing other questions. So I have a question. How many times per week does Dan ask you to switch to MKSI instead of your beard image tool? >> Say it again. I'm sorry. >> How many times per week do you get asked to switch to MKOSI instead of your own? >> Constantly people asking why you don't use this or that or that. Uh the thing is that we would love to but to some extent we are uh we own legacy platform huge legacy platform that we cannot rebuild in modern uh way right without getting into the world news with the headers like hey WhatsApp is down for three hours and you know >> yes sorry for being >> I will also add that whatever Make OSI is doing would never ever ever do what they what they need. Antler is very much the right tool for the job. >> And to to make you happier, right, this is a fleet that runs services and technically internal. There is a uh se at least two variations of metal OS uh net OS that runs on our switches and uh now we built AOS that will run on the corner and those are more uh agnostic to new platforms they built from from scratches. So we are going to use more modern technologies there. Are you running this um all on bare metal hosts or is this about VMs and then how are you >> it's bare metals but same uh antler builds images for bare metals for containers for services that run on bare metals including uh images for uh twain and our which is our scheduler for uh jobs and production workload right the goal is that no uh service userfacing service runs on bare metal. Uh they run in a container uh under our container management. >> Okay. How do you do the reimaging? Is it do you do AB updates like if you look at VMs or >> it's it's a total rebuild. If you're asking about bare metals, it's it's a total rebuild. Uh wipe out and install new inits, install new everything. >> Okay. Thank you. Mhm. It's uh we have an entire net booting infrastructure and and machines come from the manufacturer to try to net boot first. Then they are going always to send the HCP requests but the HCP responds only to machines that are configured for net booting. So when we want to reimage them, we configure them for net booting into the networking infra. they reboot and then they get an imaging ind that DDS the the GPT image and then uh inside that it downloads the other battery fest uh send streams for the other pieces. Uh, one quick question. Uh, you I think you said that like 97% are utilizing this OS. Um, are the others just like uh is that a haunted graveyard where you can't turn off a server or um what's happening there? And also >> it's long tail. >> Okay. It's long tail and might be special hardware that's not uh we we haven't uh got to to to that LST that type of hardware we are comfortable for running of this but we are getting there questions comments we have oh yes are you generally metal fans on a team or the metal reference These are from the time when film was on the >> you saw that we had t-shirts with that Metallica classic. We don't wear them today but u uh as you can say I I I grew up on on metal music. Yes. Deep purple lad Zeppelin and all of that. Yes. >> Okay. >> Not Metallica though. >> I appreciate all the references >> and uh those antlers are for antler because they they have a similar spelling. So that's a Linux penguin with antlers because it's built on antler and yeah with a rockar. >> Any other questions? >> It was named before we name it company. Sorry. >> Sorry. That's all the time we have. So thank to the speakers. [Applause]