Container Networking With Netkit: The BPF Programmable Network Device

Uh uh thank you everyone. Um so thanks for coming to my talk. Uh today I'm going to be talking about container networking uh using Netkit. Um so just really quickly a little bit about me. Uh my name is Mike. I'm a software engineer at Meta. I work on the containers team. Um I live in Brooklyn, New York City. Um there's just a photo of my weekend hobby. I like to do um rally racing. So happy to talk about that as well. Um, but today I'm here to talk about uh the Netkit device. Um, so I'm going to quickly go over, you know, kind of introducing container networking. Uh, talk about how the Netkit device came to be. Um, then I want to talk a little bit about how we implemented Netkit at Meta. Uh, kind of some of the issues we ran into as we rolled it out and some of the benefits that we saw. Um, and then at the end I'll try to leave some time for questions. Uh, so uh, just kind of introduce like container networking in general. Um so at Meta we have obviously a lot of network traffic coming into our data centers. Um and if you look at a typical data center you would see rows and rows of machines like this. Um on each machine uh we have of course a desire to run like many jobs at the same time. Um so this is kind of logically how it might look. Um so there's a network switch at the top and it's connected to uh multiple hosts in a single rack. And on each of those hosts we use containers to run multiple services on the same system. Um so a typical uh host might have you know sometimes uh multiple even thousands of uh services running at the same time. Um and of course we want to be able to isolate the networking um between these services. So before we had network namespaces we would have this issue. Uh if there's two containers running on the same host and they both want to use like a similar IP address in port, there would be a conflict. They're all kind of like sharing the same um network environment. Uh this has, you know, kind of practical implications like all of these services would have to coordinate uh their networking setup. Uh but of course there's also security implications there. Um you know, a service could come and like snoop on another uh service's network traffic, things like that. Um so network namespaces was introduced kind of to solve this problem and introduce additional layer of security into the networking. Uh so with network namespaces once you um put a service into a network namespace it has basically its own view of the network. Um you could almost think of it as if it's running on a completely separate machine um on a different network. Um so here we see that this you know problem with like trying to bind to the same port. Uh this is solved because each uh service on this machine um has its own list of ports, its own interfaces uh and it can now bind to that same port. Uh so like service A in this case, it's not aware of service B at all and the networking uh can cooperate together. Um they don't need to kind of like uh talk to each other at all. Uh and the way this is set up um initially was using these like virtual Ethernet devices. Uh so when you create a network namespace, you would create uh in this case the virtual Ethernet A device and connect it to another virtual Ethernet device inside network namespace A and you would create virtual Ethernet B device and connect it to the network namespace B. Um, and this would kind of act similar uh to if there's like a virtual Ethernet cable wired from the network namespace uh to the host name space. Um, so this works for security, but it turns out it introduces a performance impact. Um, and the reason is that every time you transmit now a packet from the container to the outside world, it has to essentially get transmitted twice. uh so in the beginning it gets transmitted from the virtual Ethernet device inside the container uh to the one outside in the host and then it gets transmitted again from the host network adapter uh out to the outside world. Um so what's happening here is essentially there's a lot of extra um activity going on on the CPU. It has to handle these additional IRQ um interrupts. Uh so the netkit device was introduced essentially to solve this problem. Um and the way it works with netkit is when you create the first adapter on the host side uh it actually creates two devices. There's what they call a primary and a pier. So the primary exists in the host network namespace and the pier can then be moved inside the task network namespace. Um, and this allows the traffic inside the container to immediately be available in the host network namespace. And if you have BPF programs that you want to filter the traffic, they can um run it like essentially they're running inside the container. Um, but the packets are able to be processed immediately. Uh so in like practical terms what this means is with V you can see on the far left here um if we use kind of an imaginary 100 megabit uh adapter um with V uh we were only able to get about like 60 megabits per second out of that adapter. So essentially it's about 60% of the performance of the host um uh kind of baseline performance. Um and we see here uh using like netkit and bpf programs we can get the full uh performance. So we get the uh equal to the host performance. Um so that's kind of the the headline here is that once we enable netkit uh we can use network name spaces now we get that isolation between the tasks um but without any performance implications. Um, so that's kind of a summary of kind of why we wanted to use Netkit and how it works. Um, so I'd like to just quickly describe how we rolled it out at Meta, like what went into implementing this. Um, so uh, quickly how we work on um, containers at Meta, we have this uh, service called the Twine um, scheduler. So whenever a user requests to start a service, the request goes to this um scheduler program uh which is kind of a a back-end service that um will assign that job to a particular machine. Um the the scheduler sends a request down to our Twine agent which will um create the container for the job and do all of this networking setup. So this is where we implemented um the netkit device. So every time a job starts up, it automatically creates these adapters and then does the networking configuration for them. Um, and this agent, it's responsible for the resource control for these jobs, like assigning how much memory and how much CPU they're able to use. Uh, of course, like creating the actual container and setting up the isolation between the jobs so these services don't kind of step on each other's toes. Uh so in terms of where we're at with the rollout status today, um about 66% of all of the containers that we create or containers with root or without root. Um either case it should work. So um yeah, I guess kind of to follow up on some of the previous questions. So the the way that this appears Yeah. the way that this appears to this um task is the same as kind of any other network adapter. Uh and in fact inside the container if you enter the container it looks like east e east zero in the container and there was like no changes that the tasks need to make. Um they can just send like normal network traffic. Um and then under the hood it's it's implemented as this netkit device but to the container it just looks like a normal network device. So they don't have to kind of do any special setup. Um the setup is all done from the host side and then inside the container it just is like a normal networking setup. All right. So yeah so let me uh finish the slides quickly then we can do more questions. Um so yeah as I was saying the uh networking space at this point we have enabled about 66%. Um the way that we roll this out is via a feature flag. So as I mentioned like there's kind of no work for the task to do. We turn this on in the back end and next time they restart their task they will just start using netkit. Um we have it rolled out based on the criticality of the job. So when we say 66% of the tasks we started with like the less critical tasks. So uh every service owner at meta will specify the criticality of their service. If it's a tier zero, this would represent like if the service goes down, the whole website goes down and it goes all the way down 1, two, three, four. Um, so we started with the less critical ones and we're working our way up. Um, and you know, as we get to the end here, you can see it kind of slows down. We want to be more careful with these like final highly critical jobs. Um, but yeah, importantly like this is fully transparent to these service owners. Um, they don't need to do any any special work here. um once we turn it on for their job, it just starts using this new device and they get the performance benefits. Uh so just to go through like a real world example here, um this is one team that was trying to enable um what we call like stacking of their jobs. So what that means is um instead of like one single large task that takes up an entire machine, they can run uh four copies uh of their task at the same time. Um which is it's better for the service owner. they don't really need to specify um any of the particulars of the hardware. It's also better for our like operations in that sense like there's no specific um like hardware requirements that we have. We can kind of move the jobs around more easily. But what we were finding was when they tried to do this like stacking their latency was was just going way up. So uh in the beginning here like without Netkit the latency was going up to around like 12 seconds. Um, and once we turned on netkit, that latency immediately dropped to like around 100 milliseconds. Um, so this was really kind of an extreme example, but it does uh show how the um that like soft IRQ usage that I mentioned before, it can really kind of add up and start to affect these tasks if they are really like heavily using the CPU and um if they're affected by this um like performance. Uh so it's not every single um task that is going to have this like uh heavy network usage, but for tasks that do do have a performance dependency, it really does uh make a big difference. So I wanted to kind of quickly go through some of the next things that the team's working on um at the moment for this. Um so one of them is called uh IP router system. So uh I mentioned like the ability to like uh attach BPF programs in the container to do like traffic filtering. Um today the way we have it set up each container um will get its own like new BPF program. So if we have a thousand tasks on a host there would be a thousand BPF programs kind of in a chain. And this starts to eventually affect latency as well because in the worst case the traffic that we're trying to send has to go through each one of those BPF programs. Um, so at the moment we're working on this IP router system which would replace all of that with a single uh a single BPF program that just has the map of all of the containers on the system and it just knows where to route that traffic. Uh, another thing that we're working on is NUMA aware networking. Um, so when we have these containers running on the host, uh, for performance reasons it's very beneficial if we can keep all of the networking in the same NUMA node. Um and this is uh you know very important as we have like GPU hosts where the system is kind of divided up into these numa nodes and any time you have to copy any kind of data outside of the numa node there's a big performance impact. Um so we're working on making sure the container is aligned within the same numa node all the way kind of end to end. Um and the final one I wanted to mention is uh the next step for netkit um would be to use it also with like VMs. So not just a container but if there's a VM running on a machine um we could have the same kind of performance benefits uh using this system called AFXDP. Uh essentially what this does is create a shared address space inside the VM and the host and then um the like similarly any traffic being sent out of the VM will immediately be available in the host. Um so so we can take the same kind of uh performance that we've applied to containers and then apply it also to VMs. Um, so in closing, uh, I just wanted to mention quickly that like there was a lot of, uh, open source community support here. Um, the, uh, you know, we couldn't have done this without all of the authors, um, uh, the developers working at different companies who who helped write this code. Um, at Meta, I wanted to mention um, Tukshock, Ramen, and Martin who really helped with um, you know, implementing this and like working out some of the bugs. Uh and at isovalent in particular there's a lot of really helpful blog posts out there other talks that were given uh Daniel Borgman um Nicolay Nico and Bill Mulligan um they've really put a lot of helpful useful information out there. Um and then finally um I know there's a few questions about like the particulars of um how this works. So, if you're interested to learn more, I've put up some uh demos um and some additional like links to other resources um up here uh uh and at that link. Um and yeah, that was all I had. So, wonder if there's any questions. Good. >> Yeah. So, um a lot of people use like nspawn for for launching containers. what is the status of integrating netkin and net spawn so we get that instead of vs from now on. >> Um yeah, that's a great question. So that's actually you know a task that we have in our backlog um which we're hoping to get somebody to work on probably in the beginning of next year to add this into like uh systemd networkd. Um so it would be available as another um like kind of like device type flag that you could add into the uh systemd network setup. Um, and then yeah, so at the link here, there's kind of like the step-by-step walkthrough. So once you create the container, um, as of today, you could you could create the devices using just like, you know, IP link add, um, those type of commands, but eventually we'd like to have this like support in like systemd, network. >> Uh, yeah, phone here. You kept mentioning BPF programs, but uh do they you did not explain how uh frames are moved from the external Netk device to the actual network interface. Is this the purpose of the BPF programs? Are they provided by your orchestrator by the driver? How does this part work? Um yeah, so the question was around um like how does how do the BPF programs actually manage moving the data between the network name spaces? Um the reason that this works is essentially it's a single driver code um and then in the driver code in the setup it's creating two devices and those two devices have access to the same memory under the hood. So with ves there's actually like it's you know you have to create both separate devices and then the only way that they can communicate with each other is you know by sending like network traffic. Um but with netkit th those BPF programs can can access the data you know kind of instantly in the host name space. Um since it's just like a it's just a single device that just has like two ends basically. Um does that make sense? >> We'll talk about later. Can netkit be used with uh SIOV or is there any relation between SOV and netkit? Uh >> I'm not not that familiar with with that. Um >> the ability to like defer the virtualization on the network card. >> Um I I think it it should be Yeah, it should be compatible. Well, I'm not sure that there's any like I don't know of any reason that it that it wouldn't work that way. It kind of should just work like any other um network adapter, but yeah, I'm not too sure. >> Thanks. >> Are there any downsides with this approach compared to uh traditional VHD? >> Um yeah, that's a good question. So, I would say so far the only real downside that we've seen is around like debugging. So, one of the nice things about Ves is like because it just works like any other network card on the network, all of the normal networking stack, you know, kind of applies. So, debugging like if there's a network connectivity issue, it works just like any other like debugging on any other machine. Um, once you introduce Netkit, it uh because it's kind of creating this like short circuit from the container to the host. um like you kind of have to be aware of that um like that extra link that's in there. Uh so yeah, I'd say the only real downside that we've seen so far where like teams have asked us to go back to Ves is if they're like trying to debug uh some particular network issue um and they just want to go go back to V which is like more a more familiar solution um and they can kind of get get additional uh debugging output that way. Uh so it seems that the operation model for we and netkit is roughly the same. So the service owner shouldn't really be able to tell the difference except he has the better performance. >> So why uh was it why there was a reason for netkit in the first place? Wasn't it possible to change the v or add like an additional mode to v e or something like that so anyone can reap the benefits without changing anything really? I mean who is using we today? Yeah, I'm not I'm not too sure the technical reasons like why this couldn't be done kind of as an upgrade. I think the the main reason is like um like with V you you generally have to like create two devices um and then like move the second one into the container network name space. Um Netkit it's kind of a different like from the container creation standpoint it's a little different because you just create a single device it automatically creates the peer device which you move. Um, so I think it's like in the driver code setup, it's doing that like creation of the second device. Um, I'm not sure why it was decided, you know, to be done that way. I assume it's because um like it they wanted it to be like a specific choice that the like container creator is choosing to make this um netkit pair. Um, and that that allows existing ves use cases to keep working the same way. So where does ebpf come in the in the game there? Do you absolutely need to to add ebpf programs to the containers to make this networking at all or is it just enabling this as an additional fil filtering features and you you don't actually need them? >> Um yeah, so technically you don't you don't absolutely need the BP the BPF programs um to to be there to work. You could use it um just like uh any other network adapter. Um but I think the the BPF um that it's kind of an additional like hook point uh that they have added for the netkit device um which allows it to work much faster. Uh it kind of like hooks the packets at an earlier spot um uh in the transmitting process. Um, and so in the best case scenario, um, if you if you detect in the BPF program that a packet is going off of the host, you can just immediately redirect that packet to the host Ethernet adapter and there's no additional processing that needs to be done. It just immediately gets transmitted out. Um, so like that's like the best case uh performance scenario. Um, you know, that being said, like for other cases that the BPF program doesn't handle the traffic, like there still is a performance benefit. Um, but I think yeah, like to get the absolute best performance, it's like the combination of the netkit with the BPF programs like picking up the traffic earlier. >> Any more questions? One last questions. >> No. Okay. Well, thank you very much, Mike, again. >> Cool. Thank you. [Applause] [Music]