These bootc are made for mailin' - DevConf.CZ 2026

So, welcome for the last session of the day. Contrary to what the title of that talk will be, this is not a karaoke session. But, it will be a talk about Booty and mailing and mail server. So, quickly, wait. Ah, so quickly some technical problem. So, my name is Michael Cherer. I'm working at Red Hat in the open source program office, newly renamed open source AI program office, previously known as the open source and standard. And before I used to be in Red Hat IT, so people might know me because I've been speaking at that conference since a long time. And so, quickly to explain why I decided to do this talk and not just because I found a good joke to put in the title. That was very important, that's why I started by that, but >> [sighs and gasps] >> first, because that's part of my annual goal. My boss told me that I need to speak about something which will be a new technology and I want something which is useful and solving real problem, so I decided to go with Booty. One second reason, it's related to the news, but I've heard that since we cannot discuss about political topics, so I need to find a political way to speak about ICC and judge being cut off from Microsoft and this kind of thing. Turns out that I didn't found anything a political, so I just remove the slide by myself, but just imagine that if I was dating an ICC judge, I would be in trouble, so I would need to have my own email hosted by myself, which I already do, but just again, tough experiment, if you take a shower like me from time to time, you have to think. And the third reason is because I had to fix my email server, so that's an exclusive look of my current data center. It's just under on desk and yes, there is plenty of dust. And yes, um that's old server, like that's 2014 running some 2006 distribution. The company uh that created the Linux distribution Mandriva crashed. So, yeah, I need to revamp my mail stack, so I can as well do that uh during my work day. Um revamping your mail stack that mean like doing something for SMTP, installing something for IMAP, for anti-spam. This is 2026, so you need to have TLS, you need to manage DNS. You need to handle plenty of other stuff. Basically, that's as complex as factorial, it's just, well, not as fun because it seems that people like to do factorial but dislike doing email, something that I really do not understand as a system administrator. So, today my objective is how can I get something which is cheap because, right, inflation, something which is sovereign enough for whatever sovereignty means, and something which is maintenance-less because as you can see, I already do not uh clean my apartment, I do not update my system. So, I want something which is, well, up-to-date without me doing anything, so I can spend more time discussing on internet, obviously. For that, we are going to look at three things. First one, BootC, main reason why I'm speaking. Second, Stalwart as the SMTP server that I will be using and how we will be integrating that, and the integration of uh BootC and Stalwart. So, let's start about BootC, uh which is used for Fedora Atomic, but I think it's maybe called Fedora Image Mode. Um the branding is not exactly over strong and sweet in Fedora as uh my coworker can explain. But what is BootC? BootC, it's mostly a technology and it's command line. Uh Uh, on the under MIT Apache 2 that's written in Rust. The it's integrated with Fedora and CentOS at the moment and likely with Red Hat, but I'm not supposed to speak too much about product and I do not know how much integrated it is right now. It's not limited to Fedora and CentOS as there is people trying to bring the technology to other distribution. So, there's GitHub.com bootc organization. So, if you don't like Fedora and CentOS, uh, you can get there and get all the benefits of bootc, benefits that we are going to speak about quickly. It's a project that have been donated to the CNCF. Uh, that's pretty important because we think at Red Hat, at least in my department, that it's important to have a neutral ground and CNCF is perfect for that. And the idea is you have an image-based OS, um, which is a kind of innovative, but at the same time I found some old paper from a Google where they explain they were doing like me, which is not updating their system for 10 years and suddenly replacing everything with an image-based system. That's a old paper from 2013, very interesting. They explain how they updated million of Red Hat Linux to Debian. And it's available on the web. >> [gasps] >> That's not a new technology. That's what you use for OpenWRT. That's what you can use with Ubuntu Core. That's more or less what is done with Nix where you build the system and reboot on a different prefix. Uh, to explain what is different from a classic installation of a distribution. So, usually you get, uh, well, a DVD. It's no longer like when you could get a floppy and install a Linux distribution. So, you get your DVD, you fill the, you answer the question like, "How do I want to partition? What package do I want?" And your system is assembled on the spot. It's kind of like when you go at IKEA, you get your package and then you build your bed directly at home. With image based ways, that's basically the same idea, but someone else build uh build uh the bed for you. And then you get a build uh uh sorry. >> [snorts] >> A better build uh directly deployable in your room. Uh the only problem is you need to reboot on the new image cuz again that's based on the image, so you boot on the image. And beside rebooting, there is plenty of advantage. First one, it's a read-only system. Not read-only as protected against modification and signature. That can be done, but not by default. I think there is a talk about that either at Flock or today that you can look at that. It's more like protected against while you are a sysadmin, it's Friday, you fix something, then you go to the pub and get drunk because you're a sysadmin, and on Monday you forgot what you did. 6 months later, you figure that something was changed, you do not remember what. With a read-only file system, each time you reboot, everything that you modified is wiped. So, you are sure that everything is committed. Well, it's also possible to roll back an upgrade. You discover that there is some problem, well, you roll back, and then you can find what it is. It's a byte-for-byte copy. So, all sysadmins have a test environment. The most conscientious have also a production environment. With a byte-for-byte copy, you can be sure that your test environment and your production environment are running the exact same package and same version. Which um is usually not at all not the case. And which means that if there is a regression, you can look at the configuration, you can look at the hardware, but not usually at the software. And there is a whole atomic update cuz we named Fedora Atomic due to that, not just because of the song of Blondie, where you update everything or you update nothing. It's not like I just updated the library and not uh the utilities, and now there is weird error. And it's also offline update. Um, so I do not have a driving license, but I know that if I need to change the engine, I need to stop my car. And it should be the same for your Linux system. You should not update, for example, Firefox while it's running because it can break. You should not change the kernel while it's running because it can break. And by doing things offline, you avoid a whole class of problems that you sometimes see with live update and that you see at scale. Um, I have some good story from a when I was in IT, but I do not have time for that. Um, it only manage uh /user. Uh, OTC is managed with a three-way merge like a Git, so it's kind of complicated and surprising, but it's also already what happened with RPM. I just mentioned that so you can be aware. And yeah, and the idea of boot C, it's for bootable container, which means that uh is based on OCI format, the format used by Docker Podman, which bring two advantages. First one, it's standard storage. Um, you can use quay.io, you can use hub.docker.com, you can use a registry from your forge like github.com, gitlab.com for your self-hosted forge. You can use something like a Pulp uh to host your registry. It's like really standard. You do not need any support, which is quite cool there is a whole ecosystem for distributing your image. And it's also standard build tools. So, that will mean Dockerfile. And if like me, you do not like Dockerfile, that's okay, you can substitute with something else. And again, um you have existing technology. So, something that most sysadmins nowadays understand because Docker is like more than 10 years old now. Now, and okay, so that's basically for boot C. That's just a way to have a container that you can boot. Now, let's speak quickly about Stalwart cuz I did not want to do just a simple hello world with boot season, not that interesting. So, what is Stalwart? Well, it's that's part of what I would call the modern mail movement. Back in the day when I started doing email means that you need to do install postfix, then you need to configure something like Dovecot, and you need to configure so they can discuss together, which means oh a shared uh shared user database, and then you need to configure DNS, etc., etc. And people were like, "Yeah, this is too complicated." And I agree, this is too complicated, that's why I'm being paid to do that. And people started to fix that with various solutions. The first one is Mail-in-a-Box, where people just preconfigure a VM, and they give that to you, and they say, "Sure, you can use VMware, and here is your mail server, and that's it." So, it's still complex, it's just hidden and preconfigured, and not really that much modern. Then people created a tool called Mailcow, basically the same, but instead of a VM, that's a container. So, that's still like a dozen of uh process running in a single container, but just do not need to deal with a virtual machine. But that's not really an improvement. Then someone came and created a tool called mox, which is a single binary that does everything. Like you have SMTP, you have IMAP, it negotiate uh TLS with Let's Encrypt, so you can have certificate, etc., etc. The only downside is Well, it seems [snorts] to be no longer maintained as of January this year. The main developer was uh working on it thanks to European Union and it stopped, so he decided that he had to pay his rent and get food, so get another job than working on mox. Well, the tool is pretty full-featured, it's usable, it's just like I do not know what it's I'm not sure about its future. >> [gasps] >> And then came Stalwart, Same idea as Mox, except that it's written in Rust, which means that it's maybe better. I think it's better for something network facing. But I do not know if there is people ready to say, "Yeah, Rust is great." in the room, so I'm not going to speak more about that. And it's also has a better financial situation. So, it's using sponsoring, GitHub, etc. It used a version which may or may not suddenly stop. But it also managed to get a partnership with Nextcloud and Mozilla. So, there was one single developer that created his own company. And Mozilla is using Stalwart for their Thunderbird Mail service. I know that Nextcloud is going to do something, and I met some people in France also using Stalwart for all uh Gmail competitors, so there is at least some way to make money and sustain the development. There is also an enterprise version, which first kind of annoyed me because that means there is a dual license. Uh you and the CLA. Turns out that the guy who created Stalwart created a different CLA, one that says that if you stop following free software principle, then all the code that we gave to you go back to people who have it. It's basically something to prevent MongoDB type of situation. That's a fiduciary contributor license agreement that was done by the FSFE. So, if you know people from SFSF FSFE, you can ask them about that. I think there is a someone presenting about the book from FSFE president. You can go discuss with them. And right now, the developer do not accept external contribution at all because he has a vision. He wants to release 1.0, and he say, "No, it's going to be my way, or you're going to wait." That's kind of annoying, but it's like not completely shutting. He gives talks at FOSDEM. He go discuss with other people, so he try to build a Uh so yeah, Stalwart has plenty of integrated feature. For example, there is anti-spam integrating with SMTP and with IMAP, which means that you can tag from IMAP something as being a spam and it will directly go to the anti-spam. It has a groupware because why not? And then so manage DNS and certificate, which is usually a big problem for people. This is 2026. We need to have TLS for all SMTP server. And for the record, I don't on my current stack, but I know that I should have. And everything is directly managed there because yeah, we can have automation for that. We should have automation. And it has a flexible storage. And I want you to remind that and to remind you to remind that. That's a picture of Anton Chekhov. You will see later why there is this picture. And now let's go to work. You are not here for me to speak about Stalwart. We're here to see how we can use Booty and all the problem I had. So first, I say I wanted something um some sovereign hosting and something cheap. So important to be cheap. I don't want to spend more money than I will spend on I don't know Gmail or something like that. And well, for sovereign, I will basically mean that something which is not used by a US hyperscaler. Ideally, I would like to have something where I can just use Booty image, but for now, nothing is available. So I need to upload them. And well, that's kind of annoying. That's more automation that I need to write. So instead, I used Booty install to existing root. One of my co-worker One of my wallet co-worker, Fabio Alessandro Locati, wrote a blog post on we use Booty on Scaleway. Um The idea is you boot on anything and you just erase the system and you put Booty. That's one way to do it. Uh if you do pay attention on that option, so you inject the right SSH key and do not do like me and lose 1 hour trying to figure why it was not working. I decided to go with OVH um because Gateway was already done by my co-worker. So, we got that's a funny company because I have a proper 2FA on the account, which means that if I lose my YubiKey, I'm not locked out. So, I have a two YubiKey plus my phone plus another indicator on my phone plus we are really I'm very paranoid with two-factor authentication. I can also pay directly by SEPA, you know, in case my Visa card get cancelled because I'm dating someone from the ICC. It has an open stack API, which means that I can do plenty of automation and it offer more service and that's where check off come back. So, building the image, I decided to go with something simple, uh just GitHub action. One of the issue of the Booty project is people are not choosing enough GitHubs. So, well, GitHub works well for that. You just Docker build, Docker push or if you are corporate, you can do buildah build and buildah push. And it's also free, at least for now. When they going to charge for my $20 of uh CI usage, I will move somewhere else. And yeah, you can push to GitHub with SSH because again, that's free, but you can push where you want like quay.io, hub.docker.com, etc. etc. I decided to go with a minimal image. I want something cheap, which means using the least amount of uh storage, especially since it could be free, but it for now it's free, but I do not know what are the limits. Then came the first problem. How do I deal with the update? The simple solution is you rebuild your container every night. But if you are already use GitHub action, you know that do not work because if you do not commit anything on a GitHub repository, they disable action or at least automated action after 3 months. Which means that every quarter, I need to push a button, and I want something which is maintenance-less. That means that I need to push zero button. So, I decided to work around that by abusing Dependabot and Renovate, because it turns out that if you add a hash on your Dockerfile, uh Dependabot will make a commit. And since I want to be as maintenance-less as possible, I went full you only live once auto merge mode. Yes, I know that could be a problem for supply chain attack, but if we have supply chain attack on Fedora, we have bigger problems than my mail server. So, I decided that's good enough. Um that's also solve another problem with upstream image tagging and um uh pinning. Because right now, if you pin on a specific hash and there is a new version, you cannot download the old one. I discussed with upstream, and they're like, "Yeah, that's something we should fix." Well, it was not fixed yet. So, yeah, I need to have the update because otherwise I cannot work on my Docker image. Well, that got solved by another problem, which is we have days off in France. And you're like, "Why is that a problem?" Because one guy, someone that I know in the office, I live in Paris, I go to the office, is the one that accept every build for upstream image. And when it take 1 week off, well, there is no update. I discovered that because I was starting to notice that there is a problem. I ping him on Slack, and he's like, "No, I'm not here." And when I I come back, I discuss with him, he's like, "Oh, yeah, because I need to approve every build." I'm like, "Mhm, that's a problem." But >> [sighs] >> yeah. At least that was not because there was no update, I didn't need to update uh for at least for 1 week, but otherwise that's something to keep in mind. So, if you get the minimal image, you have no network, no SSH. Again, I spent like 1 hour trying to figure why I could not SSH to my image. Uh, I decided to go with systemd-networkd because it will consume less resource than network manager. I think network manager is nice, but that's something which is getting new feature all the time and they change their uh, storage system on a regular basis because that's innovation and you do in a better way. systemd-networkd is always like one config file that didn't change since years. It will use the same library as systemd, so it consumes slightly less, uh, space on disk. So, I went with that. So, of course, if you install anything, do not forget uh, the DNF option to not pull the all the bounty. Like, I do not really want to have libwayland installed on my server and I do not need that. And I also discovered that in the minimal server, there is no preset policy, which means that nothing installed was starting by default. So, you need to create one. You need to make sure that you run systemd preset all at the end. And while you're at it, you need to apply two timer. Uh, one to apply uh, new version bootc fetch apply update. That one is quite interesting. I do not recommend to use that on your laptop because in my case, my laptop is locked on code, so I did that and I woke up the morning, my laptop is blocked waiting for my password. And there is the Podman auto update timer, which will automatically update everything we deploy with Podman and we'll see later that we use that. And yeah, there is the question where oh, it when it break, how do you debug? Um, well, you can use Podman because your image can also boot as a regular container or you can use some nice tool called bcvk for boot kit virtual no, boot container virtual kit that can start a boutique container with QEMU directly on your system. Pretty useful for debugging, especially when you do not have access to your hyperscaler machine. As a rule of thumb, everything should go in /user, so you can update and remove. As I said, if you add something in OTC, there is always a risk of merging in the wrong way. So, if it's in the user, that's managed by the image, that's much simpler. I decided also to replace cloud-init with a shell script. I like cloud-init, that's a nice software, but it's just in Python and it's huge, like 100 MB. That's too much, so I just decided to use curl and jq for getting my IPv6 and my hostname, and the hostname is not hostname change is not working on OVH, so just getting IPv6. And I decided to use Caddy as a front proxy, cuz I want to have a web mail, and I want to use a web interface for my SMTP server. And that's where I started to get fun. First, because I discovered that Caddy is using sysusers, like everything, like plenty of stuff in Fedora. And sysusers create a user on installation. But it do not create always the same user. Sometimes it give a different UID. And not for my mail server, but for cluster infrastructure, I discovered that sometimes it get a different UID, which means that it cannot read the certificate it was generated 1 days ago, and everything break. And in that case, it was broken just as I went for 2 weeks vacation, so yeah, not cool. Yeah, that's something you need to keep in mind. If you generate on the fly, users they need to be generated with the same user ID, or you need to do some uh owner change for some file. So, I did fix with a TMP file to fix ownership. I also found that KD is creating file in the /var and bootie do not like that. So, again, you need to either fix the package to create uh file it needs in var on the fly or use the TMP file. And I discovered that because there is a bootie container lint uh option that will lint your container to warn you about something which is broken. >> [panting] >> But uh that one can be broken without me doing anything. Like, it's not verified in the CI, so sometimes you get a bootie image and it's already giving you warning. So, I decided to disable it. Uh but it's sad. And for example, it was broken each time I had a systemd update. So, since I installed systemd-networkd, it installed a new systemd. Then systemd created a seed in var-lib and then uh bootie complained about having a file that needs to be recreated. >> [gasps] >> So, yeah. Um not perfect yet. Then there is a question of debugging tool. Do I need to install them in advance? Do I need to install them when there is something? And since I need to reboot to get a tool and I need to reboot to remove them, that's not exactly cool. Well, turn out there is one trick, which is a bootie user overlay, or one trick with DNF. Uh someone spoke about it today, but I did not want to fix my slide. So, you can switch your system to not be read-only to have a overlay in memory, so you can install whatever you want without rebooting. And when you reboot, everything is wiped. That's really great for debugging and really something that should be explained more in uh the documentation. Like, I knew because someone spoke of it in the CentOS Connect in Brussels. I think you should not have to travel to learn about it. And yeah, and then I went with Quadlet and uh for Stolwort. Um I was ready to say yes, everything is ready for production in 20th of April. This is not a marijuana joke. Well, Stolwort developer decided to like, "Oh, before 1.0 I want to trash everything and redo the configuration." And I uh So, yeah, it was not ready. I have to rethink the whole system. Before it was like one config file that can deploy and we start the container. Now it's going to be much more complicated. So, instead of working for that, I decided to see if I can go Distroless. Which was a bad idea cuz yeah, sure I can rebuild with all feature removed. That means that I have to wait for 1 hour and a half for building the system. But uh my upstream build went from a three 300 MB to something much smaller. But it was also much smaller because for example, TLS was broken. Yeah, if you go Distroless, you need to deal with your certificate by yourself. It was also broken without log because the logging was broken. Logging was broken because yeah, if you go Distroless, you do not have /var/log. There is no syslog and this kind of stuff. And it was also broken because I could not access to the admin interface because yeah, there was always no /tmp. So, yeah, do not do like me. Do not try to think that you can do a better package than upstream. Usually upstream has something that work. Or if you go like me, like be prepared to debug without logging, without anything. Even if now I think about it, if I had used Podman right from the start, it would have been I guess better. So, what do I have on my to-do list? Um well, first security. Um that's quite important. I know that my boot image is running with SELinux, but Caddy is running unconfined. and uh Stuwart is running in a container so with a generic um policy. I'm not exactly sure what will be the best practice here. Like, do I need to create my own uh SELinux policy and ship them with a RPM when I build the image? Do I need to do that with a container? Do I just figure that Yeah, that's going well so no big security problem? Like, yeah, that's quite unclear. Then there is the issue of firewall. That one is more interesting because Yeah, I decided to add a firewall. I can just add NFT table, and with NFT table I create my config file, I drop it, I start the service, and well, Podman bypass uh firewall because if you start a container and you do port forwarding, it goes before your rule for filtering, and yeah, you just forward something and your firewall is not usable. So, yeah, that's something that you need to keep in mind. Uh how do deal with firewall with a container-based deployment. Then there is a question of uh quadlet and user group dynamic user because right now Stuwart is running as root. I didn't found a good way to run it as a non-root user, and I think that will be much better even if it's running There is nothing on the system. So, if someone compromise my mail server, they would have the access to my mail, and that's the only thing on the server. So, maybe that's not a big deal. But if like me you want to maximize what you do with your small server, yeah, that's something that needs to be uh well, you need to find a solution for that. And there is the issue of partition management. I wanted to have a separate partition for my VM so I can have my mail and have the system that I can trash. And just with systemd, there is nothing good. Like I cannot tag my partition to say mail and have something inside the container to mount it at the right place. I need to configure everything by hand and it's not portable because depending on what cloud provider you have, it will not be the same device. So >> No. >> Um something that needs to be fixed at least on OpenStack level. There is a big question of config management. Do I ship the config management on my image? Do I ship it outside? How do I do that? And if it was just like my domain name, that would be easy. Like I do not care about domain name to be public. But there is also a secret because if I want to modify my DNS, I need to get not only one secret, not two, but three secret. I don't know why OVH does this way. So yeah, and I do not want to publish my secret. I mean, that's in the name. That's supposed to be secret. So [snorts] yeah, you need to find something for that like secret management. And I need to also deal with backups. Uh right now, with still work, everything is stored in a works DB uh format on disk. Um I guess I will just ask him and hope that it works. So yeah. So conclusion, um did I succeed into getting something without maintenance? Well, I will tell with this I will tell in 1 year when it's in production. Did I succeed in sovereignty? Well, I will tell when I manage to date a ICC judge and see if I get arrested. What I can evaluate is price. Is it cheap? Well, for that I need to compare with existing service. I decided to pick FastMail because uh Gmail is offering much more than I can run. Price for FastMail, it's €10 for 120 GB of storage for two account. If I go for OVH, I need to pay €7.1 for the VM cuz I really like their price. And I need to pay 0.0 euro per month for 1 GB per 1 gig, So means for 10 Euro I get 72 gig of storage. Well, that means that FastMail is cheaper. But, as I said, following the chicken curve principle, if you speak about something in act one, you need to use it in act three. I said that systemd can use S3 storage. I also said that OVH offer plenty of service such as OVH object storage, which means that I can get something cheaper, like 1 GB it's well, less than 1 cent per hour. That's 0.07 Euro for regular months, like uh 30 days months. It's cheaper in February and more expensive half of the time. Which means that for 100 GB, I can get that for less than 1 Euro, which means for 8 Euro per month, I can get something as competitive as FastMail. And I can get something as competitive as FastMail if I pay month by month, but if I get the rebate over 3 years, well, that will be 10 Euro.27 cent for six user with something like some amount of data per user. And that will be 9.6 Euro on OVH if I can host six user with 2 GB of RAM. So, yes, because systemd is like 100 MB, kded is 52 MB and systemd and other stuff is like 20 MB. So, I have like 1.7 GB of RAM for six user, that's plenty enough. Conclusion of the run, yes, we can get something cheap and maintenance-less with bootc without too much trouble and that is really competitive. So, if you want to see how I did it, I have one example um on GitHub. You can see all my fixing commit and this kind of stuff. Do not reuse that image. Try to get some inspiration about that. And thanks for listening and I think that I have like 3 minutes for a question. So, yeah. >> Um >> Do you >> Atomic are as I understand the host for all of the steps that have been more atomic very right. >> Yes. Well, I I'm not sure about the burning, but yes, I used a Fedora for the base image. Oh. Well, Justin asked if it's based on Fedora Atomic. Is that your question? >> Well, I asked cuz I'm looking at CoreOS and I'm just trying to understand why we would have chosen say Fedora CoreOS versus some of the Fedora Atomic. >> So, the question is why did I use Fedora Atomic and not Fedora CoreOS? Well, because I have opinion Fedora CoreOS have opinion and they are not the same. So, I wanted to do my way. Um yeah, Fedora CoreOS would have been a good solution, but uh my objective is of speaking about boot C not CoreOS. So, that's what I got. But, I thought CoreOS uh come with its own uh ignition system when they are own opinion. I wanted to get something which is well, sustainable. I know that Docker is not going away. I know that I understand Dockerfile and I know that I can build something with Docker and GitHub. Basically, that's that. I wanted to do my own stuff. But, CoreOS is also a good choice for that. Or just a regular Fedora is like running a static binary. It was really me trying to do something complicated where practice it should not be needed. Yeah, I got again questions. >> I understand, right? Kind of Fedora Atomic as a technology that we as admins have the same kind of opinion that CoreOS has on its how it ships. >> So, the question is >> technology, >> So, yeah, the question is does Fedora Atomic have the same opinion as CoreOS? No, because I use the minimal image. The minimal image is the one where there is nothing installed. So, no opinion. Yeah, but they give me the ability to write basically my own variant, which CoreOS people did and they made some specific choice for their use case, which is well, running OpenShift or something like that. It is just that I figured it's easier to start with a Dockerfile. Any other questions that I need to repeat? >> Nope, no question. >> Well, thanks for listening. If you want to contact me, that's my email or you can find me on IRC or since this morning on Matrix again because I fixed my Synapse server that was broken after Fedora 44 update. Um Well, that's it. >> [applause]