Flock 2026 From Upstream Pytest To Fedora CI: Modernizing Shadow Utils System Testing

Okay, thank you. So, hello everybody. Well, in the context of uh operating systems, just like in critical safety software, we rely on rigorous testing and certification. We need absolute certainty that a change won't break a system before it hits production. Well, in Fedora, that gatekeeper is our CI pipeline. Before I joined the identity management team at Red Hat, I used to work in the automotive war and there I was responsible for the eco anda services which is which are in charge of calling the emergency services in case of a car accident. That job took me on a journey that I will never forget. I was standing in Russia in an official regulatory certification for automotive emergency services. The air was tense. The stakes the stakes were high. Millions of euros and months of work were on the line. The inspectors initiate the test sequence. We watch the diagnostics monitor. The diagnostics monitor flashes red. Test failed. We failed the certification. At this point, my stomach drops. We are thousands of kilometers away from home with broken software. Digging into the failure, I looked down at the official certification specification document in my hands. Well, the entire test document configuration is right there on paper. We could have easily built and run this exact same test on our workbenches back home, but nobody had taken the time to do it before we boarded the plane. No matter how good your code your code looks on your machine, if you can't effortlessly simulate the exact certification environment locally, you are just gambling. Discovering a failure at the final gate is the most expensive way to write software. FedoraCI is our ecosystems regulatory validation facility. As packagers, we shouldn't have to push a change to the cloud, cross our fingers, praying it passes the gate. We need absolute certainty and perfect symmetry between the remote gate and the developers desktop. And that brings us back to shadow utils to ensure our developers never have to experience that drop small dropped stomach feeling that I felt. We completely overhauled how we test core packages. Let's look at how we left the dark ages of fragile old sh all old bash test behind and how we brought the true power of Fedora CI right into the developers desktop. So as I was mentioning before uh we will start with the dark ages of the old bash test. Then we'll move to the uh upstream refounding where we used pi test image specific uh pi test uh plug-in and shadows test framework to run all the test upstream and well next we'll check how we use tmt and testing farm to do the bridge with speed federi we'll finish with the wrap up okay to understand where we are going we first have to look at what we were one running away from let's step back into the dark edges of shadow utils testing where trying to verify basic system felt like tools felt like playing Russian roulette with your operating system. First thing to know the developer had to remember to call the tear down phase to restore the system. This wasn't done automatically after each test. If a test failed or crashed unexpectedly, if le it left critical corrupted files behind. All of this felt like the fuse in a bump and it created a massive gap between what the uh project expected and what the contributor a new contributor knew. So it scared them away. The true irony of all this situation comes here. uh we are working with shadows. It takes part of the core identity of almost every Linux distribution and the upstream validation happens only in one specific single OS YUbuntu. The test suite was rigidly tied to this distribution as it compared complete system files and the baseline came from YUbuntu. If you were a Fedora packager trying to ensure compatibility or test a downstream patch, you couldn't do it natively on your own system. You had to rely on Ubuntu uh and how on and all of this working together. So here comes the requirements for a new modern system test framework. The system should be able to run anywhere with a single command. It should also handle cleanup automatically after each test case. It should collect logs and artifacts automatically so that any failure that happens in a remote system can be properly diagnosed and it should run in a wide variety of distributions hence making it universal and that's where the upstream refounding comes by test image and shadow test framework. So in reality this is not just a new language but a modular architecture. We have three different things here. First of all, we have the test themselves that test the shadow uh framework. Then we have uh the second layer which is the shadow test framework that takes care of the specific domain knowledge of how the utilities from this package behave. And third, we have the piest mh as the engine that manages connections and environments. All of them together they completely replace the old shell script chaos. Do you remember the fear of breaking a machine due to failed cleanups? Well, before a single test executes, the framework takes an instantaneous snapshot of the systems identity break. In this case, we are speaking out about etc password shadow etc group etc etc etc. The best part of this is that it doesn't matter if your rest fails, passes, or it throws a catastrophic Python exception. You always get the same pristine and and health environment after the test finishes the execution. We could say that the system completely healths itself after the execution. And this happens for all the tests that are being executed. In the dark ages, verifying if a user was added meant executing user at and comparing complete file systems. We brought native Python wrappers for core binaries like user ad, user mode, groupan and so on. And nowadays writing a test is highly readable. It looks like a standard clean Python code. This drastically lowers the barrier to entry. So we have new contributors uh coming and giving their part by writing new test or transforming the old test to the new test framework. In fact, you no longer need to be a core shadow maintainer with decades of experience to write a perfectly secure system test. Okay, what happens if a utility behaves destructively during a test run? Well, the this is the core magic of pi test image. It decouples the test controller from the target system under test. So in reality, you have here two two test systems. The test run inside isolated remote disposable environments. We set up ephemeral containers or shortlive virtual machines on demand. All the destructive changes occur inside that throwaway environment. We guarantee absolute protection to your local host workstation. You can trust extreme boundary conditions with zero risk to your machine because the framework neatly abstracts the system interactions and targets isolated environments rather than the host system. We unlock through distribution agnosticism. The exact same test suite executes seamlessly across Fedora, Alpine, Debian, and Open Sus. This is currently happening in our apps, PRCIs. And why does this matter to a room full of Fedora packages? Well, because if a package change introduces an edge case regression that only shows up on Alpine Muscle Lipsy, our upstream framework catches catches it before it impacts the distribution ecosystem. And I mentioned alpine muscle but I could have said federas gypsy. It doesn't matter what you are using there because it will be catched by the upstream PRCI. And here comes the part where I guess most of you are interested in. It's about TMT as 13 farm and how we bridge all of this with FeroraCI. As downstream maintainers and packagers, the last thing we want to do is to maintain two separate test words. One for upstream development and an entirely different one for for the downstream development for Fedora, for central stream, for rail, for whatever distribution you are working on. This is a recipe for divergence and unttracked bucks. By bridging the gap with TMT, we completely bypass duplicate work. We effortlessly pull the same exact upstream by test framework we just looked at and execute it inside federal gating infrastructure. Upstream features and downstream packages updates are validated by the sec same exact source of truth. And here comes TMT. Instead of writing fragile Jenkins files, complex B glue or bspoke CI pipeline scripts, TMT uses simple declarative metadata. You look at the configuration and instantly understand what it does. In this case, we are simply declaring a summary of what we are testing here. The environment to provision some preparation steps like hey, here you have the URL with the shadow test repository. And finally, we have the execution phase where we run the actual test. TMT handles all the other lion pipeline orchestration for you. You don't need to care about it. As we established, testing user ad or changing the passwords, requires root permissions, and mutates foundational configuration files. You cannot safely run these tests on a standard S CI build runner without potentially breaking the host or poisoning the next building line. Well, testing farm solves this infrastructure harder for you. It couples with TMT to dynamically set up completely isolated dedicated disposable virtual machines in the cloud for every single pull request. The destructive test runs in a sandbox verify the package and the entire virtual machine is completely deleted afterwards. And these slides represents the core resolution to my automotive certification story. Do you remember the frustration of discovering a failure late in the development process at a remote facility thousands of kilometers away from home because nobody took care of to test against the those parameters locally. Okay, with TMT we completely eliminate that barrier because TMT beautifully abstract the execution layer. You can run the exact same command on your local system inside a local VM or container that testing farm that that testing farm executes on the bare metal cloud infrastructure. It's just the same if a package fail build fails it's getting checks in federal CI you don't have to guess or push blind coms to debug it. You can replicate this locally on your local machine in minutes fix it and push with absolute c certainty. Okay, when a destructive system test fails out in the cloud, you can just SSH into it or connect into it in some way because well, as I mentioned before, you have a disposable virtual machine or container and it's been deleted. But TMT and testing farm ensure you never fly bind by automatically collecting a compre comprehensive diagnostic suite. We have three pillars here. First of all, you have the infrastructure logs from TNT and testing farm to verify that the environment was set up correctly. Next and and most important to us, it captures the raw system logs. In this case, we are speaking about bar lock messages and barlo secure. This is where shadow prints its debugging information and where you can see where the error happened. And finally, you get the framework logs containing the full pest execution std out and stack trick specs so that you can check them and see what exactly failed on the test that you were running. Debugging a remote failure becomes as simple as reading a well organized local directory. So wrapping up, we've gone from the fragile cell scripts to our robust self-hel uh upstream framework and finally to a declarative TNT configuration that gates Fedora builds automatically. When you look at the entire journey, the takeaway for everyone sitting in this room is clear. To tie everything together, we successfully moved away from the dark ages of fragile execution and manual cleanup anxiety. By building on top of pestm, we established a safe self-healing upstream framework that treats multiple Linux distributions as first class citizens. And by bridging it with TMT, we brought perfect local to remote symmetry straight into Federal CI. The ultimate lesson is clear. If we can safely and cleanly automate the the system testing for shadow utils, the literal funding of system users and permissions, then you can confidently leverage these exact same tools to get your own package. But we aren't done yet. So the engine is built, but we need to fill the tank here. Right now, our new framework has 100 robust tests, but the old legacy suite we are replacing has 1,000. So, if you want to practice modern Python testing or get familiar with TMT or just help secure a critical core package where the framework handles all the SK parts for you, come join the shadow team. We are waiting for you. So I'm here leaving the QR code to the URL where the shadow uh um repository [clears throat] lies and you also have my contact information in case you want to contact me for anything. Uh thank you for attending and in case you have any doubt this is the right moment to do it. >> [applause] >> Thanks for the great presentation. Uh I'm wondering like you have to migrate a lot of tests. Some of them are mine actually [laughter] like the old batch test uh from my start in Redhead. [snorts] That's funny. Uh but um the I was wondering are you planning to use maybe some uh AI help to gener to basically move all those tests around? Maybe that could that could help a little bit because thousand tests it's a lot right? >> Um yes I was wondering to do it but then somebody came and opened an upstream ticket uh stating that we don't want any AI help. >> Oh wow. >> Yeah. So we were discussing this topic and we are ripe to some kind of agreement that we will not use it to generate code but we can use it for other purposes. >> Yeah like I I think like ordering tests when you already have 100 tests. >> Yeah. >> Which provides a good context right. So I would definitely try it because it's a lot of work ahead of you. Thank you. >> Thank you. >> No more questions. Thank you very much. >> Thank you.