Flock 2025 Testing Farm Test Environment Setup Profiles

noise. Uh hello everybody, thanks for joining our talk. Uh my name is Mirro Slavatkerti. I'm here together with Yan Havlin. We are both developers on the testing farm team and we will be presenting today about the testing form environment profiles. Let me reset the counter. Uh so yeah, this is what we have prepared for you. Uh in the beginning we will do a short introduction about testing form and TMT. We will look at the uh test pipeline in testing form to see how testing form uh handles test requests. Then we dive into the details of test environment setup before running the tests. Uh we're running the actual tests and uh we will look at some long-term issues we have with it and how to plan to solve it with uh this test environment uh setup profiles. In the end, we share the current status of the implementation in TMTN testing form. And yeah, we will look at how we plan to roll this out to federal CI, central stream CI and packet and it also partially applies to LCI if you are a employee. So just a few words about testing form if you do not know uh testing form is the testing back end behind Federraci streams 3II and packet and it's basically unifying the the the testing experience for the users of the CI systems. Uh the service itself exposes a single HTTP endpoint that is public and its main input is a git repository with TMT metadata which basically describes the details of the testing environment and selection of the tests to be run on those environments. Uh we are focusing on testing uh or running tests for the operating system mainly functional tests of the operating system right so we are provisioning VMs and experimental machines. Uh yeah and uh next to it for the federal contributors we are providing an easily accessible way of reserving a host and connecting uh via it SSH. It's easy as those four five steps. You need to first install the tool on board uh to uh get a token as a Federa uh with the Federa single sign on and and just get your hands on a hardware that we support. Uh yeah we you can also request BPVMs bera hosts and GPU instances and I would very much like Amazon to sponsoring Fedra to provide this hardware to you Federal contributors and making the CI better. Uh a few words about uh TMT itself. Uh yeah TMT is an open-source test metadata format store in Git repository and it's also a convenient CLI tool for discovering developing and debugging tests for the operating system. uh with TMT we were able to unify basically how the operating systems uh operate the the tests for the operating system are defined and shared across uh like projects uh upstream projects Federa testing central testing but also rail testing rivals testing so it's like a unifying piece uh for the operating system tests it's packaged in Fedora it's really easy to install it in the example below you can see how easy it is to actually run tests for bash component which is using TMT. So it's like really like super easy. I will hand over now to Hanza. Uh so I'm going to briefly describe the testing pipeline that testing farm executes to point out the focus area of this talk. Uh as Miro said testing farm exposes a single HTTP API for users to request test execution. So uh users have to provide uh description of environments they wish to run tests in and a git repository URL with TMT metadata and once a user submits a request to the API testing farm pipeline will begin its execution. Um I'm sorry uh sorry the execution pipeline consists of several steps. First the requested git repository is cloned to uh discover TMT plans. Um then a testing matrix is constructed. So each plan each TMT plan is executed on each requested environment and uh the the plan execution consists of provision provisioning a machine running guest setup or environment preparation executing the plans via TMT and reporting results. Uh in this talk we are focusing on uh environment setup or we call it guest setup as a legacy name. It consists of three phases. Pre-artifact installation which is implemented by an un anible playbook. Artifact installation for example installing Koji build or other builds and post artifact installation which is again an univil playbook. uh and the main focus area is the first phase the pre-artifact installation. Uh this part is implemented in anible as a single playbook as I said and import it's importing some task files. We are maintaining the code in our infrastructure repository close to our infrastructure as code. Um in testing farm requests we are exposing logs from the unsele execution in the results viewer. So it's quite n a bit nicely discoverable by users for the execution. Um some examples of the tasks are updating the system and restarting if necessary installing cooji so we can download cooji builds later on uh and install the RPMs. setting the time zone, enabling some other repositories such as appel or others and so on. Uh the current imple implementation has couple of issues. I'm going to briefly describe them and uh say how we are approaching the issues to mitigate them. Uh first thing is that they are a bit hidden in our infrastructure as code and they are not easily contributable by users and uh they are being imported as tasks and they are not modularized nicely. Uh as a solution we are moving them in a separate repository called profiles and we are rewriting them into rewriting the tasks into anible roles and structuring them into collections and we will say more about the structure in the next slides. Um, another issue is that currently the guest setup or the environment preparation is implemented by a single playbook and it's used by all our users or CI systems and each CI systems has different requirements which results in messy code with uh many when conditions and which are based on the image that's being run on. So as a solution we are splitting the logic into separate profiles based on the user or CI system that's executing the request and each profile will be implemented by an un anible collection in the separate repository. As a result, it will be more easily contributable since the contributors will be able to focus on just a single profile that's interesting to them instead of the whole thing. That's getting quite quite over the hand. Um, next issue is that testing of the guest setup playbook is done as a part of our uh, infrastructure integration tests and we don't expose them publicly due to some credentials and other sensitive information about our infrastructure. So since we are moving it to a standalone repository, we are enabling CI just for these profiles and we are able to make it public. We will show it. We will run it in public branch and it will be contributable by users from forks if they provide their own uh testing farm token to the forked repository. Um another issue is that the current implementation is not easily executable locally using TMT. So we don't show we don't include this part in the TMT reproducer script which is available in every testing farm request. So that means the reproducer script is is not matching the testing farm production. So it's not possible to reproduce the same environment locally. Uh so uh we are distribute we will be distributing these collections using anible galaxy and RPM packages and they will be consumable by TMT uh TMT prepare plug-in and this will allow users to to repro to reproduce it locally. We will be able to include it in the TMT reproducer. So the environments will be closer more close to the production. Um currently it's not possible to change or disabled the test test environment setup. So it's not and it's not so clear what's the what's the playbook that's being executed. the code is a bit hidden and we will allow setting a guest setup profile with via testing farm API or in repository configuration because in the future there will be no preparation by default users will be will have to set it themselves and we will also have to improve the visibility in our results viewer and add a link to some documentation because currently there is no dedicated document doumentation. So we plan to add some documentation to the collections and show it in the unseable galaxy. Okay, I'll hand it over back to me. Okay. Yeah. So uh as once I described for each this use case or the CI system we will have a dedicated anible collection under this testing farm profiles name space and yeah that will be basically providing a unified playbook uh apply YAML that will be basically the entry point to like set up this profile on the on the machine right uh yeah there will be separate separate collections separate profiles for the central stream CI federa packet and it will be easy to add new new profiles So if somebody wants to have their own dedicated profile whatever we'll provide good support for that so they can fine-tune basically the test environment setup as they need like this was not possible before. Um yeah this setup we also make sure that only the relevant setup for that CI is done. So and it's vetted by by hopefully the users. We would like that the uh the the federa CI and the stream CI users will be able to basically change the setup if if they want to and there will be some consensus. uh there was also some discussion if we shouldn't not move it even closer to the CI systems maybe to under the in the repositories of the users like packet or federa CI but for now we will use this single repository to maintain a good CI CI setup um yeah and uh as you can see like that's the layout of the of one collection uh yeah there is missing it's brief for the presentation there is definitely missing some record >> require record missing >> three collections Yeah. Yeah. For each for each profile there will be >> yeah one collection I see >> uh there on the picture there are three collections for the different users I think things uh and yeah there are some missing files like the uh required collection collection setup. Uh one thing that I want to point out is that um we will be having uh basically a TMT plan which will run validation test via GOS. So if you don't know go it's a simple server validation framework similar to server spec or test infra those are like or rspec uh and basically that verifies that the profile was applied correctly right so we will have this good CI setup and you change something uh there will be no surprises that like something was wrong broken um yeah this ghost test will be currently run only for CI purposes when yeah there will do changes to the to the repository uh yeah and the the profiles will be probably tested on all supported that composes. So yeah, really contributing should be easy. There should be mitigated any any possible things of regressions. Um yeah, next to that these profile collections for each user, right? There will be a special collection called library uh which will host roles for specific setups as uh as Hansa showed on the previous slide. We have set up forel set up for like time zones and so on. So this will be a role under the library collection and it this will be reusable roles which the profiles will then use uh and cherrypick uh actually what they need right and uh yeah uh the the the apply playbook will actually import those roles and cherrypick the yeah the specific set of functionality and uh yeah it is expected that some rows don't make sense in some of the for some of the profiles. they will be they will be not used but otherwise the we are making the the roles uh in a way that if you run for example Apple installation Federai doesn't fail it it just does nothing it informs you that sorry like Apple on Federra obviously doesn't make sense right um yeah and the similar approach uh that's of course the layout of the of the of the library collection similar approach will be used for uh for the for the testing like there will be a ghost test for each uh role uh that will verify that the role is doing what it should be doing. And actually the ghost go the ghost tests from the profile will actually be including this uh or cherry picking also this ghost setup from the from the from the roles. So they don't need to uh they they can verify the function that is defined in in roles or they will be able of course to define additional additional setups or additional checks if needed. uh we spent good amount of time for a well working CI. The example on the picture you can see a screenshot of GitLab CI which uses uh parallel matrix job which is actually generated. So we generate the GitLab CI configuration. So we have one single source of truth of the all supported environments and combination of virtual machines and containers. We are testing these roles on container environments but also virtual machines and we'll be extending this uh for uh for real and anything else any like there are some auto SD images that we use and so on. So we'll be extending the test test metrics more. So uh yeah we made it very simple to add new environments and the required configuration for githubci will be nicely generated. So I'm really happy about that. um on the screenshot with the black background. Uh that's actually an execution of one of the environments and you can see that it's running basically tests for uh it's running uh those ghost tests for each library and each profile. Workarounds is actually another library that we will use for various workaround stuff. It happened for example before that federal row height was broken for some because of something right. So when when when that happens and we need a apply workaround for federa CIS just to make it work because of the DNF5 changes or whatever will come in the future. So workarounds will be a special special kind of collection that we will like or the community will be able to like add workarounds for uh for these specific issues that really need need changes and so on. And yeah uh on the on the right side there is the testing farm output for a specific for a specific uh test where go is running. I think that's for the time role where yeah we are verifying that the the time zone is good and and and there is some expected output from from karon I think um with the setup NTP servers um okay now let's look a bit more concretely how the profiles will be applied in the empty this might be that one was like interesting but this might be more interesting like how it will this will be used right so uh yeah so how the uh profiles be applied by TMT so in version TMT42 to TMT gain support to be able to run playbooks from an anible collection. So that's the first example right. So uh if you do not know you can run a playbook from a collection with this notation. It's normal anible notation where there is a namespace uh namespace test profiles name of the collection federal CI and there is the apply what is the name of the playbook apply ammo right so TMT gain support to be able to run uh these playbooks which are actually like provided by collections installing on the system and in TMT 144 uh there was introduced a shortcut for this. So the the new prepare feature plug-in was created uh that you can directly specify a profile. Uh so currently we have also the the name space prefix here. I think it makes sense but like it's basically just a shortcut to the other other thing. Uh and yeah once uh actually testing form will be this running in the future but also like how you will be how people will be able to run this on a command line is that they will uh they can yeah extend or add the prepare prepare feature step. This is not adding it is completely replacing it. But you get the idea. [clears throat] Uh in terms of testing farm, we'll be introducing an easy way to set specific profile in the environments. Uh for a specific environment, you will be able to specify which profile should be used. Uh so yeah, this is the way how federates and the stream packet and other users will set the profile uh when when they are like testing testing farm. So they will most probably need to have some mapping like that when Feder is used then they will use uh it will be actually their decision like what profile to use and uh that that first part is actually part of our API so it's our API in how how you in the API call actually uh specify the the TMT profile. Uh yeah because we want configurability it will be also possible to override the profile using this in repository configuration that we introduced. So you'll be able to like if your repository needs a different setup, you'll be able to override it uh and uh like get yourself through the the struggles if if the if the profile would be not good for you, right? And uh yeah, the last thing it will be part of the reproducer steps as I mentioned. Uh so in in the reproducer you will again see that TMT commands. So this is basically the last step that will bring you the the configuration that testing form is using to run your test. So yeah a lot of lot of problems in the reproducing should be limited this way. Uh yeah and let's look a little bit briefly how we are will we bring to our main public users. So uh we have finished uh like bootstrapping the project with good CI and currently we are porting the the tasks files that we have to these rows. I think we are like 70% done. All we need to do some reviews uh and I think it will be finished like the porting for public ranch will be finished like I believe in like two weeks uh we would get we would get there once that is done like we will be using we will be moving the testing farm to use these roles instead of those tasks. So we'll be depreciating that old code but testing form will still like like use this old uh a default profile basically that it will apply to the host right and in the meantime we will work with packet federacy and central stream CI to actually use the profile option so they can uh they can select the profile that they will be using and once all users are using or the major users so we don't break users are using the profile then we will drop the drop the old code and and make it uh and and and so on. So in that time testing form will not apply any playbooks anymore. It will be a complete vanilla and do whatever you want kind of thing. I'm not sure if everything will work like like obviously Tento stream 8 and so on will be not even working because the repositories are not working right. So but yeah we will provide this vanilla experience also for the fed for the rail users and internal users. This is one of the things that we want to come up with that there will be a possibility to really get that vanilla kind of experience and for some test it's really needed. Oh yeah 3 minutes we have three minutes still on presentation but that's all what we have for you today. So uh open for questions if you have any. Yes. Uh the microphone >> uh so if writing the tests in TMT I am I'm able to test those locally right uh before I push it to testing farm. uh since you are configuring the environment using ensible uh are you able to automatically run these ensible scripts against the local container environment that I'm using for for local development >> you will need to extend basically or ideally the plan like you you will need to specify which profile you want to use right so uh we are thinking that for example if you submit it via testing form CLI like itself the tool like not speaking about local and testing form CLI can say that okay you are using source federal project or as the source and that is actually federal CL so it will apply by default the profile but for plain TMT run locally I think your plan uh like for the local use case should define the profile but it's all like it's a good point like why should you do that right but TMT would need to know from somewhere which profile to apply like for example if you would have cloned this git repository maybe it's clear that we can do that mapping mapping for you. But I think that's a good discussion topic like again testing form CLI will do that mapping for you. It knows the Git repos are actually setting some for example context and so on automatically. But for plain TMT execution uh it's common that users have their own local setup uh like in the plans using adjust tools to to basically change the behavior to the local execution. So at least for the start it will be required if you will want to use profiles then you will need to add it to your plan and adjust it for the local execution. Uh but it's a good discussion topics if if we should do some automatic finainer provision as well >> and it will work for container provision of course because uh yeah thank you uh >> if you are not a silver blue user as I am because okay that's that's good. we need to fix their problem because it's uh the the answer is run via uh podman unshare and in silver blue when the container is run on the host not in in the toolbox it doesn't work so I'm really not happy about it but like in general it will work like that's we are trying to make the as as I saw as you saw on the on the CI page we are testing it against all container versions uh I think we will skip maybe real [laughter] but feathers at the string will be will be will be tested on on containers allies. Cool. Yes. So what are the plans for backwards compatibility? So let's say we have the profile for Fedora >> and then we want to disable uh installing the uh packages by default or like doing these kind of changes over time. Like if we just specify the uh profile then it will take uh we would have to like change everything all of the packages at once like all of the um this kit packages to uh to install their own uh prepare sub. So like is there any kind of like backwards compatibility? So, so this what we were presenting doesn't deal about the artifact installation. That's something that we hacked into the reproducer steps. It's TMT will be gaining a new plugin to comp completely install the artifact. Just just saying that it's it's not tied to artifact installation. It's definitely a good point. Uh like I I think that where we are getting in is that we will have profiles for the CS systems but it will be not just about test environment preparation. It will be also about more things like uh by default running ABC checks for selenop what is requirement for rail might not be requirement for github actions like triggers right or or testing on github projects or whatever on some things so I think we will get back to this problem uh and I don't have now a solution but if we disable the automatic installation of the of the builds currently the testing form does we will break somebody so yeah we'll need to But but this is more about the design of the profile. So >> let's say at some point we have tested it against the profile and everything works. >> But as time goes on like those profile change. So like how can we be compatible with the previous versions that was working. >> Yeah. Yeah. Definitely good point like we can uh store the checkout of the profile in the artifacts. >> Okay. Thank you. So it's not done a file an issue so but don't forget but definitely a good point. This is we are searching for feedback so this is a really valuable feedback Christian. Thank you. So okay we are still >> since the microphone is here. >> Yeah. >> Uh so this was the first time I heard about go. So I was I'm just wondering how I mean why was it chosen or like is it because it uh the synergy with anible [snorts] like the yaml syntax or >> well um goss is really easy to like run because it's a go binary and you inject it we use it for every any like testing container images if they are prepared correctly by if the virtual machine is uh like the content of the server is correct and so on. So we have some experience with it and it's really easily approachable. It has quite good community maybe not the largest one but I think like this and it has good support for container images. It has a wrapper that goss is like a go variant and then it has a deos wrapper which is able to wrapper around podman or docker and it can easily execute a testing container. Uh so yeah we we chose it because it's uh like a lot easier to install some python or some ruby application directly on the machine. uh and it's built for all the architecture so we can use it across the board and it's really like this easy distribution. I think that was the main selling point. Also the YL format is quite nice and it's very fast like because it's go and uh it uses uh yeah uh it uses multiprocessing. So it's like really uh like gets the job done really really fast. So that that was the reason there is some good unsele integration and it's like really like it has a good community if you check out the GitHub. So yeah, a little bit selfish reasons, but I think also we save a lot of like effort not need to deal with Python installation, you know, into the containers and so on. It's a big stack and this is just a one binary. So >> thank you. >> You're welcome. >> Thank you. Thanks for joining. Enjoy the rest of the conference. [applause]