Flock 2026 The Packager's Guide To OpenQA Failures

[clears throat] >> All right. Um good morning everybody. Um Thank you for coming. My name is Lukáš and I work on Fedora QA. And uh today I am going to tell you something about uh what you can do to find uh what the problem is uh when an update is blocked by OpenQA. Um the talk was scheduled to be 50 minutes long, but maybe it won't be that long because uh I only knew it wouldn't fit into 25 minutes. So uh yes. Uh we have gating on OpenQA on several updates in the critical path. And uh since the gating started, we have had questions. What is this? I am seeing there is a failure on my package. And it takes me somewhere into something called OpenQA and I don't understand what happens because it looks fine for me. So uh I decided to uh show you what OpenQA is in a nutshell uh so that you will be able to go and maybe make an educated guess whether the problem is on your side or if it's on our side so that you can proceed however you wish. So you have probably uh came into the situation that uh uh you created a new build of your package and it's nice and polished and everything's fine. Your tests are passing and you wanted to uh send it to stable. But, suddenly Bodhi says that uh there is something wrong and your package is blocked and you have the uh tiny little red spot uh on the automated test tab. But, uh these are some weird tests that you have never seen. It's not the unit tests that normally come from the Fedora CI. It's a different set of tests and you don't know what they do and how they work. So, if this is your situation and you somehow feel frustrated about it, you don't have to anymore. So, uh I assume that Bodhi will be the main interface for us and that your journey as packages will start in Bodhi. So, uh you know that uh if you go to the automated tests tab, uh um you will see the overview of all the tests. There are lines. Uh they have different colors. Of course, you know all of that. And uh the tests are shown there. Uh when we take a look on in the Bodhi page, so you can see that uh there are automated tests and immediately you see the lights here and uh you know that there are three failed tests. And this is the Fedora CI. And these are tests from OpenQA. Some of them are green, which is good. Some of them are yellow, which I believe for you is still good. Because mostly uh the the yellow lines are that the test is a soft fail. And uh that's because we know something uh is broken already, but it doesn't affect the test. So, we say for example, just to make an example, uh wait 50 seconds for a widget to appear, but because we know that there is a bug, if it's still not there, make a soft fail and wait 500 seconds. So, uh the widget eventually appears, but it takes longer than usual, so the test gets a soft fail. For you, it doesn't mean anything. It's probably something uh that we want to know. Uh the only problem that you could experience is that you have the the red test failing. And uh you see the asterisk here, it means that it's a gated test. So, if uh the asterisk is not there, so probably uh these tests will not affect your package in any way. But if the asterisk is there, then it can uh it can block it for you. So, why the failures matter, uh I have already said. Uh which failures matter for you, only the red rows with asterisk matter. But if you want to see uh if you want to explore more, then uh every red row could be uh linked to the error for you. So, uh then there is the need to investigate what hap- what what has happened. So, uh maybe the problem is on our side. Maybe the problem is on your side. If you want to know, and you don't want to wait until we tell you, then it's probably good to uh to find some clues. And uh in this talk, I'm going to tell you where to look. So, what is OpenQA? Open QA is an automated system. Uh it's serves like a automated junior clicking uh in the system according to a preset scenario. Uh so in the whe- when I for- when I first tried to uh identify what OpenQA was with the first LLM models, so uh they insteadly instantly said, "Oh, yeah, we know that. It's Selenium. And uh uh I said, "Okay. Uh so, if you know that, uh can you try to make a test for finding for example, uh problems with the Fedora uh Gnome calculator?" "Of course, I can find it." And he wrote a long test uh that was full of unknown commands, probably coming from Selenium. But, it is similar. Right? So, um for us, the system, well, there are basically two parts, two main parts. There is a scheduler that uh takes uh uh Fedora messages and schedules the tests, plans the jobs, etc., etc. And then there are workers that run uh the individual jobs. Uh you can have as many workers as you want. And you can have different workers uh across different platforms or architectures. So, we currently have x86 workers and r64 workers. Uh the worker boots a virtual machine through QEMU. So, uh it uses >> [cough and clears throat] >> the Fedora installation image ISO, for example, to con- to construct the virtual machine. And uh most of the tests perform uh installation on it. So, uh they really do the Anaconda stuff, and they click uh what to install, and uh at the end of the first run, uh we have the first asset with a clear installed system of Fedora. Uh depends on what ISO was there, of course. And then, uh, we test something against this. Uh, so you know that, uh, the production of the ISO images is called, uh, a compose. And, uh, we have, uh, mostly one compose a day. So, uh, the whole set of tests runs daily on the compose. Uh, if there are two composes on the on one day, then it runs on two composes. So, it's, uh, hundreds of tests that, uh, run every day. Which, uh, helps us tremendously to, uh, to keep the regressions in Fedora, uh, let's say at the side. And, uh, maybe you have experienced that, uh, over time you can install Rawhide, and it actually installs, and you can use it. It's because, uh, most of the problems are caught early by these automated tests. Uh, then when the when [clears throat] the machine is booted and, uh, the automated junior is performing the tests in the scenario, so it interacts with the mouse, interacts, uh, with the keyboard, and it guesses what it sees on the screen. So, it's basically uh, it performs like a real person would perform, but, uh, it can't think. So, you give it a scenario, and it does it to the point. And then it reports results, and, uh, some of them come back to Bodhi. Uh, minimum open QA vocabulary job test module needle logs, of course. Everybody knows it. Uh needles are interesting and we come to them uh right away. I am skipping a couple of slides because I will I will talk about it later. So, how a test run works. Uh I have already uh told you uh something, so it schedules the job, uh boots the uh virtual machine and then the scripts interact. And each step is something uh it In each step, there is an action performed. Uh so, it can be either a keyboard interaction, a mouse interaction, a visual check, uh or something uh on the console. Uh and then we either check the screen or we check the output of the of the script, right? And uh when something doesn't match, it fails. Then we get the video recording, we get the screenshots, we get the logs, and we uh get a JSON with test variables, which for us, of course, are very very important because this is how you steer the tests and uh you choose paths in the test scripts. Uh So, there is a nice picture of a duck. Uh It walks like a duck and quacks like a duck, but it still might not be a duck here. And uh that's just an in uh you know, interludium or a for uh the next slide and these are the common failure types. The The common failure types in openQA are I have uh you know, split them into five groups and uh basically only the fifth group is what you are interested in. And uh the what's above is uh probably uh or uh is not caused by your package or your update. So, there are visual mismatches, of course. When openQA expects uh to see something, but it doesn't see it. Uh there are console failures when there is a command and uh we expect an output from that command and it gives you a different one or it gives you no output, for example. Uh There is the timing infrastructure problem when uh sometimes the system is slow. So, uh the needle waits for a preset amount of time. Let's say it expects the widget to appear in 60 seconds, but uh if uh the virtual machine uh comes into a unresponsive state, so it takes longer, maybe, and then uh it fails. Sometimes uh some of our tests uh download some information from the network and uh when the network is down, uh it can fail. Sometimes uh there is a problem uh on the system and maybe caused by another package that uh prevents uh the tests from uh passing successfully. And uh then, of course, there are problems that uh maybe the update has caused. So what you should do if you want and I think that's why we are here today. You could investigate yourself if you are interested or cannot wait to see what is the problem. If you don't want to do that you just uh have a have some drink and wait until Adam comes to you and tells you what to do. Pardon? Yeah, we will talk about the boss then. And now let me switch to a real case scenario. Uh Yes. So uh we can see that there is the there is some update here and update has failed. And well something has failed not the update maybe. And it's unpushed now but before it was blocked. So you are familiar with this part and you are probably familiar with this part. So here Adam explains what happens why the package has failed and uh what the problem might be and you also have the links to Bugzilla to the to the bug. So then you know that probably it's important. If it's not important then here we could say this is not important for you and you can wave you can wave it. But uh you want to see more Yes, if you wanted to figure out yourselves, so you can uh take a look and uh pick up a red line here. So, let's say for example I don't know which one it is now, but uh this one? So, uh when you click on that link, it will immediately take you to the OpenQA instance. Uh the openqa.fedoraproject.org is the production system. Yes? So, this is the where the real tests uh take place. And uh you can see the overview of the uh of the test suite. So, if you take a look here in this column you will see the names of the test modules. Each test suite consists of at least one test module. Uh mostly there are more than that because uh for example, this one makes the installation. So, it boots to Anaconda. It checks installation sources. It selects software. Then it uh uses the disk uh and uh takes it as a whole, so it installs Fedora into the empty or well, yeah, because normally this is always empty. Uh then it creates the root and the user. And then uh it uh continues in the actual installation, reboots, and uh then there are some uh follow-up tests that uh for example, uh check whether there are some selling looks problems or other outputs that we don't want to see about possible errors on the installed system. You see that the status of the result is passed or failed or skipped. Sometimes it could be also soft failed. We have already mentioned that. Then you see the screens here. And uh There are basically uh on the first look, you can see that there are three types of screens. Uh these tiny little pictures uh these are the virtual machine screens providing some graphical uh user interface. These black screens are the virtual machine screens providing uh the terminal. And then uh these are either serial console outputs or various commands produced by the tests. Uh when you click inside, but I am going to show you here. Uh you see that there is a failed screen here. Uh it's marked in red. When you click on it, you will get the result of the failure, which like 95% is wrong. Yeah. Yes. Okay. Uh so it's Sure. Yeah. It's not wrong. It's actually it it's actually correct. Technically correct. >> [laughter] >> The problem is uh that the test says for example I cannot find the button, but it doesn't tell you why it can't find the button. Yeah, it's a symptom. So, uh it says that there was an error encountered at Fedora test to install and reboot line 49. Yes. So, that's at least something. You can then take a look uh here. This is actually what was seen in the latest Yeah. Uh Okay. So, uh this is uh the latest screen before it failed. So, you see that an unknown error has occurred in Anaconda. Uh this is not you can't click on it. It's just a screen, but you see that it's probably a real error because you normally don't see uh messages like that in Anaconda. And then you could go to the left and come back in time. So, uh this would be the very first screen that we would be seeing when the test module >> [clears throat] >> started. So, uh you see that uh begin installation is a button there that starts the installation and it's marked in green. Uh this is we have a match. We expected the button to be there and computer or the open QA says, yes, it was there and I saw it. Uh you can then try to compare it with the reference needle on the left-hand side and what it really sees on the right-hand side. So, you can see uh if there are differences. This is especially useful on failed needles. Yes? Needle is uh yes, uh so needle is a PNG file with the screen paired with adjacent file that uh defines uh this area here. Basically, it tells you the X and Y coordinates uh that uh you start with and then the width and the height of the area. So, uh during the operation uh open QA reads the uh uh sees the um the screen of the system under test and then uh it tries to compare it uh with open CV uh with the reference pictures and if it find a reference, it's good. If it doesn't find it, it fails. On the right-hand side, you can see that there are 99%. This is the similarity, 99% which is high. Uh the default similarity that is set on new needles uh unless you want to change it is 96% and it's still doing it it still does fine. Uh the lower you get uh uh the more unreliable it gets and uh you never want to have anything below like uh 85. This would be a risk, I would say. So, we usually keep the similarity level uh uh above 90. The problem is with the similarity level that uh it's sometimes not visible uh with the human eye, but uh let's say font differences. If the font is one point bigger, it makes a huge difference on the area. So, if for example, uh an application chooses to uh use Roboto instead of Times New Roman, that would be a huge change, you know? And uh so, you can imagine for example then when Gnome decides to uh change the graphical user interface uh or uh they want to see some other color palette or uh a different font, then we have to They they have to So, we have to go and change all the references to the new Gnome, you know? So, uh I We are doing it manually. Okay. Yeah. So, um when you develop the test, uh the the needle has can have different type. This is the match type that compares the visual match. There are also OCR needles which do not work. And uh I approached the SUSE guys uh like 2 years ago on the Def Con and uh asked whether they were going to implement the OCR needles. And uh they said uh we don't use them. And I said, "But uh sometimes it could be useful because when the font changes and uh we have problems with that." And they said, "Uh well, we don't change the font and uh if it changes, we want to know." Which is also a valid uh valid approach, of course. Uh Font is one thing, of course. Shades are a different uh different question because uh it looks blue, right? But if the blue has a different shade, it could make a difference and that the needle still fails. If this is the problem that you can see and you would be able to see it right away comparing what was expected and what's there. And you see that the button has a different font like bold instead of thin, then you can go and wave the problem because it's not your problem. It's ours. Un- unless >> Um okay, I know. We have a mic on the phone over there. >> So, for Check, check. Okay. Uh so, for I'm not sure. I I'll do my best. For update tests, it's probably best not to wave it um because even if it is a test system problem, if you wave the failure, then the tests of every update after that one will also start failing. So, So, if you see that it's a needle mismatch or something, please just let us know and we will fix it as soon as possible. Yeah, but please don't wave update failures basically ever. >> Okay, so don't do it, but at least you will know that the problem is on our side. Uh right. And then uh if the problem uh is according to it Oh, I can't So, if it says that uh it's Fedora tests do install and reboot, line 49, so you could theoretically >> You can't see it cuz of the >> Uh-huh. You could click here and you'd be able to see the test script. And what was it? 49? So, this would be here. Uh die error encountered. If check screen Anaconda error report report. So, um which basically tells you if there is the problem die with this message. So, you can see uh what we were waiting for, what we wanted to check. Uh it's Pearl. >> Can you also then click the error because like it's really not helpful that you see the error encountered, okay, but you would like to click the down the drop-down screen with the more information. So, can you still do that like in the >> You can. >> Or it's over. >> No, uh >> Yeah, okay, thank you. >> It's uh it's a field system, you know. It works also in tanks and uh I imagine so. >> But we do do um So, we've detected that we encountered an error in Anaconda. So, we actually at at that case open QA will go and get out the logs that you would get if you click the button anyway. So, we do get the information out here. >> Yeah, I will get to it. So, this is the main overview and what you can read from here. And then there are various tabs that you can also visit. So, the comments tab is important because when we know what the problem is, we usually tend to put there a comment. So, if we have visited this already, you will know. So, you will know that the problem is being addressed. If it's empty, then it's probably not being addressed at the moment. Or maybe it's being addressed, but no results has been found. Then next and previous results is a very interesting tab because it will give you a list of other updates. And it will give you the results for them. So, you can see that uh an update before that and another before that and another before that were all fine. So, it's not a problem of Anaconda. And uh it could be located to your update, to this update specifically. Something must have been there because otherwise the others would fail, too. >> I don't know why it's not showing any later ones. Maybe if you change it to more entries per page, but actually it's best to check both sides. So, um if it passed before your update and it passed further updates after your update, but only your update failed, then it's 99.9% a problem in your update. If it failed for other updates after your update, that what that probably means is that something that's not on the critical path broke it between the previous update and your update, and your update was just the first victim. So, it's it's a important to see So, check both sides. If it's green all around it, then it's probably your fault. So, yeah, it's green all around it. So, this was definitely this update's fault. >> Mhm. Thank you very much. Uh then there is the investigation panel. Uh we are fighting with it in a in the moment because uh it's designed to give you various differences between uh between some, you know, packages to last good, SUT packages to last good. So, theoretically, we would be able to see what packages were installed on the host, and what packages are installed in the uh in the system under test. Uh so, we will know uh nothing changed in the virtual machines. We added this to the host. Maybe that might be a culprit on our side. If it's vice versa, if something changed in the SUT, maybe that's the problem. Uh we uh I am trying to set it up at the moment uh so that it works. I have a uh working uh uh let's say a working uh How do you call it? A scratch design or something? >> POC >> Yeah, that's it. >> POC uh and uh but we want to make it a little bit more polished, so uh it's it doesn't work, but you still can see uh this at least, which is the the difference uh between the test variables uh between this and the last good uh the problem is that the test variables are filled in by uh Fedora messaging and the body updates. So, uh it shows you that uh for example, advisories are different, uh builds are different, you know, it's just the way it is, so uh but you can see the last good here. So, uh and you can take a look. Uh this is probably not important, but uh sometimes it's good to uh to see whether the test is a follow-up to another test, and whether maybe the follow uh the the previous test has failed, or uh maybe its brothers are failing and so on. Uh the settings tab is for uh the test variables. Uh you can also obtain them as a JSON file, but here they are nicely uh graphically shown. And uh you for example see that uh uh the machine was a BIOS machine, so it was an on UEFI machine. And uh so that uh if you were having a test that was failing here, but not on the UEFI machine, maybe you could ask yourselves whether it counts or not. And uh uh the last, but not least, that uh actually uh Adam forbade to show. Yes. But uh you said you wanted it to be too You don't You didn't want it to be too difficult and complicated. But anyway, I think that uh you should be able to to know where the logs are. And uh they are here in uh under logs and assets. And you have two sections. One is result files and the other is it's probably a subsection called uploaded logs. So, these logs are the openQA logs produced by the testing system. And these are logs collected from the tested system. So, when you, for example, click uh here on the do install and reboot journal.txt, so you will get the journal uh output and you can take a look in there. >> And everyone can thank Kamil for there being a nice convenient text cuz I kept saying, "Ah, you can just download the var log tarball, extract it, and then journal control {dash} {dash} file." And Kamil said, "Like, just make a text file." And it's Yeah, it's way better. So, thanks for that idea. >> Yeah, well, actually you are right. Before they weren't there. >> [laughter] >> Okay. And then uh you have uh you have here uh the for example, the serial console here. The worker log uh which mostly doesn't reveal anything. Uh then there is a very complicated file, the auto inst log, which is every single step recorded by the uh by the testing uh system itself. So, you can really find everything what happened there. What was clicked, where was clicked, what was type typed, how it was typed. Uh sometimes we have problems that the virtual machines, especially in graphical user interfaces, are slow. So, when you for example want to type password, so uh it just types password. And then uh you don't you can't log in to the system and it seems like something is broken, but uh you can tell for example that uh uh that it was really attempted to to write password, but uh then uh then it just didn't get there. And there is the video. The video is so maybe if there are passwords you start counting the dots whether it's uh it's correct or not. And then there is the video. The video is a video from the uh tested system and you can Does it run? No. >> it does. It's going to be Yeah, it's it will be This test waits for a long time. >> Uh-huh. Yes. I don't want to mess with the resolution. Yeah, so you can see that uh video is here, what it does. And uh it's good to have a look before you do anything else because uh it can at least uh give you something. >> Do you want to show them the video shortcuts as well? That can be handy, like when you have a failed frame, you can jump to that spot in the video. >> And do I know how it works? >> Well, just go back to the test overview. Go to details again. >> Yes. >> Yeah. >> [clears throat] >> And then you just click any needle, like a visual needle. No, not It has to be a visual one. So, any of the greens. >> Ah, and there is the video here. >> No, no. If you go over to the right >> Over here? >> No, there, yes. So, jump to video. That will take you to the exact point in the video where that where that Yeah, it's it's useful. It's actually usually best to go back one or two needles before where you >> to be cuz >> it goes to exactly where that >> Yeah, I like watching movies, you know. Thank [laughter] you. Okay. >> [snorts] >> Uh So, uh you know, just to show you the uh the complexity of the auto inst log uh it's here. It has some colors, but not many. And uh you can see that basically start boot to Anaconda tests is where the test module starts and uh then it gives you everything uh every information, for example, send key key down. So, we are sending a key uh down arrow. And >> guess why we're go we're at the boot loader here? So, it's uh anyone guess why we're doing this? Why we click click down about 50 times here? >> In the boot loader? >> Yeah. We're trying making sure we get to where we want to be in the boot loader, and it used to be very complicated because on different architectures there's different numbers of entries, so we go make really sure we're at the bottom and then go up a certain number of times, and that turns out to work in every boot loader on every arch. So, >> Yeah, the problem um, uh, sometimes we It's fine. It's fine. It's good to know. Uh, sometimes we could, uh, solve this situation with needles. Like have a needle that, uh, expects, uh, some output on the on the terminal. The problem is that bootloaders are so fast that no needles can catch it really. So, uh, you have to go blindly, you know, no clicking and, uh, it works somehow. It works somehow. So, um, yes, uh, so we did all this. And uh, we have the last one. Uh, do automated tests dream of failed needles? Um, now, uh, of course, my attempt, uh, was to try, uh, how I could force, uh, artificial intelligence or large language models to do this for me. And, uh, I tried, uh, when ChatGPT first appeared, I tried it and, uh, it didn't bring much. Um, mostly because, uh, I wasn't thinking, uh, about how complex the problem really is. And, uh, I was thinking that maybe, uh, there could be a guess made based on the auto inst log file, for example, which is not the case. And, uh, we have attempted this, uh, before ChatGPT was here and it still wouldn't bring anything. Uh, and then, uh, when we have, uh, a we have made some progress in the field. So, I decided to ask the chat GPT like not do that for me, but how should I do it? And it gave me some tips what it what it suggested like you have to collect this, you have to collect that. Don't need a don't use agents. You don't want artificial intelligence messing around with lots of data because then you will get stories, you know. You will have to make a limited portion of the output and put it into a model and then it will be it will be closest of what you wanted. So, I said, "Why don't you go and suggest something?" And it suggested something and uh I used it to vibe code a set of scripts that basically download everything. And then they compare the they find where the errors are in the logs. And they go backwards for example, 100 lines or something and then they cut it. So, they limit the number of lines. And I add the screenshots and I make a nice package and I put it into into Claude Sonnet because that's what Red Hat pays for. And I am getting uh >> [clears throat] >> this. It's long. Uh but it gives you quickly their overview of what problems there might be. Uh and it gives you recommendations what you can do. And in this case number one is file a bug against xf progs or grub tool. Uh I think that uh in this case it probably isn't that far from what the problem was. Yeah. >> [clears throat] >> And [snorts] the problem is that uh it only sees the logs and yesterday I I was boasting about it uh to Adam and he said uh so let me check this one. Uh-huh. Yeah, well, this basically tells you what you have in this one line. Uh >> It's a very LLM thing where we had in the test system, you know, the failure on one line. This is what went wrong and the LLM gave you like six pages which were the same thing, yeah. >> Uh of course, you know, I I didn't I didn't know how complex this was for the man behind it to know all the details, you know, because if you have just the logs and you just tries to find some errors in the logs. I know uh you you just uh >> [snorts] >> you can't or you don't have to come to the very end of it. And uh so we are lucky to have Adam and uh he does it for us, so thank you very much. I try. I try, but I'm not as good as him. Uh questions? >> Uh thanks for all this work. I like everybody every Fedora user like feels that you are getting a lot of bugs and things are stable, so thank you. First of all, the second thing I have when the hell will we get raw height because like it's not getting raw height, right? It is? You get raw height? So, maybe we are still in bugs. I need to check. >> [laughter] >> So, we need to improve the test coverage then. If you are getting raw height, that's cool. I thought that it's not a gating. >> Well, uh you can if you are interested in some specific tests in it, so you could of course cooperate with us and tell us what we should test. >> We had a bug last time. This time we also found with AI analysis a little bit like closer like what's the problem. Uh, so yeah, we can like look at it because I thought that you said that it was failing also in open QA, so I thought it's not gated, so but maybe it was a test which is not gating. So, I think we need to improve the number of the gating tests then. That would be my ask. Uh yeah. Okay. Anyway, thank you. >> Welcome. >> Yeah, I'm wondering like how feasible is it for users or packagers to add their own custom tests because like one thing that I find like very valuable with open QA is like this uh needle and like the screenshot and like being able to see explicitly like how the test run, especially if you're running something like uh a console or like a app win like that changes how the customer is viewing it. >> So, uh how feasible? If you think that it helps you, then it's uh definitely good to have. And uh I think we still have some room in the in the uh We have resources and compute time, so if you wanted to cooperate on some tests for your packages, uh it's doable. And uh if you wanted it to do it under your table, for example, and so you could use the Fedora system, Fedora OpenQA. It's everything is packaged packaged in Fedora. It's easily to easy to install, you know. So you could run your own OpenQA machine and only run tests that you want to run. If you are interested in that, we can talk about it, too. Or you you just ping me afterwards and uh that's possible. >> In terms of how easy it is to write the test, not as easy as I'd like. You can do it. People have contributed tests. The easiest thing to do is probably to look at a pull request that adds a test and kind of copy what it's doing. Cuz there's various infrastructure to it, the templates and stuff. I did have a long time ago an idea to basically follow what you can do with Fedora CI, where you can just sort of drop a test file in your package directory and OpenQA would take care of all the infrastructure for you. It's possible to do that, but it was it's quite a bit of work. I'd still maybe like to do that as a real thing, but it's in the eternal backlog. But yeah, you can contribute tests. It's not as easy as ideally it would be. It's not impossible. I don't want to put people off trying. >> Uh one last thing, uh I would say that uh the problem of the complexity with the Fedora OpenQA also is that uh we tend for perfectionism. And uh for example, uh when we have needles, so I am a little bit, you know, I started as a philosopher in my life. So I am a little bit like, oh, I am having the test and it's really working, so it's done. Oh, great. And then Adam says, oh, well, but this needle is already there, so you should reuse it. You should do this. You shouldn't do that. Uh We used to We want the pearl look like this and not like your mess, so uh because I was messy, for example. So, now we have various scripts that tidy the the pearl, you know. And so, we get better through we, of course. And uh but >> I need to stop you. We are out of >> want to say, if you want to if you would have a open QA instance of your own, then you don't have to reuse needles and it would be easier. Yeah. Thank you very much. >> [applause]