Flock 2026 Fedora Test Days

Hello, welcome everyone. It's nice to see here so many people. I am Viera Kolasta and I am here with my dear colleague Jiří Prajzner. We are both from Red Hat. Uh from the accessibility team. >> [clears throat] >> Focused on uh the accessibility of Fedora and Rail. We are not alone. We have many colleagues, some of them sitting here. Uh some of them are at home listening. And today uh we would like to present you uh updates about Fedora accessibility test days. Uh explain why we would like to do or why we do a separate accessibility test day. Uh what is the situation? Uh manual testing, some information about automated testing, and future plans. And let's start. Uh so, why do we need uh separate uh accessibility test days? Accessibility is a complex uh set of technical requirements uh that demands rigorous testing. And we have several reasons for that. First, of course, ethical reasons because uh we need and we want to have uh our software as much accessible as possible. Uh other very important reason is legal uh because uh we need to follow compliance uh with section 508 uh 508. What is federal electronic and information technology that covers access to federally funded programs and services? Uh WCAG web content accessibility guidelines and European accessibility act uh followed by European norm. Uh for example uh Lenovo they have product accessibility pages. And to be honest, I hope I will not break it when I will >> [laughter] >> when I will try to Oh oh I have you can see my accessibility setting is not good for for this. Because and we will wait a second. Uh you will see there uh product accessibility statement uh for products that is offered by Lenovo and there is mentioned Fedora. So you can see if Sorry, if someone offers uh product uh it's need to be followed by the accessibility, but definitely it's not cooperating for me. So definitely uh we will continue. Okay, slide Mhm. Perfect. Mhm. >> So, uh And other motivation for us is to have better feedback uh for users because we expect or I hope uh if we would have uh regularly accessibility test days, uh people can give us more concrete and direct feedback uh for that. Mhm. So, next slide. Um and [clears throat] moving to test plan design, uh we start uh with Fedora test days in 2020 uh four. Uh we have some base uh accessibility test plan, but definitely we need to expand them. Uh our goal is to have test plan that follow uh compliance in the future. And what does it mean? Uh here uh we have VPAT, what means voluntary product uh accessibility conformance report. And >> [clears throat] >> uh this is a huge document about 50 pages long, and you have there plans how to evaluate product to be accessible. Uh there are three sections, as I mentioned before, that follow section 508, WCAG, and uh European norm. And uh the of that is that we need to many criteria to to cover. So, uh, it's very time-consuming. And, uh, for uh, presentation, how long it takes. So, evaluate one, uh, uh, Vpad, uh, for Fedora, it takes around 14 hours. So, Yeah, it was today. So, it seems that the it's it full screen will work, I hope. No. Probably, it won't. Oh, sorry for that. Okay. Uh, Okay. So, uh, starting with, uh, manual tests, uh, we need to have some test, uh, manual. Uh, reasons, uh, are difficulties in automating some test cases. Uh, we would like to have integration testing, so it means, uh, many options. And we need special hardware, uh, like Braille display and a Braille printer. Uh, >> [clears throat] >> we, uh, have started updating our, uh, manual test cases. Uh, Uh, we have started with, uh, Braille terminal and we divided it to three sections. Uh, first is Braille terminal in console that verifies functionality of Braille displays. Yeah, yeah. Mhm. Yeah. Yeah. Thank you. Uh, so outside any graphical user environment, Braille terminal X window driver and that's verify functionality of Braille output in a graphical user environment using virtual Braille device. Uh, uh, this allows to test the Braille terminal plus Orca cooperation without physical Braille devices because these devices are pretty expensive and Braille in desktop verifies functionality of Braille output in graphical user environment uh, uh, without physical Braille display. Uh, other steps is split Orca test to more detailed tests. You can see here uh, many options. Uh, so definitely test test cases and test scenario uh, expanded. Uh, exercise the update of this test case took only a small update but still uh, some of them uh, was needed. And you can see new tests that should follow as well. According to this, I would like to say all this functional functionality will be or I can invite you on DefCon workshop because we will describe there all this accessibility features. And we would like to have tests for for all of them. And definitely this part or some of this test could be automated, of course. So now I would like to give what to easy who >> Okay, thanks. Yeah, so why to automate? As you heard uh evaluation of the ACR takes like 14 hours and it's not just one-man show. It's group of people working on that. And it's very exhausting because the norms are, you know, it's a it's a really dry, boring text that is it's a legal ease, basically. So that's one big reason for automation. Then another one is to create the conformance reports automatically. So you can see which criteria is passed or is not passed. What we are using for automation is something that the desktop team or the display team called now is as we have been using for I don't know 15, 10 years, something like that. It's basically DocTail. Does anyone know or heard about DocTail? Okay, it's cool. And uh, now QE core, which is like a layer on top of dogtail that does all the stuff in in the CI. It brings the logs, it brings in videos if the test fail, and so on. So, it's like orchestration around it. And, uh, okay. The next thing that I think nobody wants to hear is there are also some tests involving AI. So, there is a reason why why AI I think is good for this use case. It helps to describe pictures. Nobody does that, right? In the application. Like, alt text is is a curse word. And, you can do end-to-end tests, like speech-to-text speech-to-text, and back to verify if our guy is saying what he's supposed to say, for example. And, maybe more use cases. I don't know, like, what we'd like to see in the future for the accessibility. Anyone? Okay. >> Thing you could use for this? It's called OpenQA. >> Oh, the OpenQA. Yeah, right. >> Yeah, we're the guys behind OpenQA. >> like to I yeah, yeah. It was cool. Like, we've heard about it a lot. >> Yeah. >> Yeah, many many times, and I think that Fedora, especially Fedora, benefits from having two approaches how to test things. Yeah. So, that's great. Yeah, so we heard about OpenQA. Yeah. Right. Uh, you >> know Peter Schindler? >> Yes, he used to be on our team. >> Yeah. Yeah. >> Yeah. So, we can all get together. >> Yeah. Why not? >> Uh so, do you have some suggestions for like upstream testing? Like if for example, you are the upstream, let's say for the Fedora installer or something like that that has like possibly huge impact on uh mhm accessibility. >> Mhm. >> Are there some tools or something that could be easy to start with and to basically provide like prevent regressions from actually reaching Fedora. Like if you are doing changes upstream pull requests for the stuff that has a lot of like UIs UI things. >> Mhm. Anaconda is web-based now, right? >> So, yeah, we we still have the old GTK-based interface, but nowadays it's a web view running a web application either locally or remotely. So, yeah, what would you be your suggestions what to use for that? >> Dogtail and Kiwi TCMS. >> Mhm. >> Like if it's if it's GTK-based, yeah. That's the >> Like technically it's Firefox running the web UI stuff. So, when that's >> Uh but if it's if it's if it's if it's if it's if it's web UI, >> Yeah. >> then there are web frameworks that that can test accessibility, but we are not using that. We are focused more on desktop. >> Mhm. >> And the web stuff is in the norms. It's it's I would say it's primarily the norms are primarily focused on web stuff. >> Mhm. >> But we don't use that. We just use a subset that is that fits the desktop. >> Mhm. >> But there are frameworks for desktop for web automation and they focus on accessibility, too. And it's frankly, it's easier. >> Mhm. >> Because you have the area labels if they are there. >> Mhm. >> It's easy just to put them there because it's basically HTML attribute. >> Yeah. So, that that's easier. >> Okay. >> I would say. >> Thanks. >> You have more slides to cover or Q&A is good? >> Other questions there? >> Yeah. >> Okay. >> [laughter] >> I'm really happy that somebody mentioned the installer because uh yeah, it's kind of interesting to hear that it seems that we are not testing accessibility on the installer, right? At least not the >> Yeah. >> So, are you doing any manual testing of the installer? >> Yes. >> Okay. Like specifically accessibility. >> Yes. >> Okay, that's good. >> I think I think Pavel Vachek or Lukáš is here. >> This is what I think that we could do better as well. >> Okay. >> And it's definitely a problem that we know of. >> Yeah. >> So, any other questions? >> I've got just one little bit off-topic question like we in our team are developing or test or more testing the smart card integration. We talked probably with your team and also with Adam about open QA and about that. >> Mhm. >> But we ended up creating something custom in the end that is somehow talking uh it's not it's not I think it's not even running in the VM, but it's like talking emulating the stuff through the some kernel modules. Uh I don't have all the details how it works, but it's also something that might be like another uh way or approach of testing and it might it might also help with the testing of accessibility because from what I understood uh the way how you are testing, you are testing through the accessibility API which is not which is not the same thing that might be showing on the on the display or showing otherwise. So, uh you probably know what what I'm what I'm mentioning, but uh that might be also something that might be helpful for some other teams if you want to know how how else the stuff in GUI can be tested. Please talk to me or I will let you uh send you to other people who know that. >> great, actually. Thank you. >> Maybe what I can say for that, uh uh from the question from audience, we can see that we have many approach and options how to test accessibility. Uh and uh we don't have uh single approach that would cover everything. And that's amazing because we can use all of this option and together we are capable somehow cover as much as possible. Uh and I think that's nice, but definitely it's a lot of work because uh we need to have manual test, automation test, and of course covered by by different tools, and each has own limits. >> Yeah. And I think that like for the future, if anyone wants to use like some kind of automation on the desktop, accessibility is still crucial to that. So, that's another reason besides helping other people to make the system accessible. >> Ondra. >> Uh hey. Uh thanks for the talk. Are there any examples of the automated tests for accessibility which I I for example as Fedora user I can I can run after I install Fedora? >> Yeah. Uh there is a link to repo. And those tests are focused just on the norms, like they should There are some tests that are not all the tests for the norms, but they should cover each section of the accessibility norms, either US or European. >> Mhm. >> So that's something to start with. And of course, I think that uh throughout the years, we have open-sourced some of our tests that test various application on the desktop. So you can see the Dogtail automation in practice. >> Mhm. >> And use that as a reference. >> And in case uh for example, I'm maintaining some up application in Fedora, and I would like to try to have automated check for the accessibility. That is working What's the best like approach? >> I I'm not sure I'm following correctly. >> Okay. Uh so, let's say I maintain a desktop app in Fedora, and I would like to have accessibility check. So when I update the app, I can check that it's still working. >> Yeah. >> I think for now we don't uh have covered it. Something like that. >> Mhm. Yeah. No. But you want to test the accessibility itself? >> Uh yeah, I mean whether I can go somewhere and copy maybe code. >> Absolute absolutely. Like like uh >> Like what was the best approach? How to How to quickly get the coverage, the test coverage for my app for accessibility? >> Oh, which one it is? Which one is it? >> Just a radical question. >> [laughter] >> Like a random one. Like if there is if Actually, there was there was uh years ago there was an article on uh in the Fedora magazine about how to use Dogtail. And if you if you just search the web, I don't have I don't have any example ready. >> Mhm. >> But you can find it definitely. It's out there. >> Mhm. Okay. >> Yeah, if you if you use if you use any kind of LLM, it probably knows Doc Tail. >> Mhm. >> Which is surprising, but it does. >> Okay. Thanks. >> Yeah, but if I understood correctly, you ask for example, you write some code, put it to some automation and injects for example if all labels are correctly described and accessibility is followed or >> No, it's just >> Yes, okay. Yes. Mhm. Yeah. Mhm. Yeah, to be honest, this approach is more connected to end users uh because the compliance is uh created for users and how they see or use it, you see. Uh it's not exact word. Uh but definitely uh this one is good question and and I we can think about it. >> Yeah, that's I don't I don't have it in my slides, but but you can find documentation to Doc Tail or QED Core online. And there are examples how to use that. The one of the benefits is that we still use the behavior-driven testing, which is realized on Gherkin language. So, it's like user-friendly description of what it should do. And so, it drives the life cycle of the test. >> Yeah. And Sushma, we have here question. Uh thank you so much. Yeah, sorry. You had the mic microphone. Uh sorry. I I haven't seen it. >> Okay. Let me ask this question. Um I tried when you were talking about LLMs, so I tried uh uh to set up an agent that would use DocTail to investigate the accessibility tree of the application. And I realized or it complained about uh some tags not being correctly filled. But when I checked visually and I also tried to listen to the Orca output, it didn't matter actually because uh for example, the the button was hidden in a container that it named before. So for for the user, it was still understandable. If I run that tool again, do you want to report those failures or not? >> Yes. Because Orca is using something like it has its own settings. >> Okay. >> So that's a little bit more involved. >> I have a slightly different question. Do you mind if we go back to the manual test slide? I think it was one before. Yeah, exactly. So your talk is called Fedora Test Day. So if there is a new Fedora and I would like to maybe participate in testing the accessibility tools, like how can I like jump on the board? Like where do I submit, you know, where do I start a test case run or whatever? >> Yeah. Yeah. >> Where do I submit the results? >> Uh definitely uh we plan uh Fedora Test Day officially like you know, uh Fedora Test Day is focused on different uh uh area. And uh we uh when we prepare that, we will announce uh Fedora Test Days. Definitely it will be a little bit more time week and people I hope uh can participate. Definitely it work on wiki pages. So people can give us feedback about if something wouldn't fit or would a problem. >> And the most what you will find the the announcement you will find it on in >> Not the announcement for me. I feel like we suck at it like with the manual testing in our beta releases because like we are currently using Google Doc with like really you know like extensive sheet of tests and then it sucks. And I am looking if you have like a better way how to share something with users where they can see similar bugs and so on. Like I remember night rate, right? From >> [clears throat] >> But like I guess this is not what you show users, right? >> Yeah, I I don't know if you have anything better. >> Like like you mean the the report the reports? >> I don't think so. >> Google Doc has nice thing that you can see other people their hardware and you see the same issues that you are hitting and >> you don't report them again, you know? But like that's that's the only nice thing. >> I don't know I I also don't know how accessible is that about. Yeah. >> But thank you. >> We are out of time. >> There was a there was a question out >> I have a quick question. >> Yeah. >> How do you folks incentivize participation? Do you have a Fedora badge associated with these test days, [laughter] right? >> That is That is a little bit difficult. We are from Red Hat. >> [laughter] >> Fair enough. >> So and I don't know I don't know if anyone in in the Fedora community actually if there is any accessibility working group or anyone taking care of this. Definitely would be nice. It's something that if we can we would like to ask the Fedora guys to provide so there is as you say some incentive to to participate. Thank you. >> Are you out of >> I know we are at the end of our time. So if If would have more question, please catch up outside and thank you for all of you that you are interested in the accessibility. It's amazing and continue working on that. >> Thank you. >> Thank [applause] you. >> Thanks.