Flock 2025 Bridging Accessibility Barriers: Enhancing Screen Reader Functionality In Modern GNOME

Okay, good morning everyone. So let's uh learn something about accessibility and how difficult and rewarding as well it may be sometimes. So let me first introduce myself. My name is Lukashra and I am working at Redhead in in the tools and accessibility team as a software engineer. Uh most of the team is working on the tools part but uh I'm working on the accessibility part. Of course, one of the reasons is that I'm blind. So I meet these things uh quite often but yeah it may be an advantage but also a disadvantage because we need uh new views because as we use the computers we learn basically what's wrong and don't care about it or stop doing that. Uh so if anyone wants to help go forward but before we get to the presentation I'd like to thanks thank all the upstream maintainers because without them this situation in accessibility would be much worse. So thank you everyone for doing a great job. And let's go to the next slide and actually learn what we will be talking about. First we will learn how the visually impaired use the computer and why it's so different. Why we need uh keyboard access and uh features so advanced that uh the security guys are frightened of us and it's not everything we have next slide with the agenda as well and in the second part we will look uh on the technical side we will tell the story of how keyboard input monitoring actually works, how it worked and how it will work in the future. I suppose if anything doesn't happen so much and yeah, that's basically the plan. So, let's move to the next slide. But before we get to the interesting part, we probably should get some terminology out of the way. So, first let's describe how a screen reader works and what it actually is. A screen reader is a piece of software which tells a visual impaired person what's on the screen. And it also announces changes of the screen content and allows the user to interact with the screen content like reviewing the text on of the screen, telling you the formatting like the font, font size and so on of some controls and so and so on. And on Linux we have basically only one screen reader which we can use which is named Orca but uh of course we have a screen readers on other platforms as well. Apple have its voice over screen reader family on its devices. We have uh toolback on Android and on Windows the situation is much more complicated because well the platform was promoted to visually impaired. So we actually have uh or had a lot of skin needers there but uh the most known today are probably NVDA which is fortunately open source screen reader so that's nice to have there and then we have a commercial options as well but this is flock so if you want to know more about this let's talk about after the presentation and let's move to the next slide. Well, a screen reader needs to tell the user a lot of new information somehow and it most often uses something called the speech synthesizer which basically gets some text as an input and uh it produces some sound as an output. Well, on Linux, the screen reader isn't talking to the speech engines directly. It uses uh a user session demon called speech dispatcher to do this. So the screen reader developers doesn't have to interface with all the synthesis engines and can just say hey me hello to the user with this voice trait and so on and it basically happens magically for them. And uh the most common speech synthesis engine on Linux is called is speak or more precisely it's speak for which is uh derived from is from is peak which happened a few years ago but we actually have more options there. We have a festival. Unfortunately, the development has slow slowed a little bit in the recent times. Then we have RH voice which is quite nice if you give it the right data for a voice. So yeah, the English voices are quite quite nice, but we have only a few say check voices to select from and I would like to see more but yeah that's that's a good idea for a contribution I said and yeah the AI wave didn't is speech synthesis landscape as well. So we actually have a piper TTS which uses some narrow network magic to do the synthesis part and there of course very much depends on how good your training data are. So, the English voices are quite nice, but the Czech ones, uh, it's an personal opinion, but, uh, they sound quite weird to me, but, uh, the computations aren't so complicated and you don't need some huge, uh, graphic card farm to do to run this thing. So it's quite usable if you have the right voice. And yeah, that's everything I wanted to say about speech synthesis for now. And let's move to the next slide. And we actually will talk a little bit about how the visually impaired use a computer because I very often still hear the same questions again and again. So let's get them out of the way. Most visually impaired users use a normal computer with a normal keyboard. Well, the reason is quite simple because if you would have some special keyboard, you the price of the hardware would be wow at least crazy, maybe insane as well. But yeah, some of the visually impaired actually use some special specialized hardware devices, but uh they are limited in various ways. Most of them or well I didn't hear about anyone anything isn't open source. So there are software limitations, the prices are high and so on. But it's an option for some of the vision impaired. But the majority is using a normal computer. Of course, we have to learn typing the the keyboard without seeing it. We aren't using some marked up keyboard. We are using a normal one. So this is actually done quite early in the life of the person. Personally, I started learning the keyboard even before I had my own computer. So, I'd say that was a good decision because I wasn't lost after I got the computer because there was still much more to learn because we naturally can't use the mouse or well for some gaming scenarios we can but it's not very common and if some develop a game developer actually tries it. He very often gets messages like well I have no mouse do this do it differently please. So after we get a computer, we have to learn all the keyboard shortcuts and there is a lot of them even without using Emacs which some of the visual impact actually do but uh personally I'm sticking to them well much little bit more mainstream user interfaces so I can actually fix the issues for more users. And after we learn all the shortcuts, we actually need the applications to be keyboard accessible. And that's another very complicated topic which we actually had some presentations about before on foss or defcon. So if you want to to learn about this aspect of accessibility, you definitely have some materials to use. But uh that's not this session's topic. So let's move to the next slide and actually start talking about the main thing of this presentation. But before we get to the what happened thing, we actually need to get out of the why why actually the visually impaired users need. So weird requirements for system hardware access. Basically we the screen reader definitely needs to see all keyboard events and do something with them. And well, if I could uh change the world however you could, we would have uh some API for emulating mouse input as well for reasons of some inaccessible web pages mostly. And we would have also some API to make arbitrary screenshots of a window and a specified rectangle in it. And we could of course then pass this screenshot to to a large language model and get some nice description of the screen which would be so nice to have on violent for say virtual machine installations and so on it. But well this topics definitely will be covered in some future talks but uh no this isn't this one. This is only about keyboard input. So let's move to the next slide and actually go through the reasons the screen reader needs the key keyboard events at all. One of the reasons is quite simple in theory. The screen eater has its own keyboard shortcuts and it has a lot of them maybe in the tens or hundreds. And so if you don't want to clash with uh all the other system ones, you have to often basically invent a modifier key which is something like control shift or out but something different. Most often it's the insert key or if you don't have the insert key you can use the capsule key as well. So you need to invent this modifier key and then you have all the shortcuts for say telling the current time, the battery status, system load, the current window content, the next line of the window content, the font parameters on some position for copying the content to the clipboard and so on and so on. There are really many of them. Some of them also open the screen due the configuration. So you definitely want to be able to do that. And for this keyboard shortcuts, you actually want them to be consumed by the screen reader and you don't want to the compositor or the applications to see them at all. So you don't do something unexpected in the application. So you just eat the shortcuts, do something and that's that's it. But we have other kinds of shortcuts. So let's move to the next slide and look at them. The visually impaired user very often wants to hear what is he or she typing on a keyboard. So you need to see the keyboard events for this as well. And you can't actually restrict this to text controls. You want this service to be a available everywhere. So it's consistent and not surprising from the user. So you want to see the shortcut, see the event, but uh now you want to pass it along. So it actually does the usual thing which it does in the application or compositor. And so and that's at note it's not all we have a next side with the shortcuts thing. Well, this use case is probably a little bit orca specific, but when you are moving through some text controls by line, by word, by character, you want to hear where you actually moved. And at least Orca uses the keyboard events to help it determine whether the user actually moved the cursor and and how much. So if you don't get these events, you end up silent in the text controls. So that would be also quite bad. And naturally, you want to do the cursor movement as well. So you don't want to consume these events as well. You just pass them along after seeing them. And let's move to the next slide. Well, when the speech synthesis is speaking, it may be something quite long and you don't want to hear it completely. So it's handy if you can just press some key and the speech stops. This is uh very often done using the control keys. So you don't want to see them but uh you are also passing them along. And let's move to the next slide. So in summary, the screen reader basically wants to see every keyboard event, every key press and key release even for modifiers and so on. And then it can do basically two things. It can basically consume the event so nobody nobody else sees the event. it doesn't do anything in the applications or compositor or whatever or it can just observe it and pass it along. So it does the usual thing as well. So let's move to the next slide. Now we actually know what we need for a screen reader to to work. So we can actually learn how in the technical sense of things it actually worked and it will not be easy because well it was a story I'd say. So let's begin with how it actually work was working in the past on the next slide. Well, before we can explain it, we have a a system component we weren't talking about yet. It is named the ATSPI2 registry which is al another user session demon which does a lot of accessibility related things because you actually need to find out the root of the accessible tree. But well the tree is something which tells you which controls, windows, buttons and so on exist in the tree. But it's not this time important to do this functionality but in the beginning it actually was quite responsible for keyboard event reporting as well. So when a screen reader wanted to get keyboard events, it uh basically connected to this registry demon got u got the accessibility dbus bus address connected to this dbas bus. So the session one actually wasn't handling all the traffic. We will see why in a moment. And then it basically told the registry demon, hey, I want to watch all keyboard events. And yeah, the demon say, well, okay, I will just do it. And we have the other part of the story. We have the applications which also connected to this demon. And when they received a keyboard event, they told the event the demon, hey, the user pressed say E, for example. The demon went through the list of subscribers and sent them this keyboard event and the screen reader was responsible for a reply which basically told the application whether the shortcut should be consumed or not. So you had to go through the register demon and the reply got to the application after some delay of course and the application had to honor the decision as well so things would work. So you had a lot of traffic on this bus you were bound by some latency for each key press in applications. So naturally this situation wasn't the end of this of this story and on the next slide we will learn what GTK4 did it was released in 2021 and it one of the big changes in GDK4 was the complete rewrite of the accessibility subsystem. system. It was now handling the communication with the ATSPI registry even directly without any libraries. And one of the decisions in the GTK4 refactoring was to drop this sending of the keyboard notification events. So now we had GTK4 applications with uh broken keyboard inputs for screen readers and as a result we naturally needed a solution which is described on the next slide. In 2021 not many users or at least not many visually impaired users were using Veent. So we weren't taking valent into account in this time and the solution was quite simple. We created a new ATSPI API for keyboard image monitoring and this API could use various sources of information. It could it could use the old legacy API. But when we managed to get a hold of X server, the API requested the key events director for the X server and by some well I think by horrible magic with keyboard grabs on the X server layer. I saw the code and I actually didn't like it didn't it like much but that's what we had. It actually supported this uh consumption of keyboard events by the screen as well. So now everything was working as intended as well. But as the time moved on on the next slide, we find out that uh violent actually happened and yeah we suddenly didn't have any server and the Xv valent layer was no help for now because the valent compositor couldn't supply the information to the Xvalent player as well. So now we were basically where we were before in GTK4 which is described on the next slide. Now we had a GTK4 application which wasn't sending the legacy keyboard events which was the only source of keyboard events which remained on a violent session. So in a GTK4 application, you could do nothing screen heater specific actually. You couldn't stop the speech, you couldn't open a screen reader settings, you couldn't re review the window, and you couldn't even read a text box. So the situation was quite well, I'd say sad. And after some time something get to happen. So let's actually move to the next slide and talk about it for a while. Well, because veent was so much common than a few years ago, Federa 42 actually wanted to actually dropped the valent or session for Gnome and R 10 was planning to do and did the same as well. We had to do something about this because first no visually impaired user would want to use such a system because the number of GDK4 applications was growing and growing as well. So it would come for these behaviors very often and of course they wouldn't be happy and they probably would switch away from from Linux completely probably and also the European accessibility act was um there and the date when it actually started to be mandatory for new things was uh drawing closer as well. So finally we started to do something about this and what well let's look to the next slide. First it wasn't coding something because we actually had a lot of components which would which would need to change so everything would work. So thanks to Matias we we got together with uh ATSPI and Maintainers and we got talking and explained what will happen and what definitely shouldn't happen in our opinion and we got together a solution and and let's describe the solution on the next slide. Yeah. Yeah. I was a little bit lying because yeah the so the first draft of the solution was quite simple. It was decided to be a debus interface because ATSPI was already using debus interfaces for some functionalities and calling uh debus things wouldn't introduce a new dependency to ATSPI. So it would wasn't this violent protocol. It very well could be, but uh it was much more simple to keep it a debus interface. And in the first version, this thing was quite simple. You just connect it to the compositor accessibility interface and you send the compositor a list of uh keywords shortcuts which should be consumed by you and which shouldn't uh be processed by applications. Even this was already quite nice because now the decision whether to process the shortcut normally or not was handled by the same piece of code which actually got the hardware event. So you had no interprocess communication for this decision every time. So the speed definitely improved and the compositor could really easily send you the event for all keyboard events. So, it actually did. But as it always happens, the first draft wasn't the final thing. So, let's go through the changes. The first one of them is the on the next slide. Well, it was find out quite early in the this in the actual implementation that just sending the keyboard events regardless of the thing whether someone actually wants them or not isn't a good idea. So now as a screen reader client you had to say to the compositor, hey I want to watch all keyboard events in addition to the previous uh capabilities you already had. And the compositor said, "Okay, I will do that." But as you can see on the next slide, this uh open design wasn't the final thing to to exist because well you don't actually want AP API which basically gives you a key capability without any work. You just could run as the user and ask the compositor, give me everything the user types and you got everything without any question asked. So this definitely wasn't the best thing to have. We actually needed to limit the access to this interface to some privileged clients. Well, how this access control actually works? It's nothing uh sophisticated. There's no cryptography handshakes or so on. It's just a check for owned debus name which is enough because most often someone else couldn't climb the name because it already exists on the box. The list of these names needs to get to the compositor somehow. Well, for now it's a hardcore place. Of course, in theory, the compositor could do anything. It could ask the user for confirmation and maybe it will in the future if we introduce say a portal for that. But uh we wanted to do anything to move this design to the finish line. So no interacting things were implemented and of course this decision is composite dependent. So for example matter could uh behave quite differently in the access control checks than than the others. It didn't happen yet but uh it definitely could. So let's move to the next slide which well is quite happy because now we had a working prototype of keyboard monitoring which worked everywhere on Veent X or it actually didn't actually much care but it basically is mostly useful on But now we had to get the things upstream. It required a few changes. First of course we don't want to rewrite the screen reader so much. So I created a back end to ATSPI which uses this interface for reporting keyboard events. This actually must merge and uh got to the 2.55 ATS by release which happened a few months ago. But then actually we needed some screen reader changes at least a little. the old interfaces were using uh hardware keyboard codes. So that wasn't quite good because you couldn't even replicate the key map and get these codes on land. So we decided to XKV key sims instead which uh anyway every compositor uses uh some in the keyboard event processing. So that wasn't nothing new for the compositors. Well, this changes into Orca went quite well and were released in the Orca 48 release. And then we have already the client support and we need to get the support for this interface to the compositors. This was probably the most complicated thing. do a few comments and uh back and forth with the maintainer in M but we managed to get this API to mater 48 as well. So in Gnome 48 with recent ATSPI everything now basically works on veent how I want it to work. So that's quite an achievement I'd say. But of course, matter isn't the only compositor on the planet. We have uh for example KDE and their Quinn. But that's not a bad news. We actually will get this interface in the KTE 6.4 as well. And I hope the others will join as well because well they the users probably will want that. And anyway we they probably want to be nice to the blind community as well. And let's move to the next slide. Well this this development was one of the bigger things I actually did in my work. Usually I do some smaller fixes and so I learned a lot from this experience. Uh the most important thing is that talking with people is much more important than the actual coding because when you start talking with people and you get them together, you can come up with some ideas which everyone can agree on very quite easily. And then you have the programming, but you already know what to do, what's the right approach. So you just code it. Of course, it's not just code it because the reality will be always a little bit different, but that's a little bit to be expected as well. And yeah, let's go to the last slide. We actually have uh nothing there because we are at the end. So now it's the time for questions. Hi uh I'm the one who implemented this new interface on the KD side. So first of all I wanted to say thank you for pushing this very important topic forward and also doing it in a way that we and KDE could very easily implement. Related to that, I want to ask, do you have any ideas or inputs about how we can as different communities with the same goals and the same text base work closer together on future topics like this? >> Well, that's a great question, but I don't have actually any specific answers. Well, of course there's um there are some at least uh KDE accessibility or Gnome accessibility rooms. So if uh someone from the other developer groups would be there and watching what what actually happens that would definitely help the known accessibility already. Yeah, that's that's great. Uh yeah, but maybe that's definitely not everything, but uh we actually have this same issue even in Gnome. So I I'm afraid I don't have much better answers than that. Um um I'm not too familiar with debus but theoretically what stops some other process from claiming the uh debus name? Um >> well I was afraid someone will ask that. >> Yeah. Isn't it just a a race against the clock like which process? >> Of course. Of course it is. But uh at least for at least for the blind users you the screen reader is the is the very first thing or within the very first things which start but yeah on a system which uh doesn't have any screen reader you probably would uh be much more successful er than that. But uh making this uh foolproof would really require some portal proxy and some user verification stuff probably. >> Uh yeah, thank you. And that's another thing. Um I've seen Hyperland has some drafts for a protocol of this type like um for global shortcuts. I don't know if uh Katie here or Gnome has something similar, but I've definitely seen seen it in a hyperl. So, is there like some other option that uh could be used for global shortcut elsewhere? >> Yeah, we were actually looking on the global shortcut thing before we decided to invent something different. But uh the current implementation of the global shortcut protocol requires the user to to confirm all the shortcuts which would uh bring a quite huge dialogue for a screen reader which of course nobody would read and it would just say okay I want you to be away and it wouldn't actually be useful for for all the use cases because you would have the screen heater shortcuts maybe because I didn't see support for custom modifiers in the global shortcut specification either but you do you wouldn't be able to do the speech interruption thing or you would need to use some quite annoying shortcut. you wouldn't be able to report type characters and uh reporting of u moved around text fields wouldn't work either. So we would basically need to make the global shortcuts something like our keyboard monitoring interface anyway. >> Okay, thank you. Any other question? >> Yeah, thank you for your work. Um, basically all things regarding accessibility or like often start with bad news of what was broken and then it was fixed. Um, do you have like any coordination in place to avoid breaking things first before they get fixed? Maybe like the the group that you or like the people you managed to get together um yeah have built some some teams or is there something there where we can help to spread awareness about this? Well, we actually knew about what will happen, but uh the accessibility stuff wasn't uh popular hot new stuff in the old times around 2022 and so on and there were much uh higher priorities like rewriting the graphical rendering stuff GTK Okay. Or or things like that. So even we tried to get to the developers, we probably maybe didn't say it uh so bluntly. So they didn't understand how of a bad news this will be. But I hope that the situation is much better now because a lot of develop more developers knew about access accessibility now and uh the awareness is much more higher. So I'm hopeful that uh something like this will not happen in the future. It seems like no other one more. >> Um, do you think it's viable to um replace uh passwordbased login with something like security keys for uh disabled people or not disabled people but just really visually impaired people? Wouldn't Wouldn't it be easier for them to log in that way? Well, you actually can do this. You can, for example, use uh your fingerprint to authenticate. Of course, then you have uh other issues like uh the session keying security and so on. So this makes sense at least so far for me if you either are using some password anyway but or maybe we could probably do something with two do or something like that but uh well it would probably help uh to other users as well. So I think in the future something will be easy to use. I think we could get to this state of authentication even now but uh I don't think it's uh very user friendly right now. >> Okay. Thank you. One more question. Hi. Hi. Um, what what about um speech to text? How how do do visually impaired people make any use of that? >> Well, it's u quite well, we have quite huge uh steps forward in recent years. We actually have now quite good modules to do that thanks to the whisper family. So you actually can do transcription of some audio to text locally and pretty accurately. But uh if you would want to use it for s say typing text, you would need to integrate it into the input met input meta mechanism which actually was done already. But uh I still think that uh if the visually impaired user can use a keyboard, it's still quicker and the user can be much more conf confident about what he is doing and yeah still typing on a keyboard is much more private. Seems like no other question. So, thank you, Lucky. >> Yeah, thank you for going and happy rest of the day.