Supporting WoA (Windows-on-Arm) Laptops in Debian

Good morning. Welcome to Debcon 25 day two. On stage with me is Christopher Oart and he will tell us about supporting Windows and ARM laptops in Debian. Give a warm applause to him. Thanks. Um, so I was hoping today that I'm going to collect a few of us together who are interested in Windows and ARM laptops. Um, and have a discussion about basically the state of where we are, uh, what we're going to do next and how we can support Windows and ARM laptops, specifically Qualcomm laptops and specifically the Lenovo T14S. Um, so who am I? My name is Chris. I've been working with Debian since 2012. Uh, been Debian developer since 2023. So I'm very new. Um this is my first dev comp as well. Uh so everyone's been very welcoming and thank you for for that. Uh I work for Lenaro. I basically build uh Debian images for customers internal projects around Qualcomm. Um so anyone who's running a Qualcomm process or that kind of thing. Um so what I wanted to talk about first was what the Qualcomm laptops actually are or what the Windows and ARM laptops are. Um, they're a lightweight laptop and they've got an Excel Elite SOC inside. It's got 8 to 12 ARM cores depending on what model you buy. Uh, they come in varying clock frequencies and they come pre-installed Windows as well out of the box. Now, you might be thinking why you want one of these laptops. So, I mean a regular laptop, they're quite heavy. Um, these are very light laptops. They're quite cheap. you know, you're talking about £1,000 for for one of these and they're very low power. Um, so they can run on battery for a long time, pretty much a few days really with high workloads. They've got acceleration inside them. Um, they got media accelerators, GPUs, MPUs, which allow it to be a lot lighter and low power. Basically uh the performance is quite comparable to laps the Apple laptops the M1's um in multi-thread workloads at least and I basically have one to build ARM 64 packages without cross compiling basically. So it's the main reason. So the laptops have got firmware on them that's customized EDK2 uh and they boot Windows with an EFI boot flow. Um, so my the people in my colleagues, they they bootstrap their own machines with Linux with some magic way of installing Linux on it. Um, and the Debian installer images, they're not usable out of the box. So, one of my goals is to get the Debian installer image on a USB stick, plug it into the laptop, and it just runs. Um, currently we're not there, and I'm hoping to fix that essentially. Uh so at the moment there's an Abuntu concept image. Um if you download the slides that I'll put somewhere you can click that and find it. Uh it's hack packages on top of Abuntu custom kernel custom bootloadader that kind of thing. Um and Abuntu are working on merging that into mainline Auntu but it takes a lot of time because there's only one person working on it. Uh Debian boots but you've got to manually select what device tree the laptop loads. Um, so it doesn't work out of the box. Surely we can do better than that. Surely and I want to go through a couple of the problems that we see and I don't have any answers for these problems. I'm hoping that anyone can help me with these uh that's interested we can have a discussion about it. Um so the first problem is Windows uses ACPI which is you know discoverable bus similar to what happens on x86 whereas Linux on ARM 64 uses device tree so a predetermined list of all of the hardware that's there. Um on these laptops we don't get a device tree from the firmware. We don't have any model number or anything like that. Um, so when you boot the installer or any image, the image has got no way of knowing what hardware is actually run on the device and nothing works essentially. Um, there's lots of ideas that we've had. We did a talk on FOSTM uh this year about the various ideas um but there's no proper solution upstream to choosing the device tree. Uh so the current workaround that I've got in my test images are hardcoded device tree and you've also got a menu in the bootloadader uh where you can choose what device tree you want to want to run. So we've got a grub menu basically. Um and like I say abundu works around this by reading the model number of the laptop from ACPI and then loading the device tree like that. It's not really a solution that we can do properly in upstream. So that's the first problem. Um it's quite a big problem. I think the biggest problem that we're going to see. So if anyone's got any solutions to that, I'll gladly hear at the end um what what we can do about it. Excuse me. Um so the second problem we've got is the kernel support for this device. So I think we're shipping 612 in next Debian. um we need a bleeding edge kernel for this device or these devices. So my colleague Johan maintains a kernel tree on git github and he's got about 50 patches on top and he's working on kind of upstreaming them as we go. But basically it it's not it's not ideal because you need this this latest kernel. [snorts] I think maybe in three five years everything will be merged and and we'll be there but by then we'll have the same problem with the next generation of laptops. So I think this is always a problem with with latest devices in Debian and other kind of OSS where you're relying on the new kernel but so to work around this we just build a custom kernel package which again isn't ideal. the experimental kernel works but there's a lot of missing features. Um so maybe like I say we'll be there in a few years. Uh another problem is the device has got firmware inside of it that can be updated on Windows. There's like a you know gooey tool you can use to update the firmware and all this rancid software that that doesn't really work properly. Um the firmware update package actually can update the the firmware on the device, but not all of the firmware well none of the firmware is in LVFS yet, the kind of online firmware distribution service. So that's a problem that we need to work with the vendors to try and solve. But I don't know whether this will work if the vendors aren't interested in supporting Linux. Um so another problem is there's a lot of device specific firmware that you need. Um the general firmware for the general device the Qualcomm processes in Linux firmware. Um the Lenovo T14 firmware is there. Uh but a lot of other devices aren't maybe the manufacturers aren't interested the vendors aren't interested in supporting these devices in Linux firmware. So, [snorts] again, I don't have an answer for that problem. Um, so the next thing we've got is cameras. If anyone's worked with cameras or Mippy cameras on on boards or on phones or whatever, you know, it's a a mess. Um, the camera now works. My colleague Brian's actually got the camera working. He's going to put some kernel patches up shortly. Um, there's no there's no hardware ISP support on these devices in Linux, so the images do look awful. Um, Brian's written a ISP, software ISP in the GPU that works around the issue, but it's really really intensive. Um, so it's it's still a big problem. One of my other colleagues uses a a webcam, USB webcam u just to work around this. Um so the last problem I'm going to go through is essentially the documentation. So there's a lot of documentation in various areas around the internet about these laptops mainly on git gitlab, github, all these other pages. Um there's a status matrix that we maintain about Linux support, but there's no OS support status matrix. So it's all kernel based really. Um there's no Debian wiki page that I can find about these laptops or any useful information. Um so maybe that's something that we can change. Other distros like Arch and Postmarket OS have got really really good wiki pages. Um maybe we can take a leaf out of their book. I don't know. Um, so what I've currently done is I've got some custom recipes to build a known root file system. Uh, uses hardcoded device tree. You can boot it from a USB stick, mount the disc, and copy a system image onto your laptop. So, I've got a few hours here and there to improve things, but it's not high on my priority list really. Um, I want to get the installer images working, but it's blocked on the device tree selection. So, if anyone's got any ideas of that, I'd really like to like to hear. Um, and the documentation we need to start a wiki page and that's something I want to do next. Um, and that's kind of it from what I wanted to talk about. I was hoping that anyone had any suggestions, comments, anyone's interested in helping support these laptops. I'd really like to help. I really like to hear. [snorts] >> Hello. Thanks very much for working on this. I've got one of these laptops, so I appreciate um the uh perspective improvement in my experience that might be coming along. Uh few comments. >> Yeah. >> Uh Ubuntu Plucky works out of the box. >> Okay. Okay. >> Uh which is what I ended up reverting to when I failed to get Debon to run on it. Yeah. >> Um, no KVM. >> Yeah, that's another that's another issue. Yeah, there's no KVM on these laptops. >> But apparently it's possible. >> Do you know what I'm talking about? >> I do, but not in any specific detail. >> So, it would be really nice if we could figure out a way to get that. Um, if I could add that to your list, please. Yeah. Yeah. If we could figure out a way to just make KVM work out of the boxes. I realize that is there are gymnastics involved. >> Yeah. um at EFI stage as far as I know before the kernel boots. >> I think we've got we've got an ether pad for this topic. So we'll add that there. >> Um >> yeah. Uh I know one of my colleagues is working on was working on GAA support um so that you could actually you could actually load um you could you could start VMs there with that but I don't know the status of that but definitely definitely I'll take a note uh and uh there's also like various missing hardware support like for example I haven't well I don't know if it's just on plucky >> Mhm. M >> uh I ended up booting uh I forget his name but I think it's the same. Yeah, I think I I built his tree. >> Um but then uh whichever way I haven't got external monitors to work, so it's I find it difficult to use as a daily driver apart from as a remote machine that's not really a laptop. It's just sitting somewhere for me to SSH into. Yeah, there was there was a lot of kind of kernel things that I left out here because it's not really related to the DRO side, but there's there's lots of things like, you know, if you unplug the USBC and replplug it, >> it it can crash and if you go into suspend and out of suspend, it crashes, but they're all kernel problems. I mean, I'm not really that interested. >> Yeah, when they're fixed, they'll flow through to DRA. >> Yeah, exactly. But like I say, that that process is quite long. Um so it might be few years till we get those fixes. >> I think the biggest issue to make it run out of the box at all is the DTB issue. >> Yeah, exactly. >> So Ubuntu fixed it in an interesting way. >> At least in the concepts images it was a workaround. I don't know what it's like in Plucky, but >> So I heard I've not seen it myself, but I heard that it's Grub that's been patched to do what you described. >> Yeah. So at least in the concept images there was a grub script um that that loads before the menu and kind of chooses what device tree to load based on the model. >> So I don't know if that's the right solution for for us really for Debian. Um I don't understand the the only alternative that I can see yeah >> to something like that is something ACPI based >> but I have no knowledge of the um details that >> I mean there's a lot of interesting solutions out there. So there's a project called DTB loader um that does similar thing. it it goes before the boot loader and loads based on the ACPI tables the the right device tree but again it's not really upstreamable >> is that packaged in Debian >> it's not >> maybe that's an answer then >> maybe but good luck getting that into the installer images >> into the what >> into the installer images yeah that that's not >> not really going to work >> because the installer images are meant to be generic and the minute you have hacks in it for various platforms It it it removes the generic side of it, I think. >> I If it's if it's generic that it can support multiple platforms. >> Yeah. >> Or uh say no nothing for me here, go do it the normal way. >> Yeah. >> Then that feels like it might be generic enough. >> Maybe. Maybe. Maybe. Okay, sorry if this is a bit of overtake of this talk. Uh, are are you interested in only Windows on ARM laptops or more ARM support in general? Basically, let me explain. I have MN Terraform which is also ARM >> and I can see that I'm dealing with similar problems to what you have. So for example, suspend doesn't work. Okay, hibernation works from time to time, but after uh waking up, it doesn't detect NVMe always reliably. Uh okay, it uses Uboot and not uh grabb it has the TB but at the same time Josh which is main Debian developer behind providing support, he provides his own images and event kernel. So for example, I'm now running on 654 or something. So this is >> taken from experimental and so on. So I can see that >> different ARM laptops have similar problems and I would like to avoid having different groups working on similar problems because I'm also I live in Munich Germany and I know that uh Tuxedo computers they were working on their own ARM laptop based of some Qualcomm chiefs but it looks like they also have similar problems with having reliable hardware support and so on. So I would like if you want so different people solving the same problems many times. >> Yeah. So so I I'm I'm interested in ARM laptops essentially the Windows and ARM part was just because a lot of the laptops that are commercial come with Windows and we want to remove it and put Debian on them. So yes the answer to question is I'm interested in ARM laptops. Maybe there's some sort of team we can create around ARM laptops. I don't know >> but let's talk afterwards if anyone's interested. I don't know if anything even exists like that already in Debian even. So that's another thing to to research. >> Um I I think people who are building custom lap ARM laptops have the advantage that they can at least build firmware that passes the right TTP through. Right. The the problem with the Qualcomm one seems to be that they're very much targeted at Windows. That's ACPI with a whole bunch of extra things in Windows. I I actually have one of the earlier Lenovo laptops. I have the the Qualcomm XCX Gen 3. >> Yeah, the X13S. >> It's beautiful hardware. Like I got 32 gigs of RAM and an MVME SSD. Um >> like yourself, I I would have brought it to DevCOM, but I can't plug it into a display. Suspend doesn't work. It's It's a very nice piece of ARM hardware, but it is really not suitable. And and while yes, you've glossed over the kernel bits, I think that the once you get Debian installed on it, Debian ARM just works. It's ARM 64. There's no problem with it whatsoever. >> Yeah. >> But there's enough rough edges on the kernel side. >> That's actually the main problem I see with it. Um >> so I mean I'm I'm working very closely with with Qualcomm engineers around this topic. And and what we need is is user focused list of problems, that kind of thing. So the more users we can get, we can we can tackle the problems in the right order like that. >> Yeah. Suspend and external displays are the two main things. >> There there's a lot of problems though with the laptops and the firmware that they run and we're relying on on their on Qualcomm and the vendors to update the firmware. That's that's one of the main issues that we have really with those kinds of topics. Um, but yeah. >> Is the the X Elite better? >> It is much better >> because I've kind of been put off upgrading because of the sort of I can't use this as a daily driver experience. >> So, so I don't use it as a daily driver. I've got the uh T14 regular here, the the the AMD 64 version. So, make of that what you will. Um >> uh you were mentioning the issue of device specific firmware. >> Mhm. >> Is this firmware present on the window on the original partitions and could it be fetched from there? Yes. In mobian for example we use droid juicer which is meant to do exactly that on Android phones. So we read the per device firmware which is stored on the device and >> basically just copy to the root FS debian root FS uh could it be done similarly on ARM laptops >> it can and there's already a package I've completely forgot to mention that so thanks for that there's a package in Debian that does this already um that's a small script that's used by the Auntu images to copy the the firmware over so Yeah, there is. I can't remember the name of the package. It's a bit silly though since I'm the maintainer. Um, but yeah, thank you. >> Hi. Uh, I just wanted to mention there is a a IRC channel on OFTC called R 64 hyphen laptops. I've left a link in uh uh etherpad. Yeah. >> For exactly this thing to discuss cross dro things. Yeah, I I was under the impression this IC channel is more just for kernel though really and and some light distro work. It's more for the kernel developer side of of the story really. >> Okay. >> So there was a cross dro a long time ago which I think the nario people were involved in. >> Yes. >> And there was that was more of a highle uh uh out of kernel user space stuff to talk about these kinds of issues. Yeah, I'm not sure of the status of that. I think that was a longer time time ago. >> Yeah, but it might be a place to try and resurrect it. >> Possibly. Possibly. Linux is supposed to support ACPI. Is anyone looking at why why these things can't be booted using the ACPI data and why they need these device trees? >> Not that I know of. Not that I know of. I think it's more of a thing in the server world basically, but I I'm not sure of the status of that. >> The the issue is, as I understand it, is none of the DRM GPU drivers do ACPI. Um, and and it's passing through that custom information. They're not identifiable. And Ben is looking at me and going to correct me on this. I know Matthew Garrett was looking at trying to unify both so that they could pull in ACPI data for the things that are common um and then use the device tree for the bits that are sort of nondiscoverable. But it's not as simple as sort of the the server world where you've got PCIe discoverability for all the GPU bits and everything and then ACPI for the standard bits. Ben, I'll defer to you. >> I know I'm sure you know more about this than me. Yeah, I was just going to say, well, well, as far as I was aware, uh, Windows is dependent on ACPI for discoverability. As far as I know, I don't think it uses device trees or anything. >> That's that's right. It >> it doesn't. And there's a bunch of extra custom drivers that are being pulled in. So, it's not actually a fully complete. >> So, you have what a Qualcomm driver that that that generates the equivalent of platform devices. >> I think there's a bunch of extra bits in there that know like this is the platform and then I have to do some special stuff. So there is knowledge built into the driver rather than knowledge built into >> ACpa is not a complete description of the machine as I understand it. >> Um >> uh and part of the problem at the moment with Linux is that once you enable device tree all of the ACPI parsing gets turned off. It's an eitheror in sort of the kernel itself. So you can't pick and choose about being able to go hey this thing's telling me this. What we really kind of want, as I understand it, is a device tree overlay on top of ACPI. Um, but that just isn't a thing that exists at the minute, right? So, it's a there's a whole bunch of core infrastructure in the kernel that needs to change to make that happen. Um, >> so to be clear, as far as I understand, and again, please correct me if I'm wrong, it's not just a question of uh the kernel getting the right ACPI support. It's that this the there is no way in ACPI currently to describe the hardware that is there. So for this particular hardware, it's never going to work as far as as far as I understand because the description is simply not complete. >> That is my >> So yeah, I mean I just wanted to make that clear because that relate relates to our plans here, right? that that's a long-term thing that needs fixing. But as far unless there's firmware updates to this laptop, it seems unlikely that would that would fix this. Even if we implemented all of this, it would never solve the problem for this particular hardware. >> I I I think there's enough uniqueness available in the ACPI tables that you could build sort of the equivalent of DMI uh conditionals that would pull that in. that will always need my understanding is the way it works at the moment is ACPI cannot represent all the information about the GPU that is required and therefore the GPU driver would need to have some way of going hey it's this particular hardware platform therefore I do this thing so yes that extra piece that we're currently putting in the device tree we would always have to maintain somewhere that could live in the kernel I don't think there is a reason why it couldn't semi cleanly live in the kernel in the same way as we have quirks for everything else >> I But it's a new it's a model that is significantly different than the ARM 64 server world where SPSA gives you enough to actually boot a generic DRO on anything that's SPSA. But once you bring the sort of GPU and suspend and all of that that sort of the laptop special world stuff in it gets very complicated. Um but I don't see that there is no way forward. But I think part of the problem is that um and not to pick on Lenovo, but they've built the laptop, have built firmware that work for Windows, and have only really got Linux as an afterthought. And I think if they put some more effort into it, there could be some meeting of minds between the colonel and the firmware authors, and we would get somewhere, but it's we're no one's target market there, right? We're not big enough in terms of Linux laptops that they care yet. >> I think that that's um very well put. Thank you. >> Yeah. Any other comments or anything? Um, no. Three, two, one. Thank you very much.