Aleix Pol Gonzalez: Rethinking KDE builds

So today I wanted to talk uh a little bit about some work I've been working on uh during my code thinking time. Coding is my employer. Um yeah let's start a little bit introducing myself for those of you who yet still don't know me. Uh my name is Alles. I'm from Barcelona, but I have been living in Verin Llin for a while. I am a KD contributor. I have been doing Kitty software for the best part of the last couple of decades. And as of late, I've been the KDV president as well. The KDV is the association around the Kitty community and we make sure that everything can happen. And uh yeah, another thing that I've been doing is um twistering with Harald. Um we welcome you to join us on Sundays at 6 p.m. And on my professional time, I work for good uh which is also an academy sponsor since last October and I've been doing all sorts of build things including this and some other stuff as well. But I wanted to do this presentation from a KD perspective because I have the privilege of wearing two hats and I try to do my my best of it. Now, um, as KD, we've been, uh, or have been at at least, uh, talking about this topic for for quite a while about how there are a bunch of things we're doing, mostly software products and plushies, I guess, as well, but mostly software. And it's very tempting so to say to have the this this software happen in a in a in some kind of vacuum. But uh well what I think we need to do where actually we have been going towards for the last uh few years or not so few is to not only do this first step of creating the solver. So like how I wanted to explain that today is that as SKD we want to be able to provide you the software but we also want to be able to care about the effect the software has on on your you and you being not the academy people also as well but like whoever is the the receiving end of our software want to be able to to care about it and uh have a conversation uh about the kind of thing that you have been finding ing and when there is a problem, you want to be able to to react to it. Um, and it's on the tooling that ties all of these together that I would like to focus this presentation on. So would be what we call bills. Builds could be well the actual build that the the users are are are using, but it's also the builds that developers do in their systems when they want to see that a feature is working. It's also the builds that we send our users to see if the fix was worked even though it's not maybe a final release or whatever. It is also the builds that we put together. Uh so to have nightly nightly releases, right? like there's a a a ton of things we need to do that uh well needs addressing needs to be in place and I think that we can uh still do a better job at at doing them particularly like there's some that we're doing several times sometimes it makes sense sometimes it doesn't um in this presentation I would like to focus on the case of KD Linux and KD flatp pack binaries u but I'm going to go a little deeper on that later. Um, going a little bit uh back on all right, you're talking Alles about this thing that we are working with and uh we are reasonably happy with. Why do you say it's a it's a problem? Well, uh we have been work arounding a lot of the problems of our own system. We've been doing KD for 30 years now and we have gotten quite good at um well making sure that we can do our tasks in some way or another. that does that doesn't mean that uh well we have all of the facilities that that uh we would need to to be in our first version right uh something that uh actually I tried to address even that on last academy the developer tooling on last academy I was trying to say we can have uh good developer tooling for flatp pack and it actually it was on that flatp pack uh specific uh use case and we can do that there there are a lot of um well possibilities that we can leverage but I also see that people are not like really building upon that and then it's also very specific to to flatp pack which is a subset of the things we do um ownership is also something that we have historically tried to not uh step beyond uh it's very easy when you're working on a smaller project smaller product uh to say well everything that happens in the outside it's someone else's problem I hope because otherwise I cannot do everything right which is understandable but as soon as we start taking on more more projects and and problems we eventually need to say all right uh I am going to be taking care care of let's say K algebra but also like how cute works when I when you're running K algebra otherwise u you cannot say ever no no no this is 's fault. My application is working beautifully, but the problem is someone else's. And um you go talk to someone else. In the end, when you have a a product, you you need to put something in in front of the user that that um is going to just work. And and the last point is actually something that I haven't really tried to explain and I have also been debating myself how how to explaining mostly, but it's a repurposing app streams. So for example, one one of the things we are doing for KD Linux and I understand completely why we did it like that and it was it makes complete sense but we are repurposing an upstream in that we are saying we're picking the work that Arinox people are doing which is good uh um Arux is a highly successful project uh and they're doing excellent work um and we're going to use it for something else that is fine and it clearly works like today you can go and grab an an ISO of KD Linux and run it on a system and it's going to work fine. But uh that doesn't change the fact that you're saying well that thing that you were doing for that other thing that actually we're trying to not care about like we're going to work on something else. We're going to use it in a different way. Um we've had that problem in the past uh before. Um well that effectively it it creates a weird respon conversation with with with your upstream, right? Like As soon as you go to to the astream and say no, we are having a problem with that thing that you're creating that is slightly different to the to the the the use you were intending for. It has a problem for me that maybe you have not um uh realized. You always have to put it for example when you are trying to describe a problem in the terms of the the the other project rather than your own. things work much better when uh your your upstream is somebody who has the uh vocation of saying I am creating this so that so that I am getting used by other projects and they are established in this kind of way. Um yeah and I guess that what you uh all should be thinking is come on Alles this is a huge problem it can't be done you're crazy right and that is probably true but well we are deciding on taking on like distribution of applications taking on the distribution of an operating system well effectively we're owning that operating system and we're going to be responsible able not only of the behavior of plasma, not only of the behavior of um queen or organ algebra, but also of like the Linux kernel that you get over there or um systemd right and it's effectively lots of things that we're not particularly uh experts on at least just yet. we will have to become a little bit more and we're also slowly like um bending uh our our premises as well by like coming up with different targets that where is that we want to go where is it that what is it exactly that that that we want to put forward KKD Linux itself for example would uh still be a research and development project right uh we still call it beta for example >> alpha >> alpha all right Well, it was one of these Greek letters, right? Um, and um, yeah, what we find then, especially in the case of the operating system is that our changes have a a super large large effect. Also, in the case of flatp pack, right? Uh, if you up cute in in the the runtime, all of the cute applications in flatub are going to suffer. If you forget a systemd app in KD Linux, all of the deployments of KD Linux are going to are going to suffer. We need to have uh tools and but also we need to have to to to nurture ourselves with people who who are uh well expert call it expert call it um knowledgeable about the different uh components. But it does bring new uh opportunities right as soon as uh well you are on a on a on a technology that is going to allow you to own everything then you don't owe anything to the others right then the limit is on yourself you can do for example faster iterations across the stack you now for example in ginux we are it's very easy for us to iterate over what we have on um on KDE software which is fine but we are if you look at the development that is happening over there uh well escaping at least the traditional things we were doing KD and moving lower in the stack or for example of how like the containers work over there or uh file systems etc right um as soon as we uh have ownership over these different components in the stack I think that it's going to start um well showing a lot more this this faster iteration. Uh something that we also have been talking about and actually we increas in increasingly need to talk about is tailored images. Um one easy um thought would be okay we could start maybe doing kinux images for um plasma big screen or plasma mobile. But actually, if you think about how ARM works in general, you need to have a a specific image for every single device and every single model of of of these ones, right? And you need to base yourself on on on on a technology that is is going to at least be able to cater to this kind of nuances and that is provided by communities that it is part of their core set of beliefs that uh they want to partake in this endeavor right you don't want to go to I don't know RC and say no now we want to do risk 5 and they go like we don't do that right uh it's going to be weird and effectively we want to have space to grow right uh you I think that uh it's important that the limitations stay within what you are and not what technologies you're you're based on which is that's kind of uh interesting now that's a weird quirky uh picture this is from when I wanted to talk a little bit about this or a version of the the work I was doing with build stream and free desktop SDK for l And uh one of my colleagues said, "How about you like you?" I had already been running an builds for KD Linux for um using Pstream and and free desktop SDK. And he said, "How about you don't put put it on the Fairform 5?" And I said, "Sure, let's try." I copied some of the infrastructure they already had for when they were doing it for um Nomos and I put it there. I forgot to to to build plasma mobile itself. I I was already building plasma desktop. So I put plasma desktop there and that's kind of what I got. It was more or less also on the same day that I was having the the presentation. So I I didn't uh end up moving to plasma mobile also because we were uh discussing and we eventually actually started like reworking how we were uh doing these builds. But um yeah, you can you can do things, you can do weird things, you can do wrong things and that's fine. And it's not that wrong. Actually, it worked pretty well in any case. It worked pretty well as long as you didn't want to call because yeah, but you had a very good system settings with multimonitor support. Um yeah so in this presentation I would like to talk about mostly uh yeah KDX and flatpack because that's kind of um yeah the kind of scope I have here in other presentations I've done about this kind of work I was talking as well about like how about we try it further on the CI or um how we have builds for Android or other operating systems but that is a little bit too esoteric to be practical for the time being. I think that this is the interesting one. Now, what are we talking about right uh free basic is uh an operating system. Okay. No, it's a set of files a set of YAML files which are um recipes buildstream recipes that define an operating system a Linux a version of a Linux distro that buildstream understands. Now what is wheelstream? will say well this is a tool that was designed for uh well being best of all trades um to be able to build um operating system images it has been used a lot in the different projects it has been developed by coding for u big part of it although it's uh right now an Apache project so upstream is is Apache Right. But um several colleagues have been working on it and it has this assumption that when you build your things they are in a container uh you get all of the dependencies from yeah the different components that you have inside of free desktop SDK and like it's going to be reproducible like that. I'm going to go a little bit deeper in the in the features uh soon. One of the actually if you seen see the free desktop SDK you probably the first time you saw this name was used in the context of flatpack runtimes. So actually when flatback was developed was developed in 2017 or so and the concept of runtimes was established and they wanted to have like uh different bases for the different operating systems to use. Uh in flatp pack again as I was um talking about last year we use normally flatp pack builder which is a different tool completely different it has a lot of shared concepts but not so many and the first version was uh implemented by Alexander Larson who is also the author of of flatp pack in in flatpack builder uh but the second version already was already um built by some of their colleagues in using build stream and we have been basing our own uh KD SDK on this f of SDK when it was built with flatback builder because it was flatback builder when it was built on buildstream on build stream and actually on our on our end nothing much did change because uh well we were simply using the binaries that it offered right and what we the only thing we cared about was that we were shipped a GCC uh G++ um set of libraries and then what we do in this case is to build cute build KD frameworks and a couple of other components that we need ourselves because we are so special and that's kind of it. Now what are the interesting things about FL build? One of the important things is that uh when you build something, you can build it once and it's supposedly gonna uh be useful for everyone. Uh it's deterministic in that when whenever one of your dependency changes, you will get um uh a rebuild, right? And since it's happening all in a in a container, nothing of your system should affect the build of of that um component. And therefore you can share it because it's the same if you build it on your computer and if you build it on someone else's. One of the annoyances is going to be then that when somebody touches a uh one of the dependencies, everybody needs to to rebuild it because it doesn't have the nuance of like uh does it do the header change? Does the um API change or anything like that? Everything needs to be rebuilt as soon as a dependency changes. Um by definition of right uh another uh very important thing is that you can compose you have your um set of components like for example when you look at the components we have in KD Linux or in the KD runtime which by the way at the moment they're not sharing any code they um they are a bunch of KD things that we buy and build or cute and then we we We we link we junction which is the name in the buildstream lingo into free desktop SDK and then we refer to the dependencies that that we might have including build dependencies like compilers and so on so forth to them right and one thing that I think it's very interesting and you might link to what I was talking about a bit earlier on is that you get to override them right like you can say um I want to use Um I am building discover and I need uh upstream. P desktop SDK does uh provide an upstream but they build it without cute because there is no cute uh in ftop SDK and that's perfectly fine. What what do we do instead of like saying now we need to fork and ftop SDK which we could what we do is uh we create a new recipe for uh upstream cute and and then we say we tell the the junction now override override the upstream that they have over there with this one and then everything in the system including I don't know flatback builder is going to be uh built against our own upstream rather than theirs which This also makes sense, right? Like you have the full coherence of all of the system is built against the the same things. And uh another super important thing that actually we historically I don't think we have been uh exploring as much but um as time goes by and particularly as the CRA becomes more of a thing we will have to uh be looking more into is the introspection of this definition like uh the the end result of all of these recipes is um a declarative um description description of all of the component in your system and we have the possibility of create tooling or actually like you could read it right but that's not uh what we like doing and and create tooling that is going to create those reports they they could be like you could ask things like tell me all of the sources that we're getting from this project and which versions they have which which components are we patching which not uh what are the licenses of um all of the projects in in in my in my uh project and how are they related in in them. Something that uh I've been dabbling with, we haven't really nailed down the ergonomy yet has been the generation of Sbombs and particularly CVI CV reports which uh I think it's it's kind of cool. So a little bit the idea. Well, first the sbomb is not that hard. Actually the sbomb would be quite similar to well saying the sources, the versions, the patches and the licenses. Um also sbombs there's like 23 standards in which you can specify them and actually some of them will include licenses some not. Uh some will include uh CV data some not. But actually as we were discussing the particularly the CV topic, we were saying well it like having a CV report about all of the things that might be um securely broken in your system. It's good but it's not very useful right like it's going to be good for you as a user in case you want to see like okay all of these CVs I hate 73 74 right but uh as maintainers I think that or we're discussing it would be interesting if you could get okay like uh with these merge requests all of these new CVEes are introduced or all of these CVS are are resolved it's some kind of information that is could be useful to have but it's it's not super clear how we're going to get there but I think It's going to be kind of cool. And actually, it would help a little bit mitigate that um fear we were having uh earlier or I was describing about like having to own a lot of things that are not really our own. And yeah, something that I think that we all are more excited about is to make sure that um the how the CI builds things and how the developers build things can be the same. Obviously like I am not going to go to anyone's system and say you should be building your your applications like this or like that. you do whatever you want but uh with this and particularly if we are using it on the CI or so you have the opportunity of saying all right why does this problem exist on the CI or on the end user's machine um you can reproduce it locally see that they are right because the users are not trying to trick you into having problems they actually have problems uh you get to address it uh the fact that you addressed it uh means that it also got addressed for them and so on. I think that this is one very important feature and I think that uh one good way of explaining it is by trying to show it. Uh when I was looking at this slide earlier the the fonts were a bit smaller. Um you should look at this line as if like can can I select things like this whole thing would be like one command. I'm going to try to describe what we're trying to do here. So let's say you are doing you have realized that when doing a screen casting on your system uh pipe wire does something odd and you need to address it right in the old world what we would do or actually what I used to do when I was working on that was clone pipeware from somewhere else assume see that it's working more or le well build it against my system which is actually by the not all that trivial. Uh it's not that trivial to make it exactly the same as uh on when it's coming from the DR row, which is what you would normally do. Uh test the the thing, replicate it fails, do the fix, see that it doesn't fail, create a merge request. Right? Uh what we're talking about here is that BSD workspace you can tell it which is the component. Uh and here we can see how the junctions work. Like here we would be assuming that we are on KD Linux repo of some sort and we would be saying from predest SDK bring the component pipe.bst and you put it in this test directory. Obviously, this could be banana pi / pipewire. Doesn't need to be. Um, but but there you would get automatically a git repository with your pipe wire somewhere that you can already like look at the code, see what it's doing, maybe already like learn how um how things work and maybe you will already be finding the the problem over there. But the code you will be seeing there automatically will already will be what you were seeing what your user user was seeing right and you can do your changes and then you can run the BST build uh tool that is again going to build the application exactly like it is built on the dro right so uh with the same arguments with the same environment like with all the same dependencies etc. So what you would get would be comparable again to what uh the user would be what's happening deck. >> All right. Um so what yeah you build it it ends up in uh somewhere and what you would do for example in KD Linux because we have an immutable user file system you would use CX which is what we're using right now we're recommending people to use when they use KD builder you would overlay it on top of your uh user file system and Bob's your uncle as they say and the uncle would be like reproducing your bug or all of the things we were saying as for I think it's pretty cool if you don't you will feel it's cool when you need to fix a bug in Pacquire I guess um and for what it's worth uh here I'm talking about pipewire but would be the same for cute will be the same for uh probably our components at the moment what we're doing for our components in the proof of concept of Kitty Linux is to do a weird mega mammoth build using KD builder But the plan would be to also have a kio.bsd file and more or less the same would would be uh doing at uh all right now um I I was talking about how we've been doing um work on desktop SDK as of late. So we uh released uh last week or so well last month anyway the fest of SDK 26.8 uh fest of SDK releases yearly in August and in August means normally it gets released the first week of September. It's a bit like October Fest, right? And um it has a lot of um the things I needed when I I was trying to get get K Linux built uh using that. So it's already pretty good for us. uh something that uh it was starting to become more and more apparent and like not only for us also for other projects was that uh Fedtop SDK well it's useful if it's like somewhat tight and compact uh there is a lot of effort in Ftop SDK to do a lot of CI and and testing and and so there is also as well for PSDK extra but the rules there are a little bit more relaxed and and still the idea idea is that it's a place where you can put all of your dependencies plus+ and they're going to they're going to work. Um we've been doing some work as well to improve the the linting um in in the different repositories making it all use reuse as some presentation was saying earlier. It's pretty cool. Also all of the uh buildstream files are um formatted um well with a with a program so it always looks the same which is also pretty cool as well. And yeah the testing also has has and will continue being one of the interesting topics both for um for predest of SDK but also for KD Linux. We had a talk uh two talks ago, right? Uh from Bushan and and Niko how testing was interesting there. Um I think that it's it's all over and yeah uh where are we? Um oh the ord was already ported to buildstream. I did a a little bit of a weird thing over there because I am fully aware that nobody likes your technology to change under your feed because somebody had a happy idea of using a different technology. So what I did with uh flatback uh flatback runtime was to create a script that took uh a recipe for uh for flatback builder and it converted it into buildstream uh recipes that way. For example, Justin, he has been um looking over the the runtime as well as several of the apps and he can continue using the same tools and all of the scripts that he had without well having to migrate everything into a different thing that has well different concepts and stuff. Uh eventually I think that we will end up uh converging into buildstream but also that would depend on what we do for other projects. Um since March or so I also started working into or at least more seriously into into getting KD Linux working with buildstream which I would say it's working quite well. It's actually even running the on the open QACI that um Bushan and friends put together and it's passing until it gets to discover trying to do an upgrade and it gets a little bit confused over there. But also K Linux is not super designed to have like two different upstreams kind of thing. We can look into um getting that addressed or uh well fixing it in um eventually and then doing the merge or not if if they want to. That's something that we need to collectively discuss if it's what we want. In terms of challenges, um I think that it's not like we have challenges that are unique to this problem, but it's nevertheless worth uh noting, we need to keep all of our software in the on the server versions. Um for example, one of the things uh we changed so uh you know how I said that VST workspace open uh pipe wire will open you a git repo. it will open a git repo as long as buildstream is aware that it's a git repo. If you use starbas and stuff you cannot do that and actually there are several projects that are really really really not meant to be built using uh git repos for that we uh I developed a new plug-in that uses um what's it called release manager.org or which is code name Anitia in many places which is something that we're already using in flatp pack and well it's going to help us do this uh upgrading so so you understand kind of where I'm coming from one of the tools that you will have in buildstream is like you will say buildstream source track and that bsd file could be that pipeware but could be any other and if the source knows how to upgrade itself It's going to say, well, I was saying I was 2.3.1, but now I'm 2.3.3 because it got released and then like you can in one run do all of these upgrades for all of uh your um components and it's going to be awesome. Something that I was also looking into and actually having some interesting progress is improving how we override the upstreams. One of the problems was as I was saying earlier, you can say this component replace it with this other one which works fine but sometimes this is not what you want. Sometimes what you want is to say uh take this recipe that is super long like for example the Linux kernel and just add this configure argument here but the rest leave it the same. And you also want that when upstream upgrades it just upgrades and you continue adding the configure um argument. Uh at the moment what you have to do is actually like create a patch on the repo and build does some magic. We're trying to integrate it into like still having some YAML files simply saying that that you want to add this uh argument and it's going to be awesome. And yeah, we're putting also a lot of effort into improving how the testing works to make sure that uh while the CI is the one suffering and not our users, which is uh kind of cool, right? So my next steps in this regard is to get Kinux adopted. That's one of my top priorities. Um second would be to get the K Linux work and uh orgate to share the components. I think that there's that's something that will also be interesting for for flatp pack as well. We do have a a problem with cute web engine for those of you who've been on on the channel and that would already solve it because for Kitty Linux we also need uh cute web engine. Um, something I also have been like dreading I don't really want to have to do or or at least we would like to do the least possible is having to maintain the KD components. That's why we have the mammoth KD builder um component as I was saying earlier. Uh I have been doing some um magic thing that will try to do the the right thing using um cake logs. I don't know if you've been looking at the CIS admin channel but I was talking about that some weeks back and I had some progress if you want uh we can talk about it later this week and yeah something that also we are looking not only me but some other people in the my team in coding is to make uh flat pack applications a good idea to be uh built using billstream. are some interesting opportunities. Uh we can discuss that over the week if anyone has any interest and trying to wrap this up a little bit. I think that it's a very interesting project uh problem to own. Uh it's daunting. We it's going to be a lot of work, but I think that our it's in our best users best interest to to do so. Um, I did put uh Bob session on Tuesday at 5:00 p.m. right after the K Linux one. So, if you want to join, I will be happy to talk to you about it. We can talk about um well all of the features. We can also talk about uh I I'm we can talk about the KD Linux work or the flatp pack work or any combination of that. And if there is still some time, we can take some questions. That's Nicole's decision. Yeah. Thank you. >> Okay, we got >> They're all there. >> Okay, >> you have to run. >> Yeah, >> run. You have to walk. Okay, run. Um, okay. Uh thank you for the talk. So I have not yet used Buildstream. I think it looks really good. Um I have used YTOR a lot at work and they do sound extremely similar, right? Like YAML recipes, build graph resolve, layered composition, overrides, reports, like every single point almost. Do you have a take on the pros and cons of the two? So I only used YTO in a project you're very familiar with and but only on one project. I I'm not as much an expert in that as some other people are. I know that many of the buildream developers are like very strong um haters of Buildstream um or opinionated in any case. Um it's definitely two tools that strives to address a fairly similar problem. I think that one of the interesting aspect of this one is that it was created by a community much closer to ours than how YTO has been developed and as such well a lot of the social aspect is more or less resolved. Not only like the fact that good thing is around but like we've been using it in flatp pack we've a lot of nom nomos for example is already being using it that's a big difference another difference I actually like by using it maybe it's just my experience is that buildstream is quite faster but um yeah >> it sounds like it might be nicer for the desktop developers workflow in terms of like tooling and possibly but then like you could probably port Kin to Yog and you can probably port your project to build stream right and it's all possible and doable. Uh you mentioned the goal of generating all the KDE software in KDE Linux using BuildStream. Uh which makes sense to use one tool for everything, but if we do that, we're somewhat getting away from the goal of having KDE Linux compile KDE software in the same way that developers do on their workstations. Have you given any thought to the idea of also using BuildStream to to be the new blessed method of compiling all KDU software on people's desktops? Well, that's kind of what I was hinting at like when I was talking about pipeware earlier. If we have um recipes for Kio, then you can say BST workspace open Kio and BST build Kio. And actually it would end up being the same, right? Uh actually you would have the advantages of not needing to have everything in USR. For example, one of the things we have in Kinux right now is that you need to have all of the SDK in your users systems. you wouldn't need that because it would come from buildtream. What as haggled is thinking here is like oh but then when you start developing you have to download a bunch of things which is true because um but I I think that the end goal would be that uh buildstream becomes part of the the workflow when when you're developing at least in the context of KT Linux and and and flatpack if we're thinking of other things obviously like for Windows or something like that that would be different but for example We don't even use KD builder for that. We use craft which is already a different tool. But yeah, I think that it should be part of the thing. And actually maybe that's kind of what we can talk about in the buff, right? Like is that good enough? If it's not good enough, can we fix it? I can probably fix some of these things at work. If it's not good enough, uh I mean we can throw it away and that's fine. >> Cool. So this one's from the chat. How easy is it to use build stream to build systemd sister X? Thinking about this as a way to deploy non-core apps to KD Linux where of course we don't have a package manager. So I am also not the most experienced with system and dx but my understanding is that what you end up doing is saying this directory is a prefix and you're going to overlay it into this other directory which would be usr. Now uh a tool I didn't show there was one called BSD artifact uh checkout and bas basically that means so when you build something in in buildstream it doesn't end up in like somewhere that you can see in your build system it's part of the infrastructure so to speak and you can say all right now let's let's uh BST artifact is the tool that you're going to use for all of the things you've built and you can operate with it. You can do things like list contents and it's going to list all of the files that are part of that artifact. You can do pull to get the artifact from the caches so on and so forth. One of the things you can do is check out and check out is going to put all all of this file which is like depending on on on the prefix and you say you put it in this directory and what you would end up doing is say and this directory is a cx now and you you you mount it into / usr and Bob's your uncle. Now I say it like that because um yeah some nomos developers have already been looking into this problem and they even have some kind of graphical tool that they put together for that. I don't really know. I haven't tried it but it shouldn't be too hard anyway. >> Okay, time's up and I hope you're going to have a pleasant journey to your hotels.