Using sbuild in 2025 (extended version)

Hello. So, we have Chris and Johan who will be talking about SB build, which is one of our favorite things in Debian arguably, but um Yoshua was supposed to be here, but he's not uh here. Um so, yeah, I'll give it to Johan and Chris for the talk. >> Thank you. So, [applause] thank you for showing up. Uh, we will talk a bit uh what's new in SBU for Trixie. So, we will give you a quick uh overview of what suilt actually is and then uh show you what's what essentially new things are in Trixie. then how we use asbuilt, how you can use asbuilt and also we want to help you use asbuilt um with us uh we mean yos yoken and me. Yosh is not here today with us. uh he has to work unfortunately but uh yeah so if you didn't know Yosh stepped in in 2015 to maintain sbuild um he added the so-called unshare back end in 2018 so it's not really new and yeah unfortunately he's not here with us today I'm Yen uh this is my first stepcom in my first talk at depcon but still I think that works um I joined the esport maintainers in 2022 after Yosh was pushing me to do that for some time and I was contributing patches already for some time and uh last year I pushed for get unshare adopted on the build demons and I will talk about that during the talk how that worked out. Yeah. And I'm Chris if you haven't seen me it's me. Um so as build is Deian's build manager. It um builds packages in a clean and minimal environment. It at the end of the day calls DP KG build package. Uh it also manages or can manage the change route around it. So uh it sets up a change road with only the essential build essential package and apps for the time being. Um it is used on the build. So this is also why we call it the build manager for Debian. There are others but the built network uses as build. And that's your slide. No, it's my slide. Okay. Sorry. Yeah. What's new uh in Trixie? So, the big thing uh which is maybe why you're here is sbuilt is now supposed to work out of the box or it does work out of the box uh using the so-called unshare back end which is the big thing. Um and the unshare back end is now used to build all packages on the builds since last year. You can uh try this setup on trixie or also from book one backwards like the buildies do and you can use these commands here. So you install sb build you also install packages from recommends and then you pass this just change route dash mode equals unshare and your favorite u arguments to sbuild. And while we will talk uh a lot today about anjair, you don't need to worry. Your existing srange root setup will continue to work in Trixie. Okay. So what do we want to do? Uh why do we want the back end? So first of all uh you maybe know the SH root back end we had for years used root effectively. So all your builds were running as root and this was a bit of security incidents and u apart from that s root was not really well maintained. So with unshare we don't need root anymore except this uid map binary which is really small and suid root but it's just does the u uh uh user ID mapping and it then closes again. So nothing there. Then we have a really nice thing that so in former times I was told uh using asbuilt was something for the wonder build team and no one else knew you how to use it or to set up correctly. So with the unshare back end now we have basically zero configuration. You still need to share say I want the unshare back end. We want to change the default in future but currently it's still the SC root back end uh as default. So people coming from bookwarm or earlier get the same behavior as before. But with the unshare back end, you can basically say I don't care. So I don't need to set up a change route. I don't need to set up an extra user, extra permissions, anything. As long as everything is recent, it should just work. So and the for the SH root specifically, we switched to M&Mstrap. If you run it manually, I think the vulnerability team still uses debootstrap. But this is a minor thing because mmstrap is actually reproducing what debootstrap does one by one. And with mstrap you can set a lot of more things in case you didn't use it till now. You can use multiple repositories, add extra packages, make more minimal change, whatever you like there. Um then the next thing with unshare so maybe more terms of if you don't know what unshare is unshare is basically the sys call in the Linux kernel to create user name spaces and usern name spaces can be uh the user ID mappings the mount name space the network name space and other things time name space um there was a nice talk by Helmwood in Hamburg the mini.com hamburg this year about names usern name spaces if you want to get more into that one but as I said one thing is the network name space. So we can actually fulfill what the Debian policy says that we want to run DPK build package without network access that we can just do easily here. We can just say kernel give us a new network name space with just the local network and no external network and then it runs just through and can't access anything. There's not even a name server configured and um I filled a lot of bugs I mean like 2030 because of packages which were using the network in one way and another and these are almost all fixed or at least all fixed in Trixie otherwise they are RC and patches are no longer in Trixie. So we can say for all packages in Trixie that they fulfill the the policy and you don't use the network. Um also with the thing like including mount points um or having a new mount name space means that sbuilt no longer leaves craft behind. So maybe you remember as build having stuff involved something something and um then you come out again in after a reboot and you said oh what's this and why my disc full or maybe people even uh tried to delete stuff and there was still a homemounted and I stories from people deleting their home. So this should be all gone because we have a new name space there for for mounts and it cleans up when as exits reliably and even more we even have our own P name space and that's led us to this GCC bug. Uh you can read other nice stories on planet debian or about that one because uh with this we can now reliably uh trace the uh the P ids of the first uh of the init process and all the first process of DPD. they all have the same P ID every time. And for GCC that meant that it tried to kill a P in a test which usually wasn't there anymore because there were so many process running but now it was there all the time dcc test was just breaking it was a fun thing to to debug for sure and thanks for all the help from the community to get that debugged but I think it's really nice to really really say we are reliably doing the same setup the same environment regardless of your system. There's still stuff which is different but I think for for Trixie it's really nice already and we can say it's validated by dep rebuild and uh the rebuild ID set up on reproducet if you have not heard about that one go to hog's talk he will talk about that a lot so we can actually reproduce what is distributed by debian or FTP masters and with a setup doing just the same thing. Um also we are doing archive rebuilds a lot. Uh so thanks thanks to Santiago Villa uh who helped a lot with that one and uh found a lot of bugs in the packages and also an S build to get that working and as we've said already we have it running on the build demons nowadays. So we're pretty sure that's working reliable and we are really trying to to keep this standard and not breaking stuff by trying to rebuild the archive before pushing major changes into sbuild. Um how to use the s unshare back end we said already. So the easiest way to do it is just say sbu call su c root mode unshare. Um, but probably you you have tried su or used sbuild before and you have something lying around an old config and so our recommendation is actually just get rid of the config. Maybe make a backup copy in case you want to have a look at it what you did before or you want to transform it to a new setup. We can help you with that later. But so remove your old setup uh install the newest versions as we said either from bookborn backports or directly from Trixie have it already if you have it. Um you can set also the unshare as a default then you have don't have to provide it to build as a command line. And then there's another small thing. If you have an old setup with uh add user or user ad from ages ago then you probably don't have this subu ID set up. Subu IDs are a thing to use user different user ids in a user ID name space and you have to uh the super user has to prize the user specific user IDs so they can use extra ids and if you don't have a setup you can use a uh you can check the etc suburu ID and subg ID and otherwise use this user mod command and uh just in case you can also adopt the the ranges there if you need more for more users this is the default for the first user. Um, we will show that live as well in a second. Um, wrong way. So, how did we get there? Maybe that one first. >> Yeah. So, um, we give you some history. Uh in 2015, Josh uh Yosh joined the aspect maintenance team or actually took over as the aspect maintainer. And uh then Helmwood Corner wrote some Python scripts uh that are basically the start of Anjair and what Helmwood now calls an Root. Uh and Yosh transformed these things into Pearl. And I mean if you didn't know, Sil is Pearl. So uh that's a start there. And this was initially called user and then uh a few years later this actually got embedded into sbuild and is the start of the unshare back end in sbuild and the this was still like not directly in suilt but like as a separate process. Um and then basic and at this point mm devstrip uh spring sprang into existence and this is now the basically this your uh table strip equivalent. If you haven't used that maybe you can uh maybe you want to look at it uh in separate uh separately and you will see that it is also a lot faster than delest strip and then y showed up right so u actually I was using sbuild for some time already with the hc root back end and providing patches and thanks to wuki actually for uh g getting me introduced the sbu and how to set it up because back then it was still like I I tried it a number of times I was always like okay I didn't get it And then I asked him and uh he helped me. Thanks for that. And uh at some point yes this new unshaded back end showed up. I tried it with some of my packages. it worked first and I think at some point it does some package didn't work and so I started providing packages uh providing patches in April 2022 according to git and did that for some years also showed up in numbers minor uh minieps and talked to other people into trying the unshare back end and found no new bugs provided more patches and then um in April 2024 uh I've become how uh I found out in IRC that the vulnerabil team actually switched some build demons to unshare and this was interesting because Yosh and me were discussing how to get vulnerability team to switch to unshare for years basically and we were thinking about using Yosh's old computer in his university to try to rebuild the archive with it to see where we are actually and then we just found out that they started with it already and we were like okay we never tried all packages So we have no idea how good the unshare back end is and they just trying it. So I contacted Santiago and he provided help that was really great and since then he does regular rebuilds with the unfair back end and that I think initially we found like 100 bucks or 100 broken packages and we got it down to 50 and then basically started filling bugs with the packages and thanks for all those who provided patches or uploaded new versions there and then we were like okay we are pretty sure the unshare back end works but then I mean it makes sense to to switch all uh build demons to unshare because you know at that point last year we had some build demons running SC root and one some running unshare and while going through this archive rebuilds I actually found a number of packages which just uh produced different results depending on the the su back end so that was what really nice and so I said okay let's push everything and so um then was it uh in October yeah in October we got everything in and I skipped one point which was August where Yosh did a really nice refactor making the unshare back end really pretty and he was saying that he had this in his mind for some time already and that uh finally pushed him over that uh we caught the vulnerable team in and then we had it. So yes, uh just to say so build demons running unshare is currently for Trixie obviously for um for all stuff newer and I think also for backpods but I'm not sure about that one anymore but it's somewhere documented in the uh announcement mail in the miscellaneous uh news developer news if you want to look it up. Um so this is the current state. Um we will show uh showcase you that later. But maybe uh just give you some outlook as well what I'll be planning to do next after the Trixie release hopefully soon. Um so the first thing here is where are we going is uh that we will clean up the UP cache. I've filled bugs for that one already. So for background uh so currently uh su starts a new change route and it has up installed. installs all build dependencies and at the end starts the DPK build package to take over and at that point the up cache with all packages which are variable at that point are still on in the change route and we have some packages which actually use this information the problem is I mean ex first of all it's extra information which is not somewhere encoded that it's needed in the build dependencies or everywhere or anywhere and second um I'm also a big fan of reproducible builds And this state of the build up cache at that point the package was built is not really reproducible or not easy easily reproducible. So uh Santago actually uh thankfully did an archive rebuild without the app cache. We found those packages. I've built bugs. they are in this buck user tag and uh I talked to the vulnerable team as well there and we will uh provide uh I provide patches to stop uh including the up cache post past Trixie so for that will be just gone and uh if you are in need for some uh some more information there or how how to fix your package feel free to approach me to get that fixed um the next big thing uh which is still in discussion there's a RT ticket for that one is getting unshare support on the porter boxes. So you know if you uh use a Porter box that you can't use Sbu with really there and even more that you can't use unshare currently it's not uh enabled because of security reasons but we think it's now secure enough and that would allow us to do the same setup on the portter box as we have on the build demons which would be really nice to reproduce bugs especially for uh architectures you don't have at home. Um then there is sites CIO you hopefully use and uh Santiago the other Santiago sorry [laughter] uh there's a really nice merge request open there to switch uh saci to sbuild as well and use the unshare back end there that seems to work there needs some more work but basically it's there uh and we'll switch soon hopefully um so that's all stuff as well directly U the next one is that we want to extend obviously as build as well. So currently we have auto package auto package test word unshare which I wrote some years ago uh to use auto package tests with the unshare back end and there's already already nice work in the auto package test merge request uh to to generalize that a bit to use other backends and so we we will probably in Debian because then you can just set up your different auto package test backends like QMO or Potman or whatever use and as build will automatically use the right thing. Um, sorry it's a long list but long list of ideas just to give you things what what you want to do. Um, the next thing is that we want to drop up from the default install inside the change route. That sounds a bit strange maybe but the thing is now with the back end and probably with other back ends we develop in future we can actually take the up from the outside to manipulate the change route setup. So you don't need up anymore in there. And the fun fact is that currently up is installed in all change routes but it's not essential or built essential and it's mostly most of the time not a build dependency of the package. So we had quite a number of packages where we tested that already with the archive rebuild and we found quite a number of packages which use commands from up or some other parts of without depending on it. And so we say in future we really want to get rid of up in there which makes the change rule smaller and more reproducible and also our build dependencies cleaner and packages which depend on up should add it to the build dependencies obviously. Um then there's one uh yeah just to give you more ideas uh this one is basically wrap up uh repare and qo helmet will talk about that. I think he has the working demo already. Maybe you want to see the next talk. Um so there's actually two things. One is to get um the build in a minimal clean environment which we do with the unshare stuff nowadays and the other one is to get an extra layer of security. So you know like um containers like uh we do with unshare username spaces can't be as secure as a virtual machine. So we want to actually have two layers to secure our build demons and one would be the outer one would be a QMO one and the inner one would be an the unshare one and Hwood is working on that one and also on something he calls unshroot which will be an or as an unshare an SC root implementation using unshare. So you can use actually the old API again if you want to and that will be in the next slot. More about that one just stay here. Uh, and then there's the last point here. Uh, Yosh actually committed to reduce the configuration and boiler plate and add more manual pages and we hope we get that in still into the Trixie release. I don't want to push Josh but would be great to have better documentation for ASL now that it's working so nicely. Um, so more for me second. So after having talked about a lot about what SB build can do and will do in the future, we now want to give you some more a small demo of how it actually works. Uh yeah, so let's start with this one. So let's say we just installed sbu on a this is now trixy but uh also book one combat ports and you don't need to set up anything node user configuration as we said you don't need to download the package even you can just say oh I want to recompile the hello package which is in debian and you just say oh maybe I just take the one from from unstable and just give it a go just to and what it does now it creates That's the change on demand using mm and debstrap. So installing the packages now installing additional build dependencies. No worries. I have an up cache behind that which is offline. So it it's really fast. And now it runs CPK build package and runs a build. This is just the auto conf uh build dep package and done. So now we have the package here, the debug package as well. This is nice and thanks for uh the hello package maintainer to produce a nice package that it just works. [laughter] But we actually said on the slides that we want to to debug as well. So we add this one. So um we will tell you later how to um automatically jump into a debug console. But say you want to debug the hello package or improve it and you want to we have a shell directly before the package builds. This is just starting build commands percents and um this one gives a a shell there. So let's start that one. Just does the same again. Okay. So, here we are. We are in some path. Um, oh, I didn't change that one. Okay. For me, it's package. Normally, it's a it's package name, but ignore that one. So it is already unpacked. All the built dependencies are there. So I can just say build build package minusb and it runs the same thing or I can hack around with that and do whatever I want to try. And just to say again this is shell is inside the ch root so you can't break anything and afterwards it's gone again. Have remember that one because if you change something in there and you don't save it it's gone as well. So save it outside. That's the only downside of this change thing here. But I think it's true again for uh for the old back end as well. Okay, let's exit that one. We will exit like this. Yeah, we come back to that later. Okay. Uh yeah so as uh Johan said this is uh the shell that you just saw is inside unshare inside the change route and this even the package source is not bind mounted outside. So whatever you do there that it will go away. Uh but it also means it needs space. So these files need to live somewhere and by default um as built in the with the unshare back end writes or uses your temper. So default this is /tmp and on trixie in case you didn't know uh systemd will mount /tmp as a tempest. So basically on Twix the default is then you have a size limit of 50% of your working memory which you can increase if you use uh swap or if you don't want to use swap or maybe you don't want to use memory then um you can either set uh the tempier environment variable before starting as build or if you want to use this uh configure this more permanently you can also set the dollar templier template And the suggestion that we give you is to set this to /art tmp. There's a question up there. Um or otherwise you can use any other world right table. Can you repeat the question? >> Okay, sorry. Uh, this is more of a plea than a question. Um, uh, please use, please make VM the default because this as it is, um, this is a trap waiting for people waiting for people to fall into it with larger packages that won't fit into RAM. So the nice thing with /temp on tempfs is that it's really fast. So you maybe know configurations where they used eat my data before to get a faster setup and so on. And with tempfs you're faster than eat my data. And so I think it's nice for most people having enough RAM. But I I agree that's a trap for the others. Yeah. Maybe open a buck and we can just discuss that in the box. >> Yes. So I think yeah so that's a trade-off. Uh so in general if you have wish wish wishes for sbuild please file box. Um yeah so just to complete this so you can use uh writemp or any other world right table directory. Um a common problem that you might run into is that all path components leading to this directory need to be world executable. If they are not then you will get weird error messages at least from auto package test or maybe lynching afterwards. I think building works but you will get weird errors. So um if you see permission denied or something make sure all the leading path components all of them are world executable. >> I think this is al also something Hmood wants to fix with this new back end. Yeah. Ah. >> Yes. And um another common issue that you might see um is when you're actually in or su then you might run into this weird error. Uh let's say something like unmet build dependencies yada yada filter clean source. And um if you have current build it will also give you a hint how to deal with this. But the thing to understand there is sbuild is not in charge of building the input to sbuild. So the input to sbu is a source package and the source package in debian is a DC and the tarball maybe other tarballs. Um what su does as a convenience feature. If if you start as built in an unpacked source directory, it will call dep source for you to build a DC but the package source um requires that the source tree is clean and it wants it really wants to make sure it is clean and for that it runs the de rules clean target and by policy that means you have to have built dependence installed. Now there are a few ways out of that. So one if you run uh this from git build package then get build package will actually make sure your source tree is clean or refuse to start. Um so if you run git build package d- builder thank you uh is built then you can just pass d-n no clean source. Um otherwise you can just accept that your source tree is unclean and you can also pass the traditional clean source. And the um galaxy brain option is to move all build dependencies that you don't actually need to run the clean target into build dependence inep or build dependence arch. Um and then you probably have the help the always installed and then you can also run uh your clean target without needing anything. um or you create the DSC yourself. So if you use Dit asbuilt then Dit actually makes the DSC and then you also don't need uh build dependencies installed. Uh you can read up on the discussion in the bug linked here. It's marked won't fix. Uh I think if you have really good arguments maybe uh it will be reconsidered but this is the state. So we have 10 minutes and >> yeah. >> Yes. So right um this is my configuration which is already quite long for a configuration that uh for a new style configuration. So I on my local machine uh I set this external commands thing build failed commands which means uh as build will give me a shell like Yen already showed you when the package build package fails. Uh so then I get the debug shell and I can do something to the package and see why it failed or just exit. Um I also turn on running auto package tests which will also run inside unshare. Actually I also tell build that yes I want unshare and I want to build invo to be cached. So I tend to work on the train and then the internet is bad and I really want my change rout to be cached for um I want to say one week but I think this is 10 days. So I get my ts cached and then uh yeah so and then some like two more arguments here. The first thing this unshare mm devstrap dro mangle thing tells um mm devstrap to create a tarball or like a change towel for unstable when I'm building a package that has in debian change log something that's called gml- something uh and this means so I'm I also develop on guml and gal is an overlay dro to debian and we usually target unstable So I don't have a full archive. I always need to build my packages in unstable. And maybe you also have that. Maybe for your company or for your side projects you have a separate archive that's not full archive then you also need this. Um and the last the very last thing is enable app cache. This if you recognize the port this is the typical app cache port and this just tells mm devster please also tell app to use appcach on local host. Okay switching to my config. So I will not read out anything uh everything because it's a it's a big config but just give you some hints. So I use this clean source as you saw uh as Chris talked about it already. Uh I also use an environment filter which is nice because by default SS build some cleaning of your environment variables but not everything and with this you get really an empty environment variable set. So in case you have um a C XXR setup or something then you can clean up with that one. the other way around also to note if you want to have a specialized build you can just set the environment variable for your C compiler and get optimized packages out of S build um the external commands I have as well u I set some extra lintion options you may know I also run auto package test and po parts and then these are more like give you hints uh you can copy them from the slides the uh URL to the slides are on the first uh first page or also I will upload them to the uh the archive. Um so I also have these um out up cacher up cacher nng set up there. Uh then I do some magic because the nice thing is the configuration uh file of sbill is actually pearl. You can run write any number of pearl commands and I do some extra things to use the same commands for p parts and auto package test later on. Uh so for example for bop parts I say no eat my data because I have it on tempfs as said that's faster than eat my data so I don't need that one. Um just as well some examples you can read that later. Uh I have a ccash setup. Um you need some special things there some variables you can have a look and uh also get ccash into auto package tests and set some more variables for that one. Um so as I said look them up as reference and we will also put some more documentation into S build later on. So now I want we want to give you some more hints what to do with sbuild in case you run into problems on it. The first thing is really interesting as I told you before su with unshare back end drops network setup so you don't have network that's actually a bit special because when you run dump uh get dumped into a shell as we showed you before then you have network. So that's really convenient. So it's only removing the network access from BPK build package run by sbuild. So in case you run want to debug stuff, you can just up install anything you want there because it's all there and stuff. Thanks get stuff. But in case you want to run a build with Sbu with network enabled, you can just pass any network to Sbu and get full network there in case you have private packages which has different things there. Also which is really nice is extra repository and similar is extra package. So with extra repository you can ask cans build take this extra repository from somewhere with a normal up nure but with extra package you can also say oh I have a local directory as packages I've already built modified something something I want to test a transition and you just pass the extra uh package directory or some individual files there and as will happily use those new packages and build with that those you so so you can test your transitions completely offline on your computer there with those. Um what else do we have? Um if you want to do try cross builds, S build is really nice as well. So there are two options either there is arch and [snorts] then the pack the architecture or host and then the architecture and the difference is that if you apply arch it will actually use Q mode to build the normal package. So we will do everything as it would run on the different host architecture. Whereas with host it will do an extra actual cross build. So it will try to install the cross compilers and cross build dependencies and then just build the cross thing not running the tests for example because that usually doesn't work. And so you can try that one out for a lot of packages that works out of the box already. Thanks to Helmwood for doing a lot of work there. But if it not works with host, you can actually just use arch and get your package cross comp uh compiled for the different architecture. This is really nice. Um then if you want to work on packages there's I mean do you want to iterate fast? Uh I had already something in there how to use ccash with su to speed up your compilation. Also up cach speeds that up. If you also run want to run it without running the tests all the time because you probably look into some different bugs just provide this profiles no check and will not run the tests by default if your package is set up correctly obviously but that works by default with newer dev helper. Um, also this one is really nice. Um, if you want to work on the package and you have it your debug shell open, h I will do it here. Uh, and then you want to work on the files in there and you want to have your normal editor set up and stuff like this. And you may be used to do that by mounting your home in there. You can actually look peek into the chain root sil created from the outside. This is what you have to do is you have first to get the uh run the unshare command which gives you like the username space to be in root user in there. You can edit all files and then uh the sbu change route is in by default depending on your temp fs or uh temp setup is by default in temp temps as build some random string and you can edit all files there and just from the outside do that and you can also run screen inside uh the the commands thing like the hooks uh we showed you before. Okay. >> Yeah. So uh thank you for listening. Um, we want you to use unshare. We want you to use setup uh switch your setup to unshare. Uh, we also want to hear about your special cases. So, usually a lot of people have big estrange root configs. Um, Yoken tried a lot or did a lot to make most things work but uh some things need you know twiddling basically and we don't know every each and every exchange config that you have. So please talk to us. Uh please find us here at devcon or file bugs. Um yeah, >> thank you. Thanks for listening. Yes, >> thank you. [applause] >> We have a question in IRC. Um give me a moment. Yes. Do I still need to create chall for every build? If yes, wouldn't that slow down build this? And um does it work with QMU user? >> Um so with the with the unshare back end, it creates it by itself. So this is what I I showed here. Um this was actually running without any configuration. I have no cache no cache as build directory or slash um surf something. So that works just at all. And the other one was with the which user was it? >> QM >> QM user >> with a QM user >> no the user space quo. >> Ah um ah yeah right. So if you run it um if you run ah sorry oh if you run these commands archet 64 it will run with user and create it on the fly as well. Yes, do the right thing. Okay, maybe to answer the middle question which was uh will it slow down your builds? So by default uh S change will call mm dev for you and make a new towel on the fly but you can set uh the it's in on my config. You can set an option to cache the tarball and the built network actually has a wrapper and creates change routes still I think every other weekend or maybe every weekend and then reuse these tar balls. And if you're Yeah, by the way, our recommendation if you have tar balls from a different builder like BB builder, don't try to copy them in. Just um let a strange road make you a new new tow wall. uh and then it will work but uh otherwise it's just a lot of hassle basically so but we are out of time >> okay so we have more questions but we are also sadly out of time so um thank you Johan Chris and of course your for the talk and all the work on this build thank you can we have [applause]