Using Debusine to pre-test your unstable uploads

Hello everyone. Um, welcome to the next session of Debcon 25. Uh, I'm excited to introduce you Colin Watson Watson who is um, longtime Debian developer since 2001 and now he's collaborating with a company Frixian which is working very closely to Dian. And uh it's very interesting to uh hear uh one more talk about new tool uh for Debian which uh helps us to make the project better. So it's about debusine to protest your unstable uploads. Please welcome Ken. [applause] >> Thank you. Welcome. Uh, I'm going to tell you about a system we've been working on called Debucine and how you can use it to make your uploads better by doing more thorough quality assurance before you upload them. Uh, Debucine has more applications than QA as well. And I hope we can also spark some interest in those as we go along. Uh, I'm Colin Watson. Thank you, Anton, for the introduction. And, um, my colleagues Enrico, Raphael, and Stfano are also here. Uh, Carles has had to go home. Uh we're all collaborators at Freaksian which is uh which is a Debian services company that among other things is also responsible for organizing the long-term support init initiative for Debian. Uh so this is the usual best practice for doing cross package testing today. Uh it might also involve a trip through experimental if you're feeling extra cautious or depending on the freeze rules that are in force at the time. And of course, not everybody uses sali, but uh I think this is pretty much what the the the median developer, if you like, relies upon. Uh so could you put your hand up, please, if you've ever uploaded anything to Debian Unstable. Okay, good. Uh put keep your hands up. Keep your hands up. Uh put your hand down if you've ever uploaded anything to unstable that broke another package. Okay. Right. [laughter] Put your hand down. Okay. I was good. There are still a few people with their hands up. Uh, put your hand down if you've ever uploaded anything to Unstable without waiting for Salsa CI to finish first. Okay. So, there there are a few people who have impeccable practices and maybe I don't have anything to offer you, but the rest of you uh I hope you'll listen to the rest of this. Uh so we have uh distributionwide CI in unstable. It's great. Uh lots and lots of uh complicated systems that run over all our uploads and make sure they're uh as excellent as they can be before we put them into testing. But there's only one of it and it's shared between all of us. Not just of course not just all of us in this room, but the thousand odd Debian developers. And as a result, it's very possible to get into tangles when multiple people are working on uh somewhat adjacent things at the same time. And we only avoid uh complete chaos uh as much as we do by quite careful coordination with things like transition bugs coordinated via the release team. Experimental does help of course. Uh but again, there's only one of it and setting up another one is very far from trivial. uh there's a lot of complicated machinery involved. So uh what we want is a system where you can run experiments on possible changes that you might want to make to Debian. Maybe you haven't even decided whether it's the right thing to do yet. Maybe you're just trying things out. Uh but you want to experiment at a large scale without a high setup cost and without fear of uh breaking things for other people. Uh and then if it all works then you can actually commit to pushing the whole lot into Debian. Uh so this is where debut scene comes in and it's what originally got me excited about it. Uh when I talked to Raphael about maybe doing some work for Freakian and he pitched this to me. Uh we now have the we know we now have a slightly loftier goal of providing an integrated solution to build distribute and maintain a Debian based distribution. Uh that's because we realized that uh a lot of the things we were doing were useful for Freaksian's extended LTS project uh that funds developers to extend security support for old Debian releases for up to 10 years. Uh and uh we we think it might also be useful to other Debian develop Debian derivatives. It may also be useful for Debian itself to use. Uh though Debian's infrastructure teams do of course tend to be rightly very conservative about the critical pieces that we need to to make the distribution work. Uh but we'd very much like to work with anybody's interested on that front. Uh by the way, it took me an embarrassingly long time to realize this after I started working on the project, but uh us is a French word meaning factory. So deb usine undeb us is a factory for debs. Uh all right so all the people working on debucine are Debian developers. Uh we've got nearly a century of Debian de Debian experience between us which made me feel very old when I when I did those sums. Uh we're working on this because we want to make Debian better by making it easier for people to do effective CI at the distribution scale rather than everybody having to put together their own infrastructure. At the same time, uh, obviously we need to pay our bills and that sort of thing. So, this project's been generously funded by Frixian as well as by, uh, a grant from the Sovereign Tech Fund in, uh, in Germany, uh, that, uh, covered our work last year in 2024. Uh, collectively, we've put over 6,000 hours of work into it so far. Uh, we have an instance on debutcene.debient.net. Uh, this is intended purely for Debian's use. If you're a Debian developer or a Debian maintainer, you can sign into it via salsa. We also have an instance on debin.frxian.com. Uh this runs Frixian's work on extended LTS. Uh but we've also begun to make it available to customers who pay us to use it and to uh have some say in our development priorities. We'd very much like to make this a sustainable project over the long term. Uh [snorts] so if you work for an organization that might be interested uh please get in touch with us after this talk. Okay that's a sales pitch. Uh we started out then by building the nuts and bolts of a system that could run a variety of QA tasks within the same infrastructure. Uh to begin with this inevitably seemed a little bit like reinventing the wheel. So uh la last uh last autumn by the mini debcon in Tulus uh we had capabilities that were in the same sort of ballpark as salsa and so of course it was reasonable to ask why we'd bother because salsa already exists and is widely used but uh CI on a single repository is mainly about making sure that the contents of that repository are as high quality as possible. What we're trying to do is more about the middle entry in that uh motivation slide uh few minutes ago. Uh so we started adding things like a way to run auto package tests on all the packages that depend on your package. Uh of course that happens if you upload to unstable or experimental. But this way you don't have to worry about stepping on anybody else's toes. You can just work completely autonomously and uh play about. Uh we've also begun planning for a way to rebuild all of your reverse dependencies in in our system. Either single stage like the the RAT tool that stands for rebuild all the things or uh multi-stage like the sort of thing that you do if you're planning I don't know a Python version transition or a Haskell compiler change or one of those things that requires a a a stack of staged uploads uh step by step. Um, of course, once you do this start sort of thing, it starts becoming pretty hard work to analyze the results. They're probably not all going to be passes and the the previous state might not be completely clean either. Uh, so we're working on a system to uh track and report on regressions against a baseline. Uh, again, this is a bit like the system for migrating from unstable to testing, but [snorts] uh without committing to anything in the Debian archive first. If you're using dootngng, we have integration for that. Uh this saves you uh having to learn how to use our debut scene client for simple cases. Uh so this uploads all of the files in your uh source package with apologies to Ian uh and then uh starts a suitable workflow on them. Uh you can ask Debusine to do the upload to the target archive for you if it all works. Of course, you have to provide a signature to do that. Uh we've begun using this for contributors who are uploading to ELTS. In that case, Debcene also acts as the official build demon. Uh so developers sign the source package, Debc signs the binaries and then uh Debcene uploads the lot. Uh we have secure boot signing. Uh so we've got uh separate signing workers uh that can hold keys either uh with either for low value keys with uh software encryption at rest or for anything a bit more important in UBHSMS. Uh we needed this once Buster moved into ELTS uh because that in Buster already had support for signed bootloaders. So so to avoid regressing uh we had to have a way to issue security updates for those. Uh we've also begun working on hosting app repositories in Debusine. Uh so you you might call these PPAs if you wanted to. Uh we're we're thinking about better names that are a bit more uh clearly distinct from Ubuntu system. uh which I used to work on which uh so maybe something like I don't there there's a pad with a bunch of ideas for this uh PDAs for personal Debian archive or I quite like NDAs for not a Debian archive. Uh there's there's a few different options. Um we've been planning this for some time. We designed a lot of the core data structures here with with this in mind. Uh but we only really started on it properly a couple of months ago. Um, nevertheless, we're at the point where you can mirror a repository from somewhere else and republish it from debris scene with snapshot capabilities uh as long as we've got enough storage. I should say something about tag to upload uh because uh there's an obvious overlap uh in in some cases. Uh right now uh I I'll confess that debut scene doesn't really work with type to upload which makes me a little bit sad because I like both systems. uh you more or less have to choose whether you want debusine to help you with your distro wide CI or whether you want tag to upload to help you make the the source package and uh track the life cycle there. You can deput to debine cancel the upload to Debian after it's run all the tests and then use get dead push separately but that's that's quite a clunky experience. Um all the same type to upload seems like a very good thing for Debian. uh we'd like to figure out how to work how to support it in a way that feels idiomatic. Um uh and we we do have an open issue about this. We've been talking about it. It doesn't feel quite sensible to tag a release and then just upload it. Internet because CI might say you have something wrong. Um so we started thinking about some kind of counter signed conditional tag arrangements or something like that. Uh we'd also like to support support to support uploading to debutcene hosted repositories using tag to upload and at that point maybe we want to host the web hook receiver and run some part of tag to upload within debucine. Uh I'm not quite sure of exactly how that will end up looking. Uh but uh I expect we're going to figure out how to integrate this uh over time. Please get in touch with us if you have ideas. All right. uh if you're if you're doing uh auto package tests across the whole distribution and rebuilding piles of reverse dependencies and things then obviously you generate a lot of load very quickly in very bursty ways. Um Frixian has a dozen or so workers attached to each of our instances to to handle u the sort of thing that if you're running par sessions you talk about as best load. uh but uh we expect a lot of bursty load as well. Um people like uh I don't know if they're in the audience but people like Lucas Nusbal and Santiago Villa who do a lot of uh rebuild testing for Debian tend I believe to use some kind of cloud provider to do so whether that's uh the French grid 5000 system or AWS whatever uh that sort of thing makes makes a lot of sense here. Uh so we do have basic cloud uh uh scaling support for both compute and storage. In the compute case, this takes the form of pools of workers that can run tasks for us and that we can scale up or down depending on our task queue. Uh in the storage case, all our files were already content addressed with uh references in the DB and our local file store already looked a bit like an object storage system. So uh hooking up cloud-based object storage was pretty easy. uh we've we've tried to avoid depending on just one big tech provider here. We don't want to be dependent on Amazon working or Google working or whatever. Uh S3 compatibility is pretty widespread for object storage. Lots of providers implement something compatible, but uh cloud compute tends in practice to be a bit more provider specific. Uh so far we've implemented both Amazon EC2 and HNER cloud to give us at least minimal diversity. uh but there is support for plugging in others if we decide we need to do that. Uh we try to avoid lock in so the core service doesn't assume any cloud provision at all. Uh and we we have support for migrating files around between different object stores and that sort of thing if we need to. Uh that said on on our main instances on our main instance right now we only have about a terabyte of files. So it's actually cheaper at the moment for us to just self-host that. Uh as we as as more of you start to start to use Debine, uh we expect that this number will grow. Uh all of our development work is in the open on Salsa. Uh the project is straightforwardly GPL. There's no contributor license agreement or anything like that. Uh if you're a Python web developer, it's actually a pretty nice project to work on. Uh we have full type annotations across the whole project. Uh 100% test coverage uh or 100% unit test coverage uh uh across branches and uh lots of documentation. Uh we have some byte-size tasks in our issue tracker on salsa. Uh if you're if you have things you'd like to change, we're very happy to help you. Uh this is a project by Debian developers for Debian. So of course it's also in Debian. We deploy production on Debian stable from packages that are built from git by SalsaCI uh so that we can deploy fixes reasonably quickly but it's all the same packaging that goes into Debian. Uh when we aren't in a freeze we usually make a couple of uploads a month. Uh if you like you can install the whole system for yourself from these packages. uh our documentation covers how to do that, but most people will just need the Debc client package uh which lets you interact with an instance that's hosted on somebody else's computer. So, uh let's have a quick look at some of the basic pieces involved in Debine. As a user, you don't need to have a detailed knowledge of all of this, but it is helpful to have a quick vocabulary lesson. uh if you want to help us extend debut scene then you will need to know more about this and to start understanding more of the detailed design involved. Uh so uh workspaces are the first piece they're the uh they're where they're the container within which everything else lives. Uh they can be they can have various access control. Uh they have various expiry rules. Uh so you might perhaps have an ephemeral workspace that only lasts for uh two weeks after the last thing you did in it. Um on WC Debianet we have uh two sort of main workspaces that that you might need to see. Debian/base has all of our infrastructure like mirrors of the Debian archive so that you can build things against them. uh the uh rules for building the container and VM images that we use to launch all of our tasks. Uh and also the de Debian/developers workspace which is very roughly it's for things that that somebody is intending to enter Debian but it's not a it's not a one-toone correspondence with Debian because some of those uploads might uh might might break in CI in some way. uh workspaces can inherit from each other. So developers inherits from base. Uh if you try to look something up in developers and it's not there, we'll look in best instead. Uh and we have this we have an idea of experiment workspaces which are uh ephemeral workspaces based on another one. Uh the idea here is that if you are uh that if you're playing around with something, you can ask us for a workspace that uh is just owned by you. Uh you can do whatever you want in there. Uh and uh you can set things up in it like your own workflows, your own uh uh published archive, uh all that sort of thing. [snorts] Next major piece is artifacts. uh the these are the the lowest level uh storage abstraction in debus scene. So uh you have some data and a few files. Source package is probably the easiest thing to look at to understand this sort of thing. So for a source package you have a DSC and maybe something like a Debian Tartgyz and a and a Ridge TXZ something like that. Uh so you've got a few files that are very closely associated and that travel around together. Uh the there's also metadata extracted from those. So for for a source package, you normally want to know at least the the source package name and version and what architectures it wants to be built for. Uh there are uh a whole bunch of there there's a whole zoo of other uh artifacts uh implemented. uh Debian linten for instance is a a result from running linten but parsed in a way that you can uh uh search effectively across multiple artifacts uh and lots of things that are used for debcen's own purposes as well. Uh all of the files and artifacts uh live in an object store. Uh they're stored by hash. So if uh this same artifact uh sorry if multiple artifacts share the same files uh then uh we we won't duplicate storage for that. Uh collections live above artifacts. They're sets of items that form something more complicated. So a suite is the is the easiest example of this. A suite contains a whole bunch of source and binary packages. uh and you can look up uh uh packages within a suite by name and version and maybe architecture. Uh an archive contains multiple suites. Uh so you can have collections of collections. Um some uh some collections are just of basically blobs of data. So the results of QA tests don't necessarily involve any actual files. Sometimes they're really just small bits of uh parsed metadata. Uh collections can control the expiry of the things they contain. So uh if you uh if you have an artifact that that is in a suite, then it shouldn't expire until it's no longer active in that suite. Uh and uh in some cases, you might actually want a little bit of uh uh some kind of grace period there uh to make sure that things don't disappear immediately. uh and uh there's there's a quite a sophisticated lookup system for uh taking a collection and saying I want to find an object within that. Uh so uh once you once you've got all of those primitive operate primitive pieces uh you need some way to ask Debc to do things for you. uh work requests are the the lowest level system for this. Uh so uh when you create a work request, you give it a task name, you give it some task data, which is a a key value sort of thing. Uh and the task data would usually say uh here are my input artifacts, the things that uh I want to to uh build or test or sign or or whatever. uh and some options for how it's going to do that. This is very kind of arbitrary code execution engine, right? So, so you obviously want some kind of sandboxing to make sure that it doesn't immediately go horrible. Um we we execute most work requests on an external worker. Uh that's usually sandboxed in some way. Uh we we support at the moment uh unshare for uh for for sort of lightweight kinds of things. uh Incas containers, Incas virtual machines and QME virtual machines. Uh the instance administrator and the workspace owner between them can control what uh what levels of sandboxing are needed. Uh work requests usually produce some kind of artifact uh as output. It might just be the log but uh more usually it will be will also include some files like auto package test output or debs or something like that. and those will often be immediately added to a collection. Uh so yeah, some examples of work requests, s build, uh auto package test, uh package upload, all that sort of thing. Workflows are where things start to get pretty special because they let you tell Debusine to do many tasks with some ordering constraints and those can involve tasks across lots of different packages. So for example, we have the reverse dependencies auto package test workflow uh that finds all of the packages in a given suite from Debc's point of view that depend on or declare a test suite trigger on one of the binary packages that you pointed at uh and then runs all of their auto package tests against those binary packages by creating sub workflows. Um eventually we want to allow for custom workflows once we figure out a a suitable domain specific language to for for doing that sort of thing. But for the time being we have a number of built-in ones. Uh so for most people probably the most interesting one is uh the Debian pipeline workflow. Uh that's that's quite powerful. It stitches together a lot of other pieces. So it uh it builds the package. It does all of the usual uh single package CI tasks. Uh, but it then also runs all of the distributionwide CI that we've written so far and it has the option to upload your package to Debian at the end of it if you so desire. All right, so uh that's a a very quick tour. Uh we promised to show you how you can use these features to do QA work in Debian right now. Uh so here are a few live demos. Um [snorts] I hope everything is up and working for this. Uh these only really scratch the surface because Debuscene is really a feature a framework for running lots of different tests in a structured way. Uh and if you have have ideas, we hope you'll come to our uh debusine workflow buff after lunch to talk about what else we could build in this framework. So uh let me see what I can do here. this uh take we're going to take a source package with a few reverse dependencies and uh I've chosen uh zip zip.esting which I happen to maintain uh and uh sorry it's a little hard to type this way um and uh I've prepared a a very simple modification to that uh so I'm going to deput that to wc.debian.net that uh and uh it will uh as expected fail in the first step because uh normally it will work out which workflow to run based on the distribution in your changes file. But in this case there's no we don't have a workflow set up for unreleased. Uh so uh in in this case we're just testing. So, uh I'm going to override that. Uh and say upload this to unstable. Uh so that uh uploads all of the files uh to to debut scene. In principle, it's a bit like deeper. In practice, this is uh implemented behind the scenes using the Debcene client. uh and uh at at the end of it, I hope you can see that at the bottom of the screen, uh it uh it creates a workflow. So, uh I'm going to uh if I can grab this and uh open this in debut scene. Goodness, that's going to be a bit giant, but uh hope you can see what's going on. So it's uh it's created a big tree of uh uh of work requests. I deliberately chose one that had some reverse dependencies but not too many so that uh it would all complete reasonably quickly. Uh and uh you can see that uh these are organized into subworkflows each of which does a smaller chunk of the work. So, so this one as I said looks up which packages have predepends or depends or test triggers on the package on zip.esting uh or on Python 3 zip.esting rather and creates an auto package test workflow to each of those. Uh so if I click click through to one of these and look at its dependencies tab you can see that it's waiting for the the builds to finish first. Actually it's not because it finished quite quickly. Um otherwise tasks will start as soon as an appropriate worker is a is available for them. Uh so if you're curious you can uh you can click through and uh look at the list of workers down here and uh see what see what they're all doing. Uh oh maybe somebody else is using depine as well. Uh the so that will um that shows you all of that and uh I'll just reload this and we can see that a whole bunch of stuff has finished. Uh I'm not going to wait for all of this. Uh if I uh click through to the excuse me a moment. Uh if I uh look at the waiting for signature task here, if I can find where my mouse has gone. There we are. Uh then uh this has uh this has some instructions on what you want to do if you want to actually upload this thing. So run dep provide signature this thing. Uh you you might well want to say actually you know what I've changed my mind. I don't want to I don't want to do this. So you can abort that and uh that will if I go back uh cancel cancel the rest of the workflow. Uh so in this case obviously I don't want to actually upload this thing to to unstable. it would be rejected anyway. So, uh so I'm just going to cancel that. Uh but if if I had wanted to sign that, what it would do is uh do something very like debar sign essentially. Uh so maybe you want more control over what Debbie does or uh you want to do a set of uh throwaway tests that uh you don't intend to upload and you also don't want to litter a shared workspace with them. Uh so for this sort of thing it's useful to be able to create your own workspaces. Uh I mentioned experiment workspaces earlier. By default only you can write to an experiment workspace you create. Uh they come with an effect of them so you can add other people to them if you want. Uh once again, where is my mouse pointer? Uh right, let's see. Uh no, there we go. I'm just going to retype this. Sorry. Uh so I'll say uh create a workflow uh uh starting with the developers workspace. Uh and actually I'll show you this uh this example here. Uh so the this is some task data for workflow. It just says here's the name for my experiment. Uh I want it to expire after one day because it's very boring. Uh but of course that might be longer. So uh this is uh create this uh and what have I done wrong here? Uh I have forgotten to say create experiment. Uh so that will that will create uh the workspace for you. At the moment unfortunately this doesn't give give you a URL to the new workspace but if you go look at the front page of wc.dew internet you can find it. uh and uh way down at the bottom. So I'll open that and here we have a brand new workspace. Uh so within that uh I can uh create uh workflow templates. So I can say uh uh uh I'll show you the uh template again. Uh this is uh kind of similar to the one that's configured by default in the developers workspace, but the the main difference here is that it's not configured to upload to Debian at all. So you can't you can't make a mistake with it. uh the um uh you you might well for this sort of thing say actually I want to uh build uh build all of my packages against something else maybe against my private archive or I want to uh uh run all of the tests in virtual machines because they're they're crazy things that involve tearing down bits of network configuration or something uh and uh and so on. So, oops, what did I do? There we go. Uh, so I can create another workflow with that uh or create a workflow template with that. And I can upload that uh same package that I uploaded earlier here. Uh, and uh just using the force option because I already uploaded it a moment ago. Uh, but that will upload it to uh to my private workspace instead where it can do its own thing. You can see that the URL has zip. Demo in it. Uh so it's outside the scope of this demo, but you can also use the binary artifact uh binary artifacts that this produces as additional input to other workflows. Uh it's all a bit manual at the moment, but we're planning to hook things up soon so that you can tell a workflow like this to publish the output into a suite collection in Debine as an alternative to uploading to an upload queue. At that point, you've pretty much got the equivalent of a PPA. Speaking of which, uh let's have a quick look at where our app repository support is at the moment. Uh so uh this uh script here uh is just a a simple thing that uh creates some apt configuration uh and uh so a kind of private bit of apt configuration that I can uh use as an ordinary user. Uh and it this points it at deb.wc.frixian.com. frixian.com/frexianbase uh which is uh Debbie's mirror of all of the stuff in uh uh in Frixian ELTS uh specifically this points to our mirror of the the ELTS repository for Buster. So I can at this point run uh oops I can at this point run uh uh apt config equals point to the new thing apt update uh and that will just do an apt update from uh from debutc all works uh that's fetched the indexes so I can then go okay download uh this is a security update that I looked up that was published on the 1st of July. Um, so so far that's just kind of mirroring. Uh, but I can also ask it about older state. So we started mirroring indexes a couple of weeks ago. Uh, and that means we have uh, some older data as well. I can say I can tell apt to uh, use the snapshot from midnight on the 30th of uh, of June. Uh there's no specific uh uh we don't have like specific uh cut offs for the snapshots. You can you can pick any date you like and we'll just use whatever uh whatever was current at that point. Uh and uh once you once you do that we can then say uh okay pick whatever was current for that package uh at that date instead. And you can see that instead of picking uh -2 plus dev 10 U1, it picked 2.1-2 instead and got the older version. So you can use this sort of thing in the same way that you might use snapshot debian or to look at the state of a repository in the past and uh even run tests against it using something like debisect. At the moment we only retain older snapshots for ELTS and not for our current suites. That's partly to save on space costs and partly because well snapshot debian or is already doing that. Uh but once we have personal archives you should also be able to get snapshots of them for free. Uh so what's next? Uh a lot of our current work uh current road map involves finishing things we started. So we want to make debut scene more useful for LTS and ELTS contributors by finishing our work to support analyzing regressions. uh that's one of the biggest remaining gaps uh with respect to previous processes for for for LTS and DLTS contributors and it's going to be very useful in general in uh in the process of getting your head around uh the the output of complicated workflows. Uh I'm personally very excited to finish app repository support as well. I think that will make a big difference to to how Debian operates. Uh but after that we're really looking for people to embrace Debine to to help us decide on its future maybe even to help us build it. Uh we've had quite a few conversations already even just this week uh with people who have specific use cases. Uh and we as I said earlier we I hope we'll uh see some of you at the deputy scene workflow buff uh this afternoon to explore those further. Uh that's all I've got. I'm happy to take questions. >> [applause] >> Yeah, thanks Colin for uh for uh the introduction of this new very interesting tool and uh it's it's nice to see that in Debian we have more and more tools which help which makes our life easier. So let's go to questions. you uh showed us the like data model basically. Um you're expecting this question, right? I notice you didn't explain how git is git source code is stored in debine. That's correct. I didn't uh it's not currently, but I think it probably can quite easily be. So, uh my expectation is that uh we'll have some kind of new artifact type. Uh you saw me exploring the artifact model. uh it's very flexible uh so you you can have uh pretty extensible sets of artifacts at the moment we do just have Debian source package which is a trad source package but I don't think there's anything stopping us having like debine sorry Debian colon git source or something like that which uh I'm not sure whether we'd store a git bundle or whether we or git archive effectively or whether we'd h just store a pointer to uh to um something as hosted somewhere else. Uh but uh uh it's it's perfectly allowed for in the data model. The only reason we haven't done so yet is because we'd end up building source packages internally anyway. Uh so we because we're uh because most of the work uh that is done by our tasks is done by existing deb existing Debian tools. Most of those expect source packages and so we basically need one for internal purposes. But one thing I'd like to do is uh uh have a task that basically does the equivalent of the tag to upload builder uh that goes from uh a git archive that we stored in one artifact produces another one which is a compatibility source package and then we can feed everything else to that. Um happy to talk through the details of that. I think is very possible. Thank you. >> Okay. Thank you Colin for this interesting talk. I I sorry I I need to um uh correct myself. I raised my arm at the at the wrong place. I for sure I did uploaded something which break other things. And so uh >> haven't real. >> Yeah. I really like this deep interface because it was new to me because this this is really great but uh and and very simple and so but it triggers a question in me is usually I contact duck with deepot is deepen somehow doing a similar top job to duck at some point in time >> similarish so uh uh deeput has deepen g at least it's a bit more complicated With uh traditional deepoot, but with DUTNG, it's got uh a very flexible workflow or sorry uh hook and profile system. Um it's a little more complicated than it needs to be in some places. Uh I've made some fixes to get it to work for us. Uh but uh you can you can tell it instead of uploading to that upload queue, then upload to to this one instead. Or actually, you can tell it to execute basically its own code. That's >> I'm talking about the putng. >> Yeah. >> Right. So um really dutng doesn't interact directly with DAC anyway. It's it's interacting with an upload queue which uh traditionally in deb in Debian there's another piece of machinery that then transfers all of that over to DAC. Um we just substitute ourselves for an upload queue and uh uh and and do what we need to do there. Uh what we what we don't quite have is uh machinery for actually putting your upload into a suite collection which would then be the equivalent of uh DAC uh of of D install effectively. Uh but we're quite close to that. >> Really good. I would have a lot more question but I'll leave it to others now. Um, [clears throat] yeah. So, one of the pain points that I have with uh Debian and auto backage test in particular is running those on lesser common architectures. Is that something uh Deben can help me with? >> Uh yes, though maybe not immediately quite yet. Uh we have AMD 64 and ARM 64 workers right now. Both of those I guess you probably count as common. They're they're reasonably widespread. Um, but there's there's absolutely nothing to stop us hooking up uh uh others apart from just getting hold of the right hardware and putting it in the right places. Um, what what are you interested in particularly >> all the supported architecture? >> Right. Uh, we want to get there. It may take a little while to deal with procuring all of that. Uh, but uh, but we do want to get there. Yes. Okay. >> Yes. >> The the model um is very generic uh which I like. Um what are your future plans for um using that genericness to be able to build other things apart from debs? So could this be a future OBS type thing? As you can yeah as as you observe it is very generic. We've carefully namespaced all of the things that we think are very Debian specific with Debian colon at the start. Uh so that we we don't design ourselves out of that. Uh I think probably for the short term we would we would rather do a better job of fewer things rather than doing a bad job of everything. >> [snorts] >> Um but uh there's but there's plenty of room for it. If we had customers who wanted us to to build that then uh it's then we're totally open to that. Uh the current team doesn't have a lot of expertise with RPMs or uh arch packages or or whatever. Um just because we're we're mostly Debian specialists. But uh there's there's there's space for it. Okay, any more questions? It's not really a question because I'm [laughter] part of the team, but just to uh tell you that the the view that we have or that at least I have is that we want this to be something integrated. Obviously we're trying to integrate it with the current Debon situation which means well uh we have to start with source package and we have to upload them back to to Deon but we can only upload source package right now but obviously if we test binary package and we know they're good we would like them to be used directly instead of being rebuilt in a different environment. So one of the obvious and interesting next step uh for us is to help convince Ourelia and the other from the build team to uh consider rely on debin to uh build the debian package the official Debian package and uh yeah we really see that as a uh ultimately as the official Debian CI uh that uh you use automatically when you do your work uploads Don't know how many people uh are worried of this future or would like to see this future. Maybe we can have a raise of hands who would like to uh see an official workflow from scratch from the initial upload up to the upload to Debian maintained by a single unified architecture instead of many random tools. >> Okay, not too bad. >> Okay, promising. Thank you. >> Okay, please. Last question. Um, thanks for the talk. You you Yes. Um, you mentioned uh that it was um doing what salsa is already doing, but uh would it be imaginable to plug salsa to trigger debuts in jobs? >> It's definitely a possibility. I think one of my colleagues was doing a bit of early exploration of that uh at dev camp. Uh we I I think we need to work out what to what extent some of that is useful and to what extent it ends up being confusing uh because if you have uh I mean we have a lot of multi-layered systems in in Debian. Sometimes if you end up with all of them calling each other, it can be quite confusing to work out what does what. Uh but uh I'm not in principle opposed to the to the idea and we may end up with something like that. Yeah. >> But really lost. Okay. >> So this seems like a really really good tool to spend all the disk space in the world. >> Yes. um in addition to all the CPU of course uh except that which is being consumed by LLMs and scraping. [laughter] Uh so what kind of controls do you have or kind of reporting and and similar do you have for for managing that in a way where we don't end up with uh people accidentally or intentionally consuming like significant amounts of of uh capacity because >> if we are going to to adopt this for for Debian and make this more generally available I can very much see people consuming hundreds of terabytes of this space before we uh yeah >> indeed uh we have quite aggressive expiry really. Um in some cases a little more aggressive than I'm comfortable with. Uh although I a few weeks ago I discovered a bug where uh we were we had uh accidentally not expired two terabytes of data. So oops but uh so there's it's moderately aggressive right now. We don't currently have great reporting for workspace owners. So uh so most of our most of our reporting is more at the instance administrator level. Uh I would like to expand that to workspace owners because once we well from Freynium's point of view once we start having customers uh who are using this uh for their own purposes most likely they'll want to to see how much space they're using. uh but there is reporting at the uh uh at the instance administrator level where we can see what's being used by which uh uh which sort of scopes. Uh and uh I expect to expand that. Um most likely we'll we'll end up needing something where you can where you can see what is what is keeping artifacts non-expired. Uh so there are only a few reasons why artifacts can can be kept. Um there's like explicit expiry or being referenced by collection or being referenced by relations from other artifacts. Uh and uh uh all of those are quite easily data database introspectable from our point of view. So we can uh we can expose that a bit more publicly. >> Okay. Thanks Colin. uh one more time for your nice presentation, for your work uh on this tool and the whole Frexen team for doing this uh job. Please continue to do it and keep us updated with news from from this tool and I can only promise that I will try to use it for my personal needs. Thanks a lot. Please welcome uh this grid for one more time. [applause] Thank you.