tag2upload - upload simply by pushing a signed git tag

obsolete, but getting rid of them is a big and complicated effort because they're kind of in the core of everything that Debian does is the source code. Uh this what I'm presenting here um is the next phase of our git transition program. Uh so with tag to upload you can deliver your updated source code upload to Debian entirely using git and you don't need to have any non-Git things locally. So, I'm probably easiest if I demonstrate. Now, I need salsa to be up for this, but uh we'll we'll see about that. Um so, first thing I'm going to do is just make sure I'm up to date with the recent branch and check salsa is actually working. Uh if this doesn't work, yes. Whoa. Right. So, um, I'm not going to actually make any changes, uh, because this is just a demo. Oh, that's not what I meant. What I meant was this. You can tell it's a live demo. I make a mistake. Uh, let's just make a change log entry. Um, you'll see that I'm uploading to experimental. That's just because we've got the freeze on at the moment and I don't want to disturb the freeze. So, that's why it's going to experimental. So we'll just commit that. And now my tree is ready. Now there's just one other thing I have to do, which is I have to make my PGP key available because I'm going to do a real upload to Debian. I'll do that over here where you can't see it. Right. Uh, and then I just do this. That's it. My work is done. [applause] So, but I suppose I should probably show you what that actually did. Uh, so you can see it pushed a tag to Salsa. So, let's have a look at that tag. So, [snorts] it's quite simple. It's just a normal git tag. Uh you can see it's got some like weird square brackets dit stuff in. Um and that's kind of the uh the critical part here. We reuse the metadata format that DG uses. Um and this particular set of metadata has this um please upload part in it. That tag is an instruction to the tag to upload service. So when I make that tag, git deb push made that tag for me. Um, you could make the tag some other way with some other tool, but you probably want to use git debush because it like checks and various stuff and make sure you don't mess up too badly. Um, that whole that whole PGP sign thing you can see there is an instruction to the tag to upload service. It's completely complete. It says do the following thing and send it to Debian. So the service will take my code and do all the stuff with it. So let's see. We've probably got the service page here. Yes. Let's reload that. There's my upload. You can see that's job number 274. Uh version 13.7 like we had. Um and uploaded I'll just scroll right. You can see uploaded to experimental there. Uh you also get an email. Here's the email. Just a report from the service about how everything went swimmingly. Uh right now there are occasional bugs and of course you can make user error. So it is possible to get a email saying, "Oh, sorry that didn't work." Um right now we're reading most of those just to see what kind of things are going wrong and and try to fix them. Now the thing that I uploaded there you can see is immediately available here on this DG git server. So there's no like there's a little bit of processing wait the tag to upload service did a little bit of processing but you can see it only took a couple of minutes. Um, and that's not driven off chron jobs. That's kind of live. And here it is already available. And you could get clone this right now. Uh, and get that upload that I've just done. The tarballs and patches stuff goes to the archive. So, it's made an actual source package. It's sent that to FTP debian or presumably in due course I will get some email from some chron job telling me my package was accepted. Uh so right now that part is just the same as it was before. So this is kind of a new front end at the moment. Okay. Um so back to the slides. That completes the demo. Uh so back to the slides. Um, we've noticed that there's often a bit of confusion about where some of this git transition tooling fits into the whole picture. So, I thought I should speak a bit about where tag to upload fits into the scheme of things. Uh, in particular, when adopting tag to upload, you don't have to change the way you're working at all. How you develop your package. Uh if you're using GBPQ, you keep using GBPQ exactly the same way. If you're manually committing quilt patches um into Debian patches by hand, you can keep doing that. Although really we think there are better ways nowadays, but if you if you like it that way, absolutely carry on. The local test builds all the same, just as you're used to. Um all that changes the tag to upload is the actual upload step. So you prepare your package. Once your git tree is in the state you want it to be in, you've committed the change log change. You run git dev push and the automation takes it from there. So I hope I've shown how convenient it is. It's super fast. You can do this on like if you got this all if you already got a git clone of this thing locally. The amount of data transfer is really small. You can do it on a train. No need to like transfer tbles about. Um, it's got a very pleasing kind of wizzywig approach, right? The stuff that you've got in git is literally what gets uploaded. There's no kind of weird stuff going on there. It's so it's really convenient. Um but also it's more traceable. Um the conversion there has to be a source package because the whole archive downstreams build these everybody is currently using source packages. Well most downstreams are using source packages. Some are using dit now. Um uh that stuff has to be generated somewhere. But with tag to upload, it's generated automatically traceably by the tag to upload service which has a very high level of software integrity. It's got like three layers of privilege separation um quite well put together. It got a good security review from Russ Aubry um who made a few suggestions which you've adopted. Uh so now the source package that ends up in the archive is directly traceable back to that git commit and the git objects are archived on the dgit git server. That's git.d dgget.debian.org that I showed you the um browse view of earlier. Um right so status currently it's in open beta um so you can use it to upload I just uploaded dit to experimental but we've also done I've done a number of uploads to unstable with this um it seems to work quite well uh it doesn't always catch all of my mistakes the way that dit does so [laughter] um and I'm I'm quite a scatty kind of person nowadays. I make a lot of mistakes. So I like the software to check that for me. But we are improving that. Uh so you can try this out right now. There's this wiki page. You want to read that wiki page and you definitely want to read the git dev push man page. You want to install that package from experimental because the one from experimental has a lot of user experience improvements and sort of general better things. Uh after Trixie is released, obviously that will go into Forky and then we'll be recommending you use the one from Fory or possibly backports. You can use Trixie's git de push if you really want to. Um I don't know if for some reason that's what you have to hand. um it's just a bit more likely that you'll get something wrong or that it will do something wrong. So during the freeze, you want to be a bit careful. Don't upload directly to Unstable as your very first test with this stuff. And [laughter] I I hope I don't need to tell you this, but sometimes people get a bit gung-ho, right? It's a new tool. It might go wrong. Uh so try it out in experimental first. just to gain a bit of confidence with it. So future work um there's a lot more to do here right converting Debian to run on git instead of source packages is a huge deal. I've been thinking about it for at least 15 years, maybe longer. um and DGET which was the starting point for some of this was in 2013 in Vaku. So there's a lot more still to be done. We need your help. We need your enthusiasm. We need you to adopt it. But also um there's near-term and long-term stuff that we we'd like a lot of help with. Um so one of the obvious things that we want to do soon is we want to support pristine tar. Uh currently uh if you do a new upstream version with tag to upload the tag to upload service will use the upstream git tag that will be mentioned. I I I didn't show that in the demo. The demo was just for a native package. But with a non-native package, you mention the upstream git tag for which what the upstream source code is and the service will if necessary if there's not an or in the archive already. So for a new upstream version it will make a new one with git archive and that's great if you want a fully git workflow. Uh we really like it but a lot of people really still like to use pristine tar. Um, and currently that's not supported. Um, so that there's a bug about that. The bug is pretty well scoped out. If you you could go and read that bug and it will tell you like, am I the person who could do this? Um, so we definitely want to support that. Uh, this is a general theme that we have with with with tag to upload with DG. Um, we have ideas. We have lots of strong opinions actually about what the best way to do things is, but Debian is a very complicated place and people do things in a lot of different ways. So, our tooling is really quite unopinionated. Um, it likes to support everybody's workflow. It's trying to meet people where they are now and just uh improve everybody's lives. So that's why although I really don't think pristine tar is a good idea, that's why we've got this bug report that says like yes, we want to support it. Here is how you know what that would involve. I can't do it myself cuz I don't use pristine tar. I don't really know how it works. Um, another thing that's really uh close by and would improve things a fair amount is being able to do local builds using only git and without having to deal with source packages. So Debian's current tool for doing a formal local test build and getting binaries is Sbuild. And SBU doesn't know how to transfer the source code from outside into the build environment other than as a source package. And there are some horrific workarounds that you can use, but there it is. It's not great. Um, and it's all a bit kind of clumsy. We would like SBUD to be able to do this. There's a bug open against SBU. We're not the SBU maintainers. I don't think that, you know, this this is a the bug has got like sympathetic noises in it, but we need somebody to do the work. So, in the longer term, we've got like more stuff that needs doing. Um, there's a couple of items that are kind of stuck at the moment for the usual Debian kind of reasons. Uh, we'd really like you to be able to t do a tag to upload upload to new for a new package and that yeah um there's there's there's there's quite a big blocker in the way there. Um, I don't know how that's going to get unstuck. We'd like to be able to do tag to upload for security uploads. I remember a few years ago I was doing a security upload and I had to do it not with with like the git based workflow with DG. I had to do it like manually with a source package and I'd forgotten how to do it. I I had to like it it it took me like 45 minutes to go and read all these man pages again and go like what do I do next? [laughter] It was quite uncomfortable experience to discover like yeah I I and and I wrote the program that drives all those tools and I can't remember how they work. >> [laughter] >> Um, so that's non-trivial because security has embargoed stuff. So we need somehow to have embargoed git branches and we need like you know the security team are working very hard dayto-day to deal with like actual security updates and what I'm what this needs is some kind of a infrastructure update where you can kind of where the the the git server that has the the secret source code can discover when the source code has been unembargoed stuff like that. It's it's not straightforward. Uh, and I'm hoping that somebody in the room or somebody in the next few years will figure out what that should be like and we can do it. And then there's some kind of blue sky stuff. Uh, eventually we're going to want to import all the old tarbased uploads into git. Right now, I mean, there's nothing really actually stopping that from a technical point of view. Right now uh we've got the git server. Um it currently receives uploads done with DG not uploads done with you know tag to upload uploads go there as well. Uh DG has 10% adoption. So 10% of package uploads are on that git server. If we were to start importing all of that into that git server right away, we would discover like firstly it would like blow up the server hugely immediately and also it would like we we'd discover a lot of corner cases quite quickly. Um doing this you we think that tag to upload will be very popular because it's so just it's just so nice. Um, and if it becomes popular, then that git server will grow organically to a size where starting to import all the tarbles won't be such a huge change. Um, so we should already be thinking about exactly what that should look like. Do we want to import the whole old tarble history of every package somehow? Um, etc., etc. Um, and I know there's a lot of people who have opinions about this. And if you have opinions about this, come and implement them. Um, if we do that, we'll have a uniform git view of every package. You will no longer need to deal with app source. Once we've imported all the apps source stuff into the git server, you'll be able to git clone any package in Debian and get some kind of git view. Now it might just be like a series of tarble imports. If you if it came via tag to upload or DG you'll get the proper history. Um you can get that more or less right now with dit clone but it's a very fat client and it does a lot of like complicated stuff. Getting this just via normal git server will be absolutely great and that would enable starting to change our internal processes to be actually git based. So we have a lot of stuff on the maintainer side that's git based. We do a lot of stuff in salsa but build these new processing all that stuff is still source packages and build these getting sourced from git is a like obvious first step once we have a git view of every package. Why would we deal with source packages on build these? Why would the build these not do git clone like any other like um automatic system? So if you like to work on any of these things and are interested, loads of contact details here, get in touch. We're really quite friendly. We we try to be nice to people. Also, um you know, we we ask that everybody who who joins our team takes the same attitude towards user mistakes that we do. like our view is that the software ought to do the right thing and it ought to help you not mess up. So we try quite hard um you know if somebody says to us huh my thing went wrong we're like well what exactly were you trying to do but it definitely shouldn't have done that right usually the answer is it should have spotted that. So that's the prepared part of my talk. So I think I'm on to questions now. [applause] Yan, thanks a lot for your work on this tool and for presenting it it here. Uh so I see you have a lot of questions. So step by step. >> All right. Uh thank you Ian. Uh I've used T upload once or twice. Uh it's been a very nice experience. So uh I do appreciate that. Uh I do a lot of work in uh teams that maintain a very large number of packages and uh of course one of the one of the steps in tag to upload that's alluded to in that tag is sorry on that slide uh is uh setting up the web hook to tell the uh tag to upload service what to do. Uh, are there plans for making that process more automatic so that it doesn't involve clicking through teen uh options on salsa first? Yes, is the short answer. Uh, we think that would be great. Um, there's no reason why this shouldn't be done by default. Uh, if somebody who knows salsa API stuff better than us would like to maybe talk to the salsa team about doing this by default or do some kind of scripting for that, that would be great. We've already got one person who did this automatically for a large number of packages in a particular team. Um, and we they did ask for their script, but unfortunately the contributor um very frankly told us that they'd used an AI slop generator to help them write the script. And both Shan and I decided that we didn't want the mimemetic pollution involved. And we we apologetically told the this person that we unfortunately would not be reviewing their code. this person was me. [laughter] So, [applause] so basically um I I can tell a little bit if you it's okay. Go ahead. I I was one of the first testers of this tool. I was excited. It it works for me. I even did upload into unstable. I I I put u I kept the release u rules and so uh it works. So as as we had a couple of um problems which were fixed by guys very very quickly. It was really uh nice to communicate with them. Uh but then um I decided to propose to enable um this feature for all packages in DBN science team. So it's over 1,000 repos I think 1,200 or something. Uh I wrote a script. Yes I use a little bit help from uh copilot. I checked it so it works and I enabled uh communicated with the team and I enabled it for all repos and it works. So I mean uh all uploads into Debian science can be done over this tool without any problem. Uh we communicated with the team. So if somebody is not agree with it with it, he can just remove web hook on the salsa and I didn't hear any complaints. So for me it works. I have done my work as fast as possible. So yeah. >> Yeah. I just wanted to come back on a thing Anton said there about like the web hook. Um so it's a piece of config that lives in your salsa repo settings, but it's completely harmless. All it does is tell Salsa to notify the tag to upload service about new tags. And if you push tags that don't have this please upload thing in the service just goes like this tag is not for me I ignore it. Um and it the salsa just does this in the background. It doesn't get in the way of anything. So there's absolutely no reason you wouldn't upload you wouldn't do this for like every repo on salsa would be plausible. Um, certainly every repo in the Debian team, Debian project team on Salsa, uh, would be a no-brainer. Um, I'm just not the person to do that. >> Okay, thanks. Um, please go with your >> Hey, thank you very much for the presentation. I really like the modern workflow here, but I'm wondering when we're using a pure git based workflow, how do you deal with edge cases like empty directories which cannot be really covered in in git? Like sometimes the oric tarable contains test data with like an empty directory. How do we deal with that? >> Right. Um so that kind of thing is becoming quite rare nowadays. Um because like that can only happen if upstream aren't using git, right? If upstream are using git, they don't have empty test data directories in their git tree because git can't represent them. So yes, you may need like if you want to take such an upstream release and import it into git, you may have to do something. You may have to add something in your Debian rule to recreate the empty directory before you run the tests. Maybe I don't know exactly what workound you you do for that, but you know, yes, you you'd have to do some kind of little bodgy workaround, but given that nearly all upstreams are using git now, it kind of it's very strange that we and Debbian have this practice where we take the upstream tarble, which is not the upstream source code. It's some weird thing that they make with a script because they think we want it. And then and then we take this tarball and then we like scratch our heads and delete all the things from it that we didn't want that weren't in the maintainers git. [laughter] And then we import the result into git. This is this and then we export it back out of git when we upload it. This is all just complete madness. We could just use git. And there are you know right now I don't think the system will work for all packages. It will work for the vast majority of packages. There are possibly some whose git history is too large or that have other kind of like anomalies, but those are going to be a vanishingly small number and we should at least adopt a sensible workflow and then we can see what the remaining leftover difficult packages are and do something about them. And that might involve, you know, having our own git history for those that doesn't include the upstream git history or something like that. That's those are all possibilities, but that's way in the future. What we need is to have like a normal workflow for most packages right away. Uh upper right corner get dgit.debian.org. Uh thank you for uh yeah that's my question. >> Yes. Uh >> this yo that n equals debian 13.7. >> Thank you. And now I got what the first D is for. Thank you. >> This has probably all been asked and answered but I would like to have that in the recording and in in a compressed form. Can I change back and forth between the old-fashioned way and tag to upload in a single package? >> Yes. >> Sorry. Um, does the web hook hurt when I do the traditional way? And is there a technical reason for team members to agree on going tech to upload? >> Right. So you can so just as with DG you can have a team where whose members some of whose members use DG push source um and some of whose members do it the old way um and the same is true with upload. You can do all three things one after the other interchangeably with when you do that with DGET. There's an option you have to pass to DGET that that says like the maintainer history doesn't want to be kind of contain some of the extra stuff. Um with tag to upload because the the flow here is kind of one way through the tag to upload service. the the canonical version of the package doesn't touch your local laptop. It just ends up on the Ditgget server. So that option isn't needed. So you can just switch backwards and forwards. Um you don't need to agree with your team members on this question. It's, you know, it's significantly better. So, I'm not sure why people would would would object, but if people do object, you don't have to. You can adopt this and they don't have to. That's fine. Um, so there's no technical reason why team members have to agree other than like where's my like I've got have your co-maintainers blessing, right? This is this is because we're in beta and a lot of what I'm doing here, we're doing here is quite it's quite new. It's quite exciting and I don't like my tooling to make a mess that somebody gets upset about. So So that that suggestion there that you should get your co-maintainers buy in is just in case something goes wrong. I don't want to get the blame. [laughter] Right? I want your co-maintainers to be okay. If you're going to try out this beta thing, you should probably talk to your co-maintainers. Can I try out this beta thing with my package um with our package? So, when it's in when it's in, you know, out of beta, there would be no reason to obtain your co-maintainers like approval. you probably want to talk to them about it anyway because that would just be a sensible thing to do, but you can go ahead. >> Thank you very much. >> Um, >> okay. >> It's a quick question. Uh, does does Salsa have to be the upstream repo that goes because uh or can you use something else or DG being the conical thing makes sense to me, but I'm not quite sure why salsa is in there. Right. So, um, yes, like if I can answer a different version of the same question, we could support like I've got salsa in this picture. That salsa there on that slide does not have to be salsa. In principle, there could be multiple things. What they have to do is they have to be able to do the kind of web hook thing. probably there'd have to be some the web hook wouldn't be in quite the same JSON format and we'd have to do a little bit of compat code. Um, so but that's totally doable. There is one requirement though, right? This intermediate system is doing a very small amount of security work for us. It's defending the service and the downstream from random spammy denial of service nightmares. So, GitHub is right out. Gitlab.com is right out. Um, the new distributed Forge stuff, I don't know anything about how that does, you know, abuse handling. um if it has a good answer to that then if we had a Debian instance of that but yes there's an access control list on the tag to upload service you can in practice right now that access control list says any Debian host so if you wanted to you could curl from like master Debian or something um but also it wants to know the forge name and to like it's got a list of configured forges and what their URLs sort of vaguely look like um and whether they're like how to decode their JSON. Uh and currently that list only has salsa in it. Are delayed uploads possible using tag to upload and is it possible to cancel them after the push? This is my first first two questions and the second is uh >> this uh Dit repository is it just a copy of what it is in salsa or is there are some other information in it okay so that's two questions I want to answer them separately so what was the first one uh delayed uploads no uh and that's going to be very difficult right um there's a longer story there, but I don't think delayed uploads make a great deal of like logical sense in many ways. Um the process is quite ungitlike with delayed uploads like kind of running dcut is a bit like rewinding the branch and you really want public branches to be fast forward and also the delayed uploads sit in some kind of weird limbo where nobody can see them. It's all very strange. Uh it might be possible to do something a bit like delayed uploads but no right now that's not supported. This is more of a this is less of a problem than you might think because it's difficult to do an NMU with tag to upload because with tag to upload you have to push the tag somewhere and so if you're doing an NMU if it's a package like that you're half adopting and kind of becoming the maintainer by doing endless NMUs fine right you configure that in your salsa fork and that's great in that case you probably don't need to use delayed but if It's just some random RC bug fix or something as part of a bug squashing party cloning. You you have to clone the the the fork the repo on salsa, clone your thing, set up the web hook, do the git dev push. This is all much more work than just using dit push source which also supports delayed. Uh your second question is what's on the datit server that isn't on salsa? And the answer is so in Debian, some of you probably won't know this because you you people work in their little info bubble and they have a workflow, but Debian has lots of different workflows. And one of the things that the canonical I say canonical here the there's a branch on the dict git server that is the complete patches applied view as if you got it out of apt source like it's treeame to the result of apt source and most maintainer git most people are using gbppq um and that is not true same to the package source So yes, that server has got additional commits in it. Um it's in fact every upload gets two tags with tag to upload. There's the tag that you make on your maintainer view that's maybe patches unapplied. Might even be the bare Debian version with like where you just commit the Debian directory. I still don't know why people do that, but fine. Okay, you do you. Um and what ends up then is the there's there's a second tag uh which if you upload a DIT is made locally where DGIT converts the source to the cononical form and sends it to the Dit getit server with tag to upload the tag to upload service converts your maintainer view into this canonical form automatically. Um and there's an option that you the first time you do an upload it has to you have to tell it like what is my workflow anyway so that it knows how to do that conversion and then on the subsequent uploads you don't need to pass that option anymore because it can look into the git history and find the previous git tag and that tells it what your workflow is. We have the last question for we are out of time. >> Sure. Thank you uh you and John for all their work. I think it makes a big progress for Davian. I really like it. But um unfortunately I have to disagree with your statement. Most upstream are using git. I take the bet that 50% of the packages in Davian are not using git by upstream. So do you can provide some solution easy solution not a tricky solution for those where is no git upstream. Uh I think do did you get that by mining Debian metadata that 50% figure? >> I got this by all my experiences I have more than 1,000 packages and about 25% are using G. It's just my personal view but I think it's >> okay. So because Debian practice is often not to use the upstream git where it is available. Um I have packages where Davian is upstream because uh upstream went away >> then we have a git repository actually >> right just um so I think I don't know what this is by proportion of source code or by number of packages um I'm active in rust upstream community and almost know I have never wanted to ma change a upstream rust package and found that the upstream was not using git. That has not happened to me ever. And I've modified dozens and dozens of upstream Rust packages. I assume that the entire Rust ecosystem is like this. And the Rust packages in Debian, there are thousands and thousands of packages. I'm sure the same must be true of like all the other language specific package managers. It's possible that there are significant numbers of like very weird packages in particular areas. I know Andreas is very active in Debian science. Maybe scientists don't use git. That's possible. But like nowadays, normal people use Git. That's the way it's done. >> Okay. Uh and you're not normal. [laughter] >> Okay. I am very glad that we had a