Reviving (un)schroot?

We have Helmut member of technical committee and who does a lot of things in Debian especially in the crossb building stuff and uh now he will talk about unshrew um a spiritual success for something related to shout. [laughter] So, Elwood, [applause] >> hello. Thanks for joining uh this little exploration that was almost not finished in time for Devcon. If you want to follow the slides and flip back and forth, there's a link. I hope you have copied it by now because I'm now jumping into the talk. Context. Uh the context is a bit asold and a bit trude. They interact with one another. uh as build builds packages in a shoot you know and shoot is the thing that manages it and there are two key aspects for this talk it's set your route and in practical configurations I've I've actually encountered a developer where that's not the case but in practical configurations being able to build packages with that backend amounts to your user being effectively root on your system and that's like passwordless pseudo. I tend to dislike that and that's what got me looking. Eventually, Aspel moved to unshare as a vague default. Unshare has lots of advantages. You not get rid of that root thingy. The setup cost is less. It can isolate the network. You can build build download demion packages with network and then perform the build in a way that doesn't and you also get rid of leftover shoot sessions processes whatever it has lots of advantages and still I consider it's maybe not the way where we want to be so why not have sh the only way to use on share at present is to store your sh as tables but some People like having them as directory change roots with overlay FS. Some would like to build on squash FS or whatever. And that flexibility is not right there yet. It's also not as easy to get a shell inside. So may you may be used to running run session with the old way and that kind of no longer works. Um it requires that your temp directory is world writable. So if you're using lip pump temper, it magically blows up. Uh there is no way of doing ID map bind mount. So getting your compiler cache from the home into your build is difficult. And one thing I'm particularly concerned about myself is that as now has become on container runtime. It's on the same level as Podman, Docker, and SystemD and Spawn and all the others. But I think Aspold should not be a container runtime but use a container runtime and that's where we are looking into. Um, oh I mentioned them already and I tried to actually plug them into Aspel and I was not that successful. We have auto package test as a generic way of interfacing with other container runtimes, but plugging them into Aspel turns out to not be as easy and you get a really non-minimal build environment that way, which is kind of sad if you are also concerned with reproducible builds. Docker and Podman uh are a bit difficult because from from a Debian point of view, I want to have my trust route in the Debian archive. when they encourage you to place their trust route and image fetching to their image services and making that work with no I want to use Debian images is not entirely trivial. Um, a number of others such as Incas and Lexe tend to require privileges on your system as SJH root did or does and all of them make it difficult to have different sets of namespaces because when we download build dependencies we want to have network access and when we actually build the package we tend to want have no network access. So that's a different name spaces and the onshare back end actually does that. It sets up different namespaces for those tasks. So it's not as easy to just plug in any of these existing ones. I tried I discussed that on the Debian devil mailing list and Simon had ideas about we should be using one of them. I was not successful and eventually I gave in and well tried something new. So that was where this onshoot idea was born which tries to plug unshare and shrew somehow together. Um the shrew name in there is implementing the API that SJR has to the point where it quarks enough like a dock that build is happy with it. And I have now it says two proof of concept. That slide was written before I came to Devcon, right? Uh and then Yon and Christian came around and helped me. And now it's a bit different. Uh my first implementation was to literally take what the SBU unshare back end does and mimic it as close as possible behind that API. And that means uh the state of a of a session is kept in the file system entirely. It's really the same thing as as build on share does. You extract a tar somewhere and then for each command that you invoke as a run session a new set of name spaces is created. It's the same mode of operations. But what we can also do now is we can have an overlay FS file system created as in in the upper layer and the temp layer. Uh and then for each command mount an overlay FS file system. So at that point we already have overlay FSB space directory change roots plugged into sbuild via unshare and no root. So that actually works. Uh I have source code and what we have achieved at this point for me very important is separating the container runtime from spelt. Those are now two distinct utilities talking to one another via a relatively defined API called SH root. It's not the same implementation. It's a different one but it the API is being reused and in having that API we already have an easy way to interact with the change route as that's what SH root provides. Great. Uh I got one patch into build that enables us to do the network isolation. Sth root didn't do that because it didn't use name spaces. But now Espel knows if uh my implementation is magically called unshrewd I may pass isolate network and then it does things great. So then I that was last year around and then I sat down a while longer and this is like finished a week ago. um I wanted to lift this on another level where the name spaces are kept for the entirety of the session. So now in the second approach the user namespace and the mount namespace for background on that I have another talk at Hamburg last year this year it's recorded uh these two name spaces are persisted for the entirety of the session and that allows also using fuse uh fuse requires a background process as the file system driver and that background process can be kept for the entire ity of the session that enables lots of funky things. We still do pit name space network namespace optionally in a per command way. So these sbuild on share features are retained but now we have this session not be in the file system but it now is in a process in name spaces and you can actually perform your build on temperfest and when your m machine crashes there is no disk whatever files left behind because it's only in memory you can use it doing that required interprocess communication because now I have this session process and sh needs to talk there and I eventually settled on viral link to do that. Has anyone heard about that? Ah some is um probably too big to fail because systemd started using it. Great. Um so what we get is uh I mentioned rest what I also as an example what is possible with this now is you may have a squash fs image who knows squash fs ah great read on file system and you can mount that using fuse put an overlays on top and then built in there so you only get to decompress those parts of your image that are actually being used Uh yeah, and uh being able to do fuse also means you can use binds. It's a fuse driver and that gets you uh ID mapping for free. There's your cache. So we actually gained features in doing this. Um yeah, some work I had to do about uh about this is uh making Linux namespacey stuff available to some programming language in a way that I can use it plumbing and that's what the Linux namespaces Python module became. It's something I've not seen anywhere else. It's an abstraction of the system calls at a slightly higher level. Uh that is effectively is something to build container runtimes from it. And the other part is async valink. It's a python module doing async io with valink and file descriptor passing and typing and fancy stuff. Uh competing with the reference implementation which does none of that except for valink. And now I was still going for a third idea. We also wanted to have the containment not just be containers but QMO. And in order to do that we need to somehow depart from this traditional model. You have the host system and the chinchroot and somehow you need to think about this containment because now you build can end up running in a different process. But then when I thought about this and discussed this with decon participants and so on, um we actually ended up just wrapping the shroot command in SSH and then it would just work and that enables us to actually build packages using backend ST root in a QOU instance via SSH running some other SH root implementation such as on share in there. That's cool. And eventually we want to further minimize the build environment and getting apt out. I think that was part of the previous talk. Uh um we were floating an idea. It has no proof of concept yet. But adding a fancy no change route option to shoot which would basically say well you do all this thing but you don't change root. And in the old world that would be stupid because it would do nothing. But now we're in this other implementation world. You'd be in those namespaces. You'd be in that QMU and be able to access the location. So there that might be an option to get that third world in without talking too much about it. It's one option we need to extend to the API and it might work. So this all has been um a bit of my pet work doing this exploration but it wouldn't have been possible with all all of these. Thank you very much for having me supported in my crazy questions and whatnot. And now I'm a bit taken a bit longer than I wanted and I'm open for questions. Thank you for listening. [applause] This sounds great. Um, I hope lots of people start using it. I'm a very very happy user of trading stuff. Um, I don't know if I want to adopt your thing. Do I want to like take your thing and start adding all my like user affordances for manual use or is that not a thing that you're interested in doing is like having accidentally replaced SGroot or are you just trying to be alongside it? >> I think it's a secondary implementation that eventually it's not apt installable at this point. It's proof of concept. It's had lots of paper cuts and so on. But I think it's a se secondary implementation and in your sbuild RC you'd be saying I want to use a different executable and you could run them alongside. Does that answer? >> Thank you. >> More questions. >> I made this short so you can have questions. You can may ask ask about how it works, what what you'd like to do with it. whether we can have features. >> So um thanks for providing the proven concept. Then I actually tried it after we we were talking in Hamburg but um then >> it actually it worked but it fell over over the details. >> Um it turns out that I sometimes do need um networking in inside the container and still separated from the host. Um, are you planning on adding something like separate network stuff in there? >> Uh, I'm not sure I have very concrete plans, but >> we may look into that and figure something out. >> I have Zcode that works. Maybe we can just rewrite it in Python. >> Yeah, I'm also not sure whether this needs to be in Python. Python was convenient to prototype it and it demonstrates how this can work. Um, given all the messy file descriptor thingy, having a language that knows borrow checking would be maybe useful. Um, yeah, and I'm I'm open to that. Not sure how that would technically work. Um, what was I going to say? Uh, yes. So, um, you've kind of extended the S2 command line API. Yeah. >> Um, >> is there any chance of getting that extension or something vaguely similar or that would meet the same needs into SG root itself? Is that a thing you've looked at yet? Um in talking to people we figured that you have the information output which gives you properties of SJH roots and it would be possible to like declare capabilities there like this SH root is able to isolate network and if it is you could pass that flag and at that point you wouldn't match on the command name anymore. and maybe instead do it on whatever is printed on as capability and then yes ST root could reasonably do that and it could as it's running secure root it could actually unshare a network name space per command. Technically, that's feasible to get that piece back in. The note change root pod doesn't make sense for it at all because it has no containment. >> Microphone. >> Yeah. Okay. Uh so at some point you mentioned auto package test uh as something you tried to tie as build into auto package test like the auto test back end of with the virtualization and containerization that okay and that didn't work because resource >> it doesn't work that >> okay my bit of question is did you also try to did you think about doing this thing about unshare um and stuff as a auto package test back end as well. I don't think it makes that much sense because uh auto package test wants you to have processes persist from one invocation to the next whereas in this way of working with SH root at where you command exits all your processes are gone and each invocation gets a separate pit name space. So that's a bit of a different use case. I've also looked at using the Linux namespaces module to implement an auto package test back end. Okay. >> And there are interesting ways to do that because you can actually run a full system container fully unprivileged which is difficult with Potman uh at this point. You can run containers that are doing application but running systemd as an unprod container is a bit challenging still and I managed to do that in another use of Linux namespaces python module. It's more work. I think we are last question. So uh I might have missed something but assuming I would like to try your solution. How to get to this basically what do I need to download? How do I need to configure as build to run unsure? Yeah. So this is currently tracked as an example of the Linux namespaces Python module. It lives on my personal git server. So effectively you it's you get to install a Python module and then run the script and you the entry point is setting the SH root command in your Sbu configuration and from there the rest works. I think we're running out of time and thank you >> [applause] >> So thank you Helmut for the talk and yeah all your work other work in Debian. So [applause]