Flock 2026 Artifact Signing In Fedora

Everybody, welcome. My name is Jeremy Klein. I work at Microsoft on the Linux community engineering team. And today we're going to talk about artifact signing in Fedora. >> [clears throat] >> Um So, here's what we're going to talk about. I really my my hope for this talk is that people will leave with a better understanding of how we sign things in Fedora. Um so, please ask questions as we go along. Don't wait till the end. I'm happy to have a conversation and explain things better because I want to make sure people understand how this works. Um but we do have a lot of ground to cover, so um I'll I'll try and focus more on what's new than how it used to work. But we will go through how it used to work or it works right now. >> [clears throat] >> Um so, first worth covering briefly is why why do we sign things? Um We uh We sign things because it is in theory uh proof that we made a thing um and that's interesting to people because we have a history of making things that are reliable and people trust us. And so, signatures are a good way for them to extend to the things we make and then they can make decisions about, you know, whether they run things or not um based on those signatures. Um So, we have been signing things for a long time. We have a system now that does stuff. Uh why why mess with it at all? Um Well, it was initially written in 2007. Um and it has been adjusted since then. Um but it is quite uh quite old and it has served us well, but security has kind of moved on and it hasn't necessarily been upgraded and improved with those things. Um, not to say it's insecure, but there's lots of cool new features in operating systems to make sure that it could be more secure. Um, it was also written in a time where the contributor workflow was very different. Um, the the initial idea was that you as a contributor would send a request for an RPM to be signed. Um, and these days it is all done automatically. Um, usually, in theory. Um, so it is it is designed to not really trust the client. Um, and that leads to some interesting properties of how the system is set up. And we'll go through the the current system called Sigul and I'll point out where that is. Um, it also relies on the client sending the full content to the server. So, the server has to be a kind of aware of different types of content. It needs to know about or, you know, RPM and and all those things. Um, and that leads to problems as well. It's dependencies are all very dead and um, it it needs a lot of work to keep running. And you know, it it does not work on Rail 10, for example. And last, it's you know, very slow. Um, so we're hoping to fix most or all of those things. Um, and I I touched on it briefly in the last slide, but we we have a lot more things that we want to sign now. We used to sign RPMs and, you know, maybe some ISOs. Uh, and now we want to sign RPMs and UEFI applications for secure boot and containers and ISOs and, you know, everything we ship. And and that's not an exhaustive list there, and I'm sure people will come up with new things that uh they would like to sign. Um and when you want to add that new thing, you have to go teach the client that, and you have to teach the server that, and it's a bunch of plumbing, and um it's tricky. Okay, so hopefully I've convinced you that we should sign things, and that we need to update how we currently sign things. Um let's talk a little bit about the the nitty-gritty details of s- secret key storage. Um so digital signatures use public key cryptography, and that means you have a key pair, you have a secret part, and you have a public part. Um the public part you want everybody to know, and the secret part you don't want anybody to know. Um but you also need the secret key to produce signatures, so you need some access to the key, but you want to limit that as much as possible. Um so, you know, the most secret key you could have would be the one that you throw away or put in the ocean and surrounded by trained sharks or whatever, but that's not very easy to then sign an RPM with. So, there's a balance. Uh I saw this on Mastodon a couple months ago. Uh if you're familiar with the uh alignment chart, um here's an alignment chart for uh hardware security modules. Uh so hardware security modules is are are things that, in theory, use hardware to protect keys from extraction, um and allow you to still use them. So, you know, you might be able to say your TPM is a hardware security module. Um or, you know, a scary box that sits in your infrastructure and nobody touches. Um so, there's lots of grades of of uh secure, I suppose, for for secret keys. So, Fedora's current balance is somewhere on that alignment chart between scary box that nobody wants to touch and, you know, something that you rack but does use some hardware security to protect the keys. Um, they're stored on a dedicated host that is isolated from the network. Um, it encrypts the keys with a combination of user-provided secrets, and in the case In this case, the user is an automated service that calls into it, and then a hardware token, like a YubiKey, to um to encrypt the secrets. So, you need both. Um, and it's, you know, restricted to only a few users who are admins, and they do all the maintenance on it, I think, via the like management console on the out-of-band uh network. Um, and then on that same server, we have secure boot keys, and those are in a I think it's a PCIe card that is an HMA or an HSM uh hardware security module. Um, and we sign certain things with those keys. Just like grub and um, you know, the kernel and stuff like that. Um, and yeah, so the it's an isolated host and and um, that that is kind of where we settled on that that alignment chart. Um, okay, so, the current system that's running today, um, is several components. There's the the signing server bridge client set, um, and that's called Sigil. Um, and then there's a a uh an AMQP consumer that calls into the signing client called Robo Signatory. Um so, we'll go through them. I'll try to be somewhat brief, but I I think it's important to look at how they are designed and set up um before we look at the new thing. Okay, so I don't expect you to understand this cuz I don't understand this, but I did draw it. Um This is kind of the diagram of the systems. Um you know, if you were in RPM, you would start up there at the uh the AMQP consumer um and it would call into the Sigil CLI, which then talks to this bridge um and the bridge is the only host that the server's allowed to talk to and it's the only way that clients are able to talk to the server. So, it talks to the bridge, the bridge talks to the server. It proxies um requests along. Um and the client has to authenticate with the bridge with MTLS. Um and then the server uh ultimately just shells out to PE sign or RPM sign or whatever, uh GPG. Um So, that is the high-level. We'll run through each little bit and then we'll go on to the new stuff. So, Robo Signatory is where, you know, your journey as an RPM starts. Um the way it works is it listens to tag events that Koji sends out. Um and these tag events are generated by Bodhi as it manipulates it will move a package into a tag when it wants it to be signed. And this happens when you submit an update. Um Robo Signatory sees that. It sees that the tag name matches an expected uh regular expression and that it was done by the Bodhi user. Um it then queries Koji to say what RPMs are in that build. It gets a big list of RPM IDs. It then sends that list of RPM IDs to the bridge. Um waits for the bridge to come back and or you know, the client to say, "Hey, I'm done." And then it tells Koji, "Please write out a signed copy of that RPM." Um and here is the first hint of of um that design choice where you don't trust the client. Um because the client does not send the RPM. It just says to the bridge, "Hey, there's a build in Koji. I want you to sign it." And that's it. Okay, so the Sigul client, it's a little Python CLI. It connects to the bridge. Um the bridge forwards its request to the server. Um it does this interesting um nested TLS session where it connects to the bridge with MTLS, and then it starts another TLS session in there and exchanges secret keys um with the server that the bridge can't see. Um and those keys are used for HMAC. And then the rest of the communication happens from the bridge's point of view in plain text. So, the bridge does an inspection of the requests, and that's how it figures out, "Oh, I need to go find those RPMs from Koji." The bridge downloads the RPMs, sends them to the server, uploads the signature back to Koji, and then tells the client, "Hey, we're done." Um but it only does this for RPMs because when it was originally written, we only really cared about signing RPMs. So, things like um the kernel are sent the the bridge can see it, but it's HMAC signed, and then it is just sent on to the server, and the server uh signs it and sends it all the way back to the client. So, there's some inconsistency there. So, the bridge, as I mentioned, it um it proxies the traffic. It also does some input on behalf of the client. Um it does some pushing of signatures, but not all for some types of content. Um and as I mentioned, it's it's the first component that was originally planned to be run in Fedora infrastructure. So, this is where, you know, you trust what the bridge does to some degree. Um and that's how you kind of filter things to before they get signed. Um okay. And then the Sigul server is the last bit. Um you know, it also connects to the bridge uh via MTLS. Um it exchanges its HMAC keys with the client. I talked about that already. Um and then it needs to be taught about every type of new content that you want to sign as if your signature is not a generic thing like um GPG to sign blah. Um so, the the real problem with this is that now the version of the tool that you're using to sign matters a lot and what version you have on the signing server. So, our signing server runs on Rail 9, which means we have RPM version I want to say it's 4.16, but um you know, it's quite old. Um and if you want to use an interesting new feature from RPM V6, you might not be able to cuz the signing server is not going to know what that is. Um it certainly won't be able to produce V6 signatures. Um and V6 introduces multiple signatures on a package. It allows for uh post-quantum cryptographic signatures. Um and a lot of other things like that. Um, an RPM sign is definitely the most egregious, I guess, of the we wish we could have more features from our signing tools, but there are other things like if you want to sign a container with cosign, um, you might need to, you know, get that in real nine and, you know, there's ways around that one, but it's it's not a great example, but anyway. So, the server tries to isolate each request by spawning a new process for every request. It does them serially, so it will do one RPM at a time, um, and it is, as a result, kind of slow. So, I've kind of covered these architectural problems as we've gone along, but the design for the untrusted clients doesn't apply anymore. We we have changed how we run things. Um, sending the artifact to the server is expensive for lots of reasons. You have to make sure your tools can handle them. You're sending big files, um, across the network, um, over two TLS sessions, um, and you have to do a lot of teaching of every component along the way about the version, uh, or you know, of the the content that you're signing. So, that's unfortunate. And the implementation problems also I've talked about a bit. There's no concurrency. Uh, Python NSS, which is what is used for some of the, uh, networking and crypto bits uh, is like really dead, um, and hasn't been updated in years and years and years. Um, there's a lot of cool sandboxing features we could use to better protect, uh, requests from each other. Um, and there was also a lot of work that needed to be done on the database side to support things like post-quantum hybrid key pairs, um, and those involve two key pairs, as you would guess. One's a traditional, um, like, you know, RSA key and then a new, uh, post-quantum algorithm key and you kind of do a signature with both. But, it's presented as kind of one key in GPG. It's a It's a bit odd. Um, anyway, it also didn't really have support for HSMs. I added that, um, in a very hacky way to support signing uh, kernels on other architectures besides x86. Um, but it really is not It's not easy to administer or, um, use long-term. So, it works, it's not nice. Okay, so, I went off into the desert and wrote a whole new signing server, um, but as you'll find out, it looks very much like the old thing. So, I did not come up with a lot of these ideas. I mostly just did a rewrite, um, and was inspired by a couple different things. Okay, another confusing diagram. Um, >> This one's bigger. >> This one is bigger. I made it more detailed, um, and so, it's the same set of components, basically. You'll see that little box, um, I think says AMQP consumer. I can barely make it out here. Um, and then the the, uh, client box is quite big, um, and it has a bunch of different tools you can use. And that client then talks to a bridge that talks to a server. Um, so, architecturally very similar to what we have. Um, the big difference here, um, and I think I we can stop looking at the scary picture. Um, if you want to see, it's on the docs online, so um, gaze away. But, uh, yeah, the design changes were Well, first of all, I didn't change much about the general shape. The database is fairly similar, although I had migrations for uh, post-quantum keys. Um, the big change is that the client interface to sign things is exclusively through um, and we'll talk a bit more about it later. Um, and I added better support for hardware security modules and the administration of those on the server. So, you can plug in a a real HSM for some value of real, you know, and add it to the signing server and it can sign with keys in there. Um, or you can sign with software keys, which are encrypted in a similar manner to the the uh, keys we currently have. Um, and it can also import all our current keys, so there's not any sort of complicated migration story. It's just you can import your our current keys, which are generated through Fedora 46 and keep on signing. Okay, so I talked about this a little bit. The client, you can only really The only way I will support talking to the the server is via this PKCS 11 module. Um, the client exposes its interface the the PKCS 11 module talks via a Unix socket. Um, and I'll talk a bit more about why later. Um, it communicates over that with just JSON-L. Um, it uses systemd sockets to activate um, and protect the the um, client credentials to authenticate. Those are all protected with systemd creds, so you can bind them to, you you the host secret, the TPM, some set of PCR registers. Um it's kind of up to you how you want to bind those. Um but that's how we protect those authentic authentication secrets and the secret to unlock keys. Um we just rely on systemd. Um it supports auto unlocking keys in some use cases, which is mostly for signing the kernel. Um and I'll go through that later. But um and it does the same communication style with the bridge with mTLS and um a complicated nested TLS session. Um Unlike with Sigil, the bridge is a pure proxy. It doesn't really do anything other than enforce mTLS. Um it does a small amount of protocol negotiation where the client says what version it's speaking and the server says what version it's speaking and then you can do you could do some fancy matching, but right now it just says, "Oh, if you don't speak the same version, you're not going to talk to each other." Um and then it just proxies traffic. Um and then the server is broken into two parts. The first part is the network service. Um this is what connects to the bridge. Um it performs mTLS. It has a pretty minimal API, which are basically list the keys you know about, give me details about those keys, um and then please sign things. Um the network service does not ever decrypt any of the signing keys. They're never touched by that bit. The database is made available read-only. Um and it signs everything with a systemd activated um signing helper. So, the signing helper um is spawned by systemd Each request Each um Each connection gets its own socket-activated unit. So, they're isolated with systemd sandboxes, and we opt into all the ones you can. Um it doesn't have access to the network when it decrypts the key. It is very restricted on what file systems it can read. Um it pretty much can just see any HSMs it needs. It can see the database, and that's it. Um so, it This is where we decrypt the keys into memory if they're software keys or talk to the HSM if it's not. Um you send a digest in, you get a signature out, and that's kind of it. So, we've done a lot of isolation improvements there, mostly just relying on the fact that systemd has a lot of this available. You can drop a couple directives in a unit and get a lot of cgroup and, you know, just general isolation features. Um we use systemd creds for a lot of secret um encryption and binding to TPMs. Um And I think critically, I didn't do any of that. Adam, yes. Yes, the kitten is outside of the box. Adam noted there's an isolation failure. Um Although, there was a a larger sandbox in which uh he was not available to escape. Um so, he did not make it out of the house. Yes. So, so yeah. I didn't write any sort of isolation stuff. We're just using standard features from systemd. Um it's pretty much just a systemd unit configuration. All right. So, I talked about how it all works. That's not necessarily super interesting to people. Um What do we get out of all this? Okay. So, I mentioned PKCS#11. Um, you might have heard this referred to as crypto key in some places. I think mostly just I've seen that in like Sequoia documentation GPG maybe. Um, it's an Oasis standard um, and it is just a shared object that implements a CAPI um, and things like OpenSSL know how to talk that API. Um, and so you can um, also just not implement most of it is what I discovered. And I discovered this by looking through Flatcar Linux's toolkit. Uh, this is what they do as well and I thought, "Hm, that's that's actually a pretty nice little trick." Um, so I absolutely stole this from them. Um, and I think they stole it from somebody else. So, we're all just sharing. It's great. Um, but yeah, you don't have to implement most of the functions. It has a long list of things for encryption and key wrapping and all these other features that we don't care about. We just want to sign stuff. So, you implement two functions and then some initialization functions. Um, and then anything that knows how to talk to PKCS11 or OpenSSL can sign with this interface. Which means you can use all the normal tools you use to sign anything with the new signing server. Uh, you don't have to implement any sort of knowledge about the type of thing you're signing. You can say RPM sign, please sign with this um, key that's in a PKCS11 module and it figures it out. Um, yeah, so the pros. It is widely supported. Um, there's some weirdness. I I will say that that not everything works perfectly with every tool because it is a big API and there are some vagaries around how to implement it and not everybody implements all the functions. Guilty. Um but it does mean that adding new types of signatures on new types of content is basically free. If upstream builds a tool to sign to format and sign some new type of content, we can just use that. Um and this will run from a Fedora host, so we don't need to worry about it being in Rel. We can use the latest and greatest of everything. Uh it also abstracts the server from everything else. So, we can have a you know, we have our AMQP client that knows how to talk to Koji and all these things, and it just knows how to talk to tools that know PKCS 11. You could get rid of sigul tree and all the other things, and it you replace it with a real expensive HSM, and everything would still work. Um I'm not saying we do that, um but it does mean that if Fedora wants to migrate to something, it's cheap, um because we're just implementing a standard interface. All right. This is where it's not going to work. But we could try some live demos, um assigning stuff over conference Wi-Fi. Um Maybe. Probably not, but maybe. Um All right, that text is probably too small. >> Sorry, can you have a white background? >> Uh see if I can find my mouse. Uh Yeah. Is that visible? >> It's good. >> Good. Okay. So, um before I I tested this before we started, uh I just listed um the available tokens. I won't run that again, but yeah, I just used P11 kit list modules. It will give you a set of standard modules and then the the one we have that's exposing um signing keys as uh um tokens. So, you can see here sigil tree. You can see uh where the path is to the module um and then a couple different keys. Okay. I have a little script over here. I'm going to You're You're not going to have to watch me type these really long commands, hopefully. Um All right. Bear with me. Okay. So, this is an OpenSSL command to sign the workstation live ISO uh for Fedora 44. Um this is a couple gigabytes. If we wanted to sign it with our old system, we would send that ISO to the signing server uh wait for it to come back with a signature. Um so, sending two gigs over conference Wi-Fi is not really doable, but here because we have PKCS11, we can just digest it here and send the digest over and have it signed because we are in the middle. And now it might hang. Only took 3 seconds. All right. Um so, >> [applause] >> in in theory, it signed something. Um okay. So, we probably want to, you know, validate that we did in fact sign something. Um so, we're going to need the public key. Oop. Not that way. Okay. So, we can tell P11Kit, which knows how to talk to PKCS11, give me the public key for that token that I just signed with. And that might work. Okay, there we go. Public key. Um Let me just We'll dump that to a file cuz we're going to need that. >> [snorts] >> Trying to find my mouse. Okay. So, um And similarly, you can export X.509 certificates that way. Uh you can't export GPG keys cuz GPG doesn't do this, but um you could you know, for X.509 paths, you can just use those normal tools to to pull all the information you need out. Um but we can also verify. Maybe. Okay. So, we did in fact sign it with that key, uh assuming I didn't make fake tools, but you know, you'll have to just trust me there. You can try this out. Um So, that's OpenSSL and all those paths and any tool that uses OpenSSL, you can make work with this. Um So, that is a good half of the type of things we want to sign with. Um but we also do a lot of GPG signing. Um and you can make this work with GPG. And one day, hopefully, with Sequoia, but Sequoia has not released support for CryptoKey yet, so I've tested it with their branch, but they've not tagged anything, so I'm not going to demo that. But, um we can tell GPG to go look at its card. I've already pre-configured it to talk using the PKCS11 module. There's a um um a a demon you have to run that knows how to talk GPG and it translates stuff and there's a lot of weird things about it that don't work very well, but um it's not their fault. It's GPG. Okay, but we can GPG sign that same ISO, hopefully. Um and I've configured GPG to use that public key already. That's boring to set up because it's hard to get the key out um right now. Ah, [sighs] so this is the thing it does. All right, that's not going to work probably. Uh I'm not going to try and live debug this, but in theory that should work. Um and it takes a lot of futzing with because it is very particular about setting up the environment just so and there can be only one token available because it doesn't have any way to know about multiple tokens and it's very much glued on top, but Sequoia has better built-in support for this. And hopefully we'll move to that. So, I'm going to skip over all that. Um although that's going to break all the RPM sign stuff. So, um we could maybe just watch the video. We made it through the first half of the demo without uh disaster. Um Yeah, okay. This is definitely not going to work. Okay, um I guess we will go back to the presentation cuz I I'm not going to try and figure that out now. Yeah, but you know, it could have been cool. It could have been perfect. >> Which means that's perfect for me because >> Yeah, yep. >> So, in your demo when you export this >> In the demo where you exported the public key >> Yep. >> if you change type to private, what happens? >> Oh, it doesn't. Um the only way you can get the public key out um oh, you mean with the OpenSSL command? Or the uh with the GPG key? >> I think it was a sig utility command. Uh The like one above. Yes, this one. >> I can save private. I mean, I know I didn't implement this, so I have no idea. I think it's going to crash with some sort of error. Unsupported. Okay, yeah, that's the error. So, you can just say I don't support exporting that. There's no API to get the private key out of the server. The client has no way of getting it. Um and I guess I wired up the right error to say good luck with that. So, yeah. Thank you for the question. Cool. So, um you can sign all sorts of stuff with this, anything that uses OpenSSL or GPG. Um RPM sign is really difficult because you have to use GPG with GNU PG PKCS11 SCD, and it has a lot of special requirements for configuration, which I clearly broke. Um with Sequoia, it's more straightforward. Um you can do this with OS tree GPG sign with the same GPG configuration. CoSign, I think, supports PKCS11 as an optional build thing, or you can ask it to export a JSON file that you use OpenSSL with. Um you can There's a tool to sign uh PCR register values, uh systemd measure. I tried that out just because I wanted to make sure it worked with stuff. You can sign stuff with that. Um it works with PE sign, SB sign. Um you can also use it for SSH authentication if you want. And you know, we can go demo that. And that might work. We'll see. All right. So, I'm going to get rid of all my keys. No keys. And then I'm going to need my password. If you want to see my password, that's my password. Don't you know, don't get to these keys, please. Um Come on. Typing live is There we go. Okay. So, I'm going to tell it to look to use my module to load SSH keys. It's going to want my password. I put in my secret password. Okay. So, all those SSH or all those keys I I listed at the beginning are now available with SSH. Um and when I set up all the uh infra for this demo, I left SSH on for the signing server so we can go log in to that with the SSH key that it has um to authenticate with it. Um >> You can ignore that error cuz usually I use a YubiKey and so it's looking for that library. But yeah, so we just logged into the signing server with an SSH key that's stored on the signing server that we all got via PKCS11 and all that nonsense going through the bridge. So you can use this with most any tool. Okay, so we should be able to sign all sorts of new things without doing any work on the signing side now, which is exciting. So if you want things to sign, we can do it. Um because it exposes a client via Unix socket, you can put that Unix socket inside of a build environment. I didn't come up with this idea, I just stole it from PE sign because that's how it works. Um you can configure the client to unlock keys um on the you know, outside that build environment so the build environment doesn't need any secrets. It can just as long as it can access that socket, it can sign things. Um and you can use it to replace PE sign demon or the weird shim that I wrote that we're currently using. Um and just use the normal tool to sign stuff. Okay, so the last bit of this was that AMQP client. Um I rewrote that as well because it needed to know to shell out to all those different tools for the content types. This is the only place we need to teach it about new types of things. Um so this is where we make that logic for what Koji tags to sign with which keys, um what OS tree commits to sign, things like that. Um it has nice things like metrics. It runs everything concurrently. Um so, you can sign 1,000 RPMs at once if you want. Um we can see what's currently happening. I'm signing all the Fedora stuff. Um it's probably not as busy right now cuz we're all here. Um well, a lot of us are here. Um Yeah, so so a little bit ago, somebody built a bunch of RPMs. We signed about 200 at the same time and uh um so, signing should hopefully in most cases be faster now. Um well, there's there's a um there's a cons slide, so brace yourself. Um so, you know, in theory, it should be wonderful and super fast and everything should be signed like that. Um but there's some problems. So, you know, nothing's perfect. Um there are some downsides to PKCS11. Beyond that, it's just another layer of complexity that you have to think about. Not that it's super complicated, but um it has a an API that works for a lot of different cases, but it um has some some issues like it's synchronous and you ask for a signature and then you get a response and then you ask for another signature and you get a response and so on and so forth. Um which is normally not a problem, except that we enabled IMA signing in RPM. And IMA signing is where we sign the digest of every file in an RPM. Um and so, if you have an RPM that has 10,000 files, you need 10,001 signatures. You sign it 10,000 times for each file, and then you sign it one time for the GPG header. Um and so, those happen serially. Um and you're going over the network, and so it's slow. You're dealing with latency. Every single hop to and from the server adds up. So, um yeah, the the main problem that makes uh signing not as fast as I want it is IMA. Um you might spend 75 ms to sign the RPM for GPG, but then you might sit there for the next 20 minutes signing every single debug info file in there or whatever. Um and that's pretty sad. However, we can sign 1,000 RPMs at the same time. So, you will not have to be in a queue, but you will, if you have an RPM that has a lot of files, wait 20, 30 minutes. I haven't seen anything above 40 minutes, and I'm running it from my desktop in my room to a data center that's, I don't know where. So, somewhere somewhat close, but it's going over the internet, so it's bad, but it might not be too terrible in the same data center. Um so that's that's the big downside to this. Uh and I'm sorry, signing will not be instant while we have IMA, um and I don't really see us getting rid of IMA, but I'm not, you know, that's that's up for other people to debate. Um I'll just say that it is very expensive. Um the other thing I ran into, and this isn't really a huge issue, I worked around it after I wrote these slides, which is that um I found some interesting behavior in systemd-creds, where um if you use a socket-activated unit that spawns a new instance for every connection, um it will rate limit you when you try to decrypt secrets using um the um its own socket API to decrypt secrets. It will do I think 16 at the same time and it leads to some weird behavior. You can manually raise those and you can also not do uh accept equals yes in certain cases. So, this isn't really a problem for us anymore. I I dealt with it, but it is an oddity if you want to run it as isolated as you possibly can when when you isolate clients from each other. Um so, with that, that's kind of the end of the presentation. Um I hope people feel comfortable contributing. I you know, I've mostly written this myself and other eyes and opinions would be most welcome. I hope that signing is not a scary thing to people. There are some a few little scary bits of code, but you mostly don't have to touch those. Um and the rest of it is just standard networking application things. So, um if you want to get involved, I'd greatly appreciate it and you're more than welcome. Um questions. >> Thank you for the talk. I came just for the second half because I had a different talk, but uh from this this part I saw, I've got one question for the uh sequential signing, did you did you look into the async signatures uh in PKCS11 3.2? >> I have not yet. I saw a few notes about that and I haven't looked really hard, but that is I I think I only implemented to 3.1 cuz maybe 3.2 wasn't out yet. So, that would be very interesting um and I will look at that. >> Okay, so that was the first question and the second one was uh what will need to be done in the Sigil Drive to support post quantum crypto signatures. Is it just updating the sequoia or it will will there be needed some other changes around the infrastructure? >> Uh so, yeah. Um to support support uh post quantum keys, you will need to have sequoia. Um so, as long as they make a release and their their support for that is also kind of intertwined with their crypto key support, so that should all land at once. Um and then the second thing we'll need to do is add the key generation to the server side. I already did all the database plumbing, so we really just need to add the um the bits where it generates those. Because at the time I wrote it, um the open SSL binding the rust open SSL bindings did not have support for those keys. But I think it does now, so um I'll have to go double-check. I saw that they had a release that had maybe some support. So, that that's the only other thing. Um and then it all should just work. Um yeah. Anybody else? >> I I have a silly question. But the But the um one of the things that uh HSM is used for is is for um a primary source of entropy. >> Mhm. >> And have you thought of the original TPM, right? Matt Dopsch, the first thing he did with it was implement a random number generator. >> Mhm. >> Have you done uh the work that might support uh adding that as a as as a feature? >> No. I mostly I only implemented the signing calls um and for PKCS 11 and I I wouldn't want the server to be used as a source of entropy. The server generates the keys, it uses the standard, you know, get random calls and um yeah, it's it wouldn't expose that kind of functionality. It really just signs stuff. Um yeah. So. >> So, for the uh over the network communication, um do you use um some some custom API or do you use something more standard like uh KMIP or something like that? >> Um, I implemented it as just JSON and it is very much inspired by Sigil mostly because I didn't want to get too far off from what they had. Um, but it is versioned and so we can bump the version and change to a more standard um you know, serial format. That's and I'm totally up for that. Um, it was mostly about what kind of dependencies do we want to uh adopt and rely on and um yeah, so right now it's just JSON over a socket with a little bit of framing uh added for seasoning. >> I have a second question. Um >> Yep. >> uh for um did you face any challenges cuz like um you have um PKCS11 CA API that is thought for like using it offline? >> Mhm. >> Did you face any other challenges apart from IMA signing um with with using an an offline API for over the network communication? >> Um, the most tools seem to be fine with it just taking forever. Um, and I in the back end you can I just retry forever um because for most cases if it's like, you know, if it's a part of a kernel build it's better to retry until the server comes back than to report an error and say, "Hey, try again later." cuz you just waited 4 hours for a kernel build or whatever. Um, so everything seems weirdly fine with that which I was surprised by but, you know, um and then other than that the only thing that I ran into that was tricky was that various tools needed various things exposed and as I was trying to implement as little as possible um I was a little sad that to make uh GNU PG PKCS11 SED work. Exposing the public key wasn't enough. It really wanted the certificate, even though it just extracted the public key from the certificate. Um so, I was a little sad about that, but um you know. It made me do a little bit more work, and that's the only other thing I think of that I ran into that was um it didn't just work when I tried it. Yeah. Anybody else? No? >> So, um we have one topic that we are currently working on is signing containers for the uh all the some of the atomic variants in Fedora. And then one thing that it's very experimental at the moment is UKI >> Mhm. >> signing UKIs. And uh one I wasn't sure I understood properly. So, you mentioned that you can use PE sign through a unique socket. So, would that mean that we could have some of your stuff running as a demon on the on the builder, and then we could just pass in the socket through through the container build? >> Yeah. So, um the same way that we used to run the PE sign demon uh on the host and expose it as a unique socket to the builder, this is the same way. The The way it would work is you would run that client on the host, expose the socket into the build environment, and in the build environment you install the PKCS11 module. And then you can I I can't remember if you have to run a command for NSS to discover it, um and then PE sign will pick it up from there. Um or if it just works out of the box, I I would have to go double-check. But, for things like SB sign, you can just give it the URI, and it works. >> All right. Sounds promising. We'll We'll play with it. >> Yeah. Yeah. >> Anyone else? Any more questions? No? Let's thank our speaker. >> [applause] >> Thank you very much.