Flock 2026 Machine Readable Package Lifecycle Information In Repository Metadata

So my name is Fred Live. I'm a principal engineer at AWS working on the Amazon Linux product and today I'm going to be talking about machine readable package life cycle data. So the problem we're trying to solve here is how do we make users aware of like the package life cycle? How do we make them aware of changes to the support status of software that they are that they are running? And this is very much a communication issue, right? If we establish that like not all humans read documentation and not all humans are able to remember documentation especially when they are running no that has a multiple lives of uh of support multiple years of support as opposed to like a few months like the favorite federal version. Uh how do we make sure that they can make security like safe decisions throughout the lifetime of the OS and that they know what they are getting into. And when you look at the when you look at the list of installed packages on the distribution or when you have a Linux box and you look at the packages that you have installed, there's very very little difference between a package that is enough live that is not getting support anymore from a package that is just not getting updates because it's so stable and so mature right there's what information do we have available right to make kind of secure decisions in those context right we have is the package in my repository and do I have security advisories or bug fixes advisories for this package. But in the absence of data, right, I cannot infer the state of a package unless I go and refer back to my release notes. So this is the problem that we are trying to solve here. And so before I go further because this was made for Amazon Linux and I know that like from talking to folks here this weekend uh not everybody is aware of what Amazon Linux is and how it differs from other distros. I want to make sure that we have we share the same context on like why this came to life and why some of the design decisions uh were made in this project. So Amazon Linux is all the AWS like Linux distribution that we use to run our own services. It's also widely used by a lot of customers running on EC2 or using other uh AWS services. Uh we offer a stable base that's supported for five years. Uh I say stable, I avoid like LTS or I avoid talking about EBIS because we do update packages. We update package versions but some packages we will just not touch for five years. We backward patches and we make those decision based on on each each package. uh we offer multiple kernels and that's where we're starting to get into territory where we need to talk about life cycle. So we ship multiple kernels each of those kernels sped for four years. So if you're really good at math you're already seeing that small kernels have a lifetime that is less than the operating system. Um so customers who start running Linux today or we start with a new version they are going to have to update kernel at some point or kernels are name spaced. Um, so they need to they won't get the update to like just a DNF update. They have to specifically go from one version of the package to another version. And we also do the same mechanis mechanism for multiple like many runtimes language runtimes on ML. We tend to offer multiple versions of the same runtimes. We have customers who want to run uh PHP got to help them. uh they um they want to run the latest versions but they don't want to take a breaking change when they go from one version to the next and so they want to control the velocity at which they are changing from one version to the next and on my part I don't want to backport patches to five different versions of PHP so the deal we're making here is I'm going to give you multiple versions of PHP but I want you to get off of the old versions before they are in supported by the upstream motors and well Amal Linux is is validated if you care about those things. So what it is that we want to know uh uh about the state of our packages as we as we run or as we operate all the distributions. Uh I want I'd like to know if my packages my packages currently supported, right? Do all packages I have installed on my box or do they are they all currently supported? Do they all have the same level of support? Right? Are they receiving are they receiving all fixes or are they only receiving a subset of fixes? Uh and will when will the sport status change like next right can I expect can I expect to keep running them for the next two years? they have a six month six month runtime on them and so on and from the perspective of the distro right we we we so for Linux and because we offer multiple version of the same packages we wanted this information to be like specific to each package and the way that we apply it to the dist at large is that the support status of the distro is the sum of the support status of all of its packages right when our distro goes end of life right it has it means that from our point of view every package just reach its end of life and nothing is supported anymore. So we don't differentiate between the sport status of the package and the sport status of the distro. It's a bit of an attempt to make this thing generic and um I want to know do all packages have a kind of consistent support life cycle uh in my distribution and this on some distros it's kind of a given on Amazon unit it's not uh we may also be running multiple repositories and different repositories could have also different level of support um into them built into And so why is it important to have this information like readily available? It's a compliance problem, right? So for customers who run distributions that are supported for a long time, right? It's important to know like it's important to be able to know are my packages supported, but also how can I prove it, right? How can I how can I demonstrate like programmatically that my that the packages are supported? How can a security scanner run and confirm to me that all of those packages are fully supported and that I can meet my compliance obligations and that I can make my security vendors and my auditors and my security people super happy about the way I operate my systems. Um I may want to end like packages support for packages before the end of support of the dro like we talked about the kernel a few moments ago. Um like we as a current as we like I want the to have the confidence to be able to release a package and know that I can let customers know that this package support is going to end at the point in time that is ahead of the DRO and they will not be able to find this information without going through like release notes and release note for point releases like what we have been doing previously we've put out release note and say so this package will become unsupported or will be deprecated two releases from now and now like you have You have to keep that in mind, right? And security vendors and scanner vendors have to build this into their product if they wish us to. So, it's really hard to have kind of a consistency in the experience and all of trying to kind of defeat the fact that the system is built on like humans being able to like kind of remember this information through their day-to-day cognitive load. Um, and things happen, right, as well. We may want to push out the end of support of our product. I may have released this Python version on that date thinking it's going to be supported until this other imaginary date. I may want I have customers using it. I don't want to break them. I may decide I will keep supporting them. So I need a way to communicate that uh to them and I need a way to communicate in a way that they can it can be verified. um security event happen right we've saw a lot of like we if people remember like lo for a few years ago if I'm I have a list running for many years like we support Linux for five years four years in right we may have a really hard time supporting some of those packages right the upstream maintainers might have moved on my packages might have been abandoned like years ago we've been backporting patches they are breaking changes protocol changes right So, how do we get customers off without just telling them about it? And prior to having those systems, we'll put out uh put out blog posts, put out announcement, put out like release notes. We never know, we can never approve, right? Our customers reading this, our customers changing their behavior based on this information that we are broadcasting to them. And I like to ask like product managers, can you prove to me that anyone has read this announcement we've made? And I've been charging them for that for years and nobody has ever been able to get back to me with an answer. So our solution to this is the the sport info.xml file. Uh it's a machine readable and version file format that we have added to the mat Linux repository. Um this was added a couple of months ago. Prior to that we experimented with this system like over many years. We have had a few iteration of a similar system. Um it offers it offers like package specific and repository specific uh support information. It it define like support levels for each of the packages and it's specific to each repositories as well. We use it to generate human available information questions. >> Y XML >> that's it. >> Is that mean? I'm sorry. >> No, it's okay. I'm I'm not a fan. I'm not a huge fan of of XML. We want the package manager to use this. And so we know that the package manager already has to understand XML. We have all the libraries all the libraries installed. It's a it's a very expressive format. It's very verbose. It can it's it's hard for a human to like write and read, but it's very easy to like uh for a machine to like write it, generate it and validate that what we have generated like matches the expectation. So easy to verify and it's it it's really good at kind of expressing complex information. >> I would have loved to use the ML for this. It's just like it's it's a very large file and we want like we need a way to make sure that whatever we generated like matches the schema and XML is very well suited for this. And I like to know that like DNF supports XML natively. So I know I have the libraries available everywhere I want to use this. Um yeah so we use it to generate the human readable information. So we know that we have consistent data across the online documentation and the the machine readable information. And we've tried to make this extensible to other distro. So we try to keep it as generic uh generic as possible if anyone like is inclined to try something like this. And we've also tried to make it extensible to multiple dimensions of support level. So we've been like for the last few minutes we've been talking about like the support level from the perspective of security right how do we are we patching are we patching all CVS are we patching only important are we patching only critical and many distri do that for like extended support and so on but they are as distenders like we make other decisions based on packages right and sometime we assume that customers kind of can read our mind and know like what we have how we think through packages and some packages in the distros Those are are made available strictly for our own use, right? We we carry those packages because we need them to build the environment. We we need them to we need some libraries to operate the package manager for example. But do we do we support every arbitrary use cases of those packages and as those packages ages right sometime we make secret decisions like four years down the road maybe we're going to evaluate this library and we're going to look so how it is used by the OS versus how it's used by the or it could be used by arbitrary customers so how will we communicate to those users that they might not want to be using this package or that we are we are like doing different type of evaluation different type of security decision for this package. >> Is that in a graphical user interface or is it just in XML and people have to know when things are going in, you know, going to go out of the support and stuff. >> Uh on the next slide or two slides from now, we're going to talk about the DNF plug-in. So there's a CLI, no graphical interface, but there's a graphical interface for geeks that's uh that's available and also it's XML. We can build other tools. uh we display this information since we generate the human vable information like online we have the you can on the Amazon website you can find information about each of the packages and all of the life cycle that is made from this uh this data but we really made this with scanner vendors in mind and security vendors in mind like our end goal would be like if you're running Nessus or if you're running other security scanners can you get notified by your tool and so it's not so much for human consumption Mhm. >> It lives in Amazon repositories. You can find it. Sorry, can't see my screen anymore. You you can find it on the on our CDN that we use for our our containers. This is available to anywhere anywhere in the world. So, this is not restricted to the AWS infrastructure. U you might notice it sits a little bit outside of the repository. It's not in the repo data itself. Uh but because Amazon Linux has a kind of a different repo structure than other industrials and because we do determistic updates, we carry some repo metadata outside of the repository itself. any one could make the decision to just put it into report. >> Sorry. How does the information get populated? Like >> Oh, thank you. Thank you. >> Thank you. >> How how does the the the the the deprecation data get populated? Where do you get that data from? Is it do humans put it in? Do you scrape it from a location? So the >> I skipped ahead here. I will have to go back. But uh we we pre-generate like we generate the original data at the time when we launch when we launch the DRO. And so our beta assumption is that we support every package we put in the dro is going to be supported until the end of life of the DRO. And then there are exceptions or the way that we Linux is a commercial product, right? We have we have teams and people working on it. those teams own packages like and that model is somewhat similar to to Federa Sigs in a way and we have each package has a has a very specific owner and every owner is responsible for the life cycle data for the package and because we integrate this into our release tooling the package doesn't get released if it doesn't have life cycle information uh attached to it so it's kind of a requirement >> thank So the way that we think about the life cycle of a package right they can have it's not just binary it's not just supported and unsupported there can be different level of support they can be unlimited level of support right you can have excellent support very excited support very very excited support and so on uh and most of those stage and you know the the most basic way to use this is to we map those stages to severity of CVS or severity of bug fix pixels that will be that will be addressed in that specific stage. And the way we structure it that we basically the file carries a list of packages. Each packages is attached to a life cycle and each life cycle has a list of phases um that are pretty fine. Um we we have the concept of a package origin because in the case of Amazon Linux we can ship multi the same package in different repositories and it can have a different life cycle in different repositories. So we have our ESPAL repositories which are very similar to EPAL and we have customers sometime will ask us to pro to promote a package from ESPAL to the core repos where it gets a higher level of support and so if we do that we need to be able to assess on the instance do we have the original version from ESPAL that's installed that has a limited support and maybe unsupported at this point or do we have the version from core is still supported and so we identify the origin of a package based on the like the the repo ID and the send the true identifier signing key and vendor and we do kind of a greedy match on those to determine what we have on the instance. You can find the code on uh on GitHub. Uh this is very fresh. This was pushed this morning. Um it's this reference Python Python library Python module that reads the file expose an API to uh to make run queries uh against the data. Uh it's also what we use to validate the file against the XML schema. Uh it's used by our DNF support and full plugin and we support the legacy and the V1.0 format and you might be wondering was published this morning why do you have already a legacy format? We've been experimenting with this for years. Uh so when we did release the L203 in 2023 uh it came with a different sport info file that was sitting on the instance and not in the repositories. uh it has a kind of a binary state of supported unsupported uh information is a little bit less rich and so what we published uh this week is a is is uh kind of the evolution of this system and based on the learning that we've made in in using this we've also been using this for years to um operate internal infrastructure at Amazon um we use this internally to be able to to determine to scan that every instances are no instances are running package that we have flagged as end of for end of support. Um, we also have made available a plugin for DNF4. It's the sport info plugin. Um, so it adds a sport info node to the DNF command line allowing you to like query data for a specific package for a list of packages or just to filter packages based on their their sport the sport state or the end of support dates. Um, it's also available on GitHub. This one is in the development tree. So the version we run this isn't released in our repositories yet. Uh we still use the legacy an older version of the NF plugin in our in Amazon 2023 and some sample output of what it looks like. If you query for a single package um like you get the status of the the package. This one is currently supported or was supported when I run this command. Uh you get the end of support date of this one as the lifetime of the this tool. till 2028 and we're currently fixing everything for this. So there's still our sending works to to keep this going. Uh we're hoping to kind of compress the size of the the format the size of the format a little bit right now because we carry an entry for every single RPM in the DRO. It can get can get pretty large. Uh so we're considering maybe making kind of default life cycle per package origin or supporting wild cards uh for the matching packages. Um we want to enable system operators to block installation of un unsupported packages in the package manager. I would very much like to like for all users to be able to define policies where on my fleet I don't want anyone to be able to install a package that goes support within six months. And we should look at the other five sport as well. So this is about it. Any any question? Is that interesting? Oh yes, question. I love questions. >> Uh is this plugin and tooling to produce this XML file available in Fedora main? >> No. And when will be? >> Oh, >> it can be. I I >> can I ask please? >> It's not It's not something I have considered, but during your question, it's probably something I should have considered. Yeah, we look we look into this. I guess similarly are there asks from Fedora that that from parts of this that you feel would make more sense in Fedora? I think uh I think in in with RA we're looking at something similar. I don't know all the details but I think being being uh discussed. So it' be interesting to know which parts make sense in Fedora from your experience. I think the the life cycle of Federa being so much shorter, right? I think this is this is less of a problem uh the way that Federra runs it infrastructure as well. The way that like repos get moved after a while like you you don't like a user who have been would have been running like Federra 8, right? Would have had to change its repo URL between then and now in order to keep using them. While in our case we we want to let our customers keep running old ancient version of Linux if they choose to do so because they control the six environment of because they have access and this is safe for them to do it. Um so we have a bit of a kind of different different constraint. Um I could I think I I will see the value and like the the end of support of Federa is almost deterministic right but it's very much relative to the next version the next two version Russian that get released so I think I will see some value there if we we could like programmatic programmatically flag that like this thing is done and you need to move on from it. Um, so yeah, I think we I will look into I will look into what it takes to to bring it into into >> Federa. Yeah. Uh how do you deal with the unresponsive maintainers? like can you just then switch the uns to unsupported and because if there is something critical on the package and you want it to be fixed like for example you propose the pull request but they don't respond they don't review they don't merge it can you do something about it with this XML >> you should present because this is a much better answer than the one I gave to the previous question like yes I think that's that would be a good way to use is into Feder from my point of view for Amazon Linux like I'm always on the hook for every packages even if like nobody is maintaining them on stream anymore from the point of view of Federal I think that would be an interesting interesting use case >> I'm just repeating this on Mike that uh I mean maybe asking a question have you tried to propose any of these kind of things against the DNF upstream team >> not yet. >> Comment in that same direction is is like if we were going to get it into onto the CentOS where that might might be more helpful. It would be necessary for us to have it in Fedora first. >> Okay. If anyone like to follow up on those questions, if anyone here is involved in like DNF, lib DNF like lib repo and so on as an interested to this, I will be happy to talk to you. Okay. Well, thank you everyone. I appreciate you coming over here.