Flock 2025 Specfile Library + NoRPM

Okay. So, hello everyone. Welcome to the second day of vlog. Uh, my name is Nicolola. This is Pavl. >> Hey. >> And we will have actually two talks in one. So, let's start with mine. And, uh, yeah, we have we don't have a lot of time. So I will just >> yeah kind of lightning talking today. So enjoy. >> Okay. So this is about spec file library. I don't know if you heard about it if not it is a Python library that's uh uh Python library for paring and modifying spec files. The emphasis is on modification because that's why we designed it uh in Uh also it means that it's not very fast because like function of over speed basically. Uh one of the main goals is that the changes that you make should result in a minimal diff. So if you edit someone else's spec file, uh the changes should be minimal. Uh it's packaged for Federa and the PAL and you it is used by petit uh and also for example cop and some other projects. Uh so from now on I will I will be showing showing code samples. So [cough and clears throat] uh this one should be pretty self-explanatory I hope. Uh let me just talk about the bottom segment. uh if you maybe you can see from the from the code itself uh it should bump the release in the spec file and add a change lock entry. So basically what RPM def bump spec does uh so the main tags are exposed as properties. So in this case we are using expanded release which is the value of the release tech with all macros expanded. Uh this is a pretty naive example because it expects the result to be a number but uh it's just for demonstration purposes. So the value is incremented and converted to string and returned back to or assigned back to release. Uh and then there is a method for adding a change entry. Uh in this case we are specifying only the text of the entry but you can also specify all other uh all other data like uh author or date etc. Uh okay let's move on. Okay. So, uh in this case, uh we want to work with with sections. Uh this example should uh basically convert from change log to auto change log. So uh there is the sections method that returns a context manager and in the target variable uh it behaves as a list of sections and within the context of the context manager you can modify it you can work with it as as with list and the changes will be propagated back once the context is exited. Uh you can also access the sections by name. So in this case change lock and the section again behaves as a list of lines. So here we just uh copy the entire change log and replace it with just one line uh that is how to change log and then we are just dumping the uh original content to the change lock file. So another example in the previous one we were accessing the section by name now we are iterating over all sections but uh this example should print just or it should print all request tax in a spec file. So we are interesting only interested only in packet sections uh skipping all else and then we use text method that returns a list of text or again a con context manager that behaves as a list of texts and we just filter the request once and paint the all the details. Okay, so that was pretty much it. There is or of course a lot of a lot more but I want to show you something more interesting maybe. So if you have a spec file like this and you want to update the version and if the version is uh defined using macros then if you if you just would as if you would just assign the version to the version property it would end up like this. This would be the div and that's probably not what you want or not what the maintainers want. So you can do this instead. There is this update tech method that tries to if possible it tries to actually replace the values of the macro definitions instead. Uh the algorithm behind it is not very complex. It's actually quite simple. it just uh uh splits the string uh on some element and then tries to match the individual parts to the micro definitions. So, it doesn't work in all cases, but I would say it uh well, I actually measured it, but uh I'm not sure about the exact number, but I think it was about um o over 90% of spec files that work. And uh it is possible to do this. Uh if it if it wasn't possible, then the version would just be replaced as before. And this is my last last slide. So I want to talk about macros and conditions. So spec file uses RPM for uh expanding macros and eval evaluating expressions in conditions. Uh you can also define your own macros by uh yeah by passing them to the constructor and you can expand any macro or expression uh using the expand method. And as [clears throat] for conditions, so in this case you have two epoch tags. You would probably if you would be modifying the spec file, you would probably want to change maybe all of them or both of them. But you are if you are interesting only interested in the value of the one that would be the correct one let's say. So that would be valid in uh in case username is defined then there is a valid property of the tech object and that is true only in the two branch of the condition or if there are nested conditions it would be true only for the text that are in two branches of all of them. Yeah that's pretty much it. So I'm not sure if you have any questions. We have probably a couple of minutes or I can pass it to Pavl and then we can answer questions at the end. And there's there are some links. So I encourage you to check this because like I said this was just a quick overview what the library can do and yes. I guess you should say it on the microphone, but I'm not sure. >> Thanks. Um, I just wanted to ask, is there anything that the spec file library can't do right now? Like specifically like tags it can't edit or limitations that you would like to highlight? >> Uh, not really. The only well not everything is like high level. So sometimes you have to do more work than you than it should be necessary probably. But the only downside I'm aware of is if there are spec files where there are conditions that uh span across or like for example there is else in the middle of a section >> or yeah it because the par is not counting with it and uh there is open issue for that. Thanks. Um hello uh Pavl again. Uh I'm engineer and RPM uh build system developer at Redhead. And in this kind of lightning talk, I'd like to explain you what are my what are my headaches with build systems and designing them and especially source RPM or RPM built systems. Uh a little bit of motivation anyone did before switch to root download some weird uh script from the internet and pipe it to bash to install some important software. Well, I did before only if I really have to install something necessary for my day-to-day job and the people decide not to package these things into Federa, which really makes me like angry. But anyway, anyone uh generated a spec uh source RPM file from a spec file before using either RPM build or fat package source RPM or title source RPM you should be you should be aware that these things are similarly dangerous except that you are not rude but but still um but let me start with probably known things. Uh can anyone tell me what's wrong with this uh with this spec file? What can happen when I try to build it? >> Yeah, exactly. Yeah, there is there is this little typo the dollar sign instead of a person sign meaning that I'm I'm referring to a non-existing a non-existing environment v. So I'm basically trying to remove vir which probably I cannot because I'm not a root but uh if I was I mean how how do you protect against these things you probably don't run as root the builds but you shouldn't do this under your normal normal user like that one that you are using for firefoxing and stuff like that and visiting your internet banking please don't uh and you probably use mock Maybe maybe you don't uh mock tries to I I mean I want to underline that on a best effort basis isolate you from this kind of shooting yourself into food. So here comes source RPMs. Uh can anyone tell me what can happen here when when you try to when you try to build a source RPM from this spec file. I'm not I'm not going to blame you now. You cannot know that. You don't know that because you cannot uh because you don't know what's inside that shell script over there. Uh actually we need to evaluate that one to get the kar uh macro content to get the version to actually place the version in the in the uh s RPM file name. uh hence uh I mean even generating source RPM is not really safe always depending what's inside that script uh and you shouldn't probably like blindly trust the spec files out there how do you protect against these failures mock source RPM doesn't help here of course you need to get the source RPM first so yeah fortunately there's a mock d-spec option which is unfortunately not well known uh but you are here so you're lucky you now know and uh yeah I wanted to point out that mock retries on the best effort to isolate you some some options are not yet well done some are buggy for example this example option uh is still using RPM from host meaning that it it suffers from these issues but let me stop here I am not here to talk about bugs in software that I maintain so quick uh quick mention of Lua. Lua is touring complete as well as the shell script. So it can cause the similar damage to your system if you build untrusted spec files. Um so how this all relates to build system headaches that I have. Uh we basically want to let all of our users build whatever they prefer. Uh they they builds are done on virtual machines. So if they break it, they have the VM broken. We don't care. We just recyclate the the machine and start a new one instead. But we want to allow them to build from GitHub or federalist git and basically cach c the sources. Uh I'll tell you why later. And then on each of the VMs uh build one truth according to users wishes. Why we c it? Uh the reason is that we don't want to redownload the sources all the time from GitHub. We don't want to do distributive attacks to Pegure.io and stuff like that. And how do we cache? Uh uh we basically run another disgracer. Uh and uh we transform the the sources from GitHub using source RPM files to this this proxy this Git. And did I say that I use s RPM files for this? So uh yeah that's not really safe and we need to be careful there while generating those RPM files. And these things are not only about security. Uh some people do weird things like they prefer using different tarbles when building on Fedora and different tarbles elsewhere. I don't know why but uh yeah these kind of things are not yet well solved in in corporate uh at this point in time. If you are curious take a look at this issue. So I was always asking myself whether we can like replace this uh s RPM file which is distribution and arc uh uh specific with a tarable that is arc agnostic and distribution as well. And most of the time we could uh but um [clears throat] yeah we have we have the sources file for example in in this git so which deterministically identifies all the sources that are or at least can be used at RPM build time. So we could basically do git archive from federadis git and append I mean create a master tarable of this archive plus all those sources uh and use that for caching but federa is not making our life easier in this case. Uh we invented a way to not provide change locks at all in Fedora. Uh yeah, this RPM autospec thing um basically means that the git archive is no longer enough at least unless we want to lose the auto I mean the generated change log and the generated uh release field. Hence we need to uh I mean first thing we need to uh do full clones when generating uh generating those RPMs which is not always uh cheap operation and we need to parse the spec file to get those things like version and release to actually have the right values for generating the change log and for that we need to call RPM parser and that's not always safe Right. So another headache is this kind of exclusive arch. Uh I mean build systems don't want to waste potentially expensive S390 VMs just to realize that they cannot be built because there is this exclusive arc thing. So we ideally want to parse the spec file in advance to realize that it doesn't make sense to uh evaluate or to to start the S390 VM. Yeah. So, how do you protect yourself or how should we all protect ourselves? We we should really like be careful. Uh the same way as we don't do or at least don't prefer doing curl and pipe to bash, we shouldn't blindly build uh binary RPMs and not even source RPMs. Uh and we should never build RPMs as root or like some kind of privilege to user user ids. Uh so for real what can we do about this in the future as as the ecosystem? Um I talked about this with the clever and brave people maintaining RPM within redhead and this issue was opened by Panumatan as a result of the discussion. So we could basically have a new new lowprivileged mode in RPM not doing these kind of dangerous things like shell and Lua maybe not doing includes um and that would basically that mode should basically ignore unknown spec file texts those that are too new for that particular version of RPM. right now RPM if it doesn't know the particles tech it bombs and fails and it doesn't parse the spec file at all. Another option maybe uh here comes a little announcement of my Christmas 2024 uh project called I mean work in progress no RPM. It's an complete RPM microprocessor at least at this point in time. It's written in pure Python. So if you if you wanted to parse spec files on nonRPM distros, you have no RPM, you can use basically this one. It could be installed from Pippi if anyone packaged for it. Uh and yeah, here comes a little example how that can work. Uh you basically optionally can load all the macro definitions on your system using the system macro registry and using that database expand a string that should have a spec file loaded. uh I mean that's a spec file as a string and then you can query the database or the registry of of macros this way. Uh to sum it up uh if can if I can have a wish uh please don't uh don't make our headaches worse and keep your spec file as spec spec files as simple as as possible. ideally at least the preamble section that defines the name, version, release, sources, patches and stuff like that because this kind of really matters. Uh for example, that's one example of many uh this kind of short commit uh references. These are using bash for no really good reason. You can use equivalent directly with some macro macro syntax. And if you could do this switch, please do because that really like that really like matters. If we do this kind of switch uh to uh to the new mode of RPM or to no RPM in build systems, uh the variant below will be parsible the the above no. Okay, that's about it. Any questions? So, if I use the no RPM library, uh how big of a chance is it will work for my spec file? It depends what what exactly you want to read. Uh I tested it on the version release. Basically those things that we need to make RPM autospec happy to successfully generate the the change logs and releases. And it was like I tested it on all those uh row height spec files. It's like 24 thousands of spec files and only like 300 of them did not gave me the same value for for release and version. So it's pretty pretty good. But if you wanted to parse other things, I mean there is a huge to-do list in the new RPM library that needs to be implemented to provide better results. So what about generate uh dynamic build requires like being able to evaluate build requires and p down a list to to identify what a build environment needs to be created with ahead of time? >> Of course not. Yeah, you probably know uh for dynamic build requires you need to extract the tarable and run some scripts that are fully on control of upstream basically. So we don't want to evaluate ideally anything to get the version of the of the software from spec file. So so no um is the no RPM packaged? >> It is not yet. As I said, it's just the Christmas idea of this year. So, going to going to make it Fedora soon. So, if I want to protect myself the best, uh how should I build uh random unknown uh specs downloaded from the internet? Uh uh should I do it in a container or something? >> Uh submit it to copper. There are VMs that we dispose. If you get it broken, nobody cares. Uh or I mean you should really review the spec file first. Uh in theory, mock protects you, but it's on a best effort basis. So if you use the mock dspec and you give it the spec file, it is most probably okay. But if the person tries really hard to escape from a mock, it would he would be probably successful. So question for both of you. So how is the spec file package like different from the no RPM? Okay, so that's probably on me. Uh well, spec file uses RPM for micro expansion and uh condition evaluation. Uh there is actually open issue for about not doing this and implementing something like no RPM inside the spec file library. But uh it's just an yeah there is no plan yet. It wouldn't be so easy. I mean well I guess it would be as Pavl said maybe the basic stuff could yeah could be possible in a short time but definitely not uh all the complexities that IPM is capable of. The spec file is really widely used nowadays. Like we use that library for every single built-in copper. So it probably parses all everything you need. But it uses RPM in the background. So you should know that if the spec file is uh dangerous then then you face potential problems. Any other question? Okay, so thanks for attention and enjoy the rest