EasyBuild 5.0

good so we're here and I'm going to talk about easy build five um the very observant or who were with us last year will note that this is entirely the same title slide as last year and hopefully I've opened the right slides and um we're going to have this year's talk uh so information uh going to talk about what's been implemented what we aim to implement in the next couple of months uh little bit about policy changes also hoping to provide why we're doing things you know why you know particularly where we're talking about deprecating things for the future or breaking changes we'll tell you why um what we're trying to do and also obviously an important one tell you how you can contribute um including a Hands-On session with some notes you'll see this sort of bit at the bottom right of the screen here it says information there are some other ones that will come up breaking change um there aren't too many of those but that's a change in how easy build works and that will be please do something you know you will to use Easy build 5 you will have to make a change to do something but as I said there aren't too many in that category bigger section is enhancements new features things that will make your life better or make our life better developing um and the bottom one code changes that's where we've uh sorry code cleanup that's where we've basically um improving our life by man get rid of a bunch of code um there though some of those sort of imply breaking changes because we've removed a few things so easy build major versions over the last 12 years from version one onwards um I'm Optimist I'm optimistic and Kenneth is optimistic that that 2024 is correct but you'll see we're not committing to a month yet um and I also have both logos on just because so the First new enhancement feature is Implement implementation of run shell command which will replace both run command and run command QA um this is very much for those who are developing particularly developing either in framework or particularly developing easy blocks run command and run command QA are how you tell easy build to run has to sort of suggest run a command uh run command is go and say make you know go and run make um run command QA is where you where the program that's running you'll get your talet to run a command the program will start putting at output and then asking a user question of what do you want to do at this point um which easy build runs reads that output and then responds to it well we're UNIF the two into one command that does both um one of the major reasons for this is actually the default thing you want to when you would want to use run command at the moment to run a command and fail if it isn't if it if the command fails isn't actually the default method of running command which means a whole bunch of and we've actually already implemented run shell command a whole bunch of our changes simplify to just go run shell command run command and it just you know simplifies a little bit of code makes our life a little bit better um we're able to use more modern python um and that will be linked to a other changes with there and it's also enabling further enhancement the simplifications we've been able to make uh one I'll talk about in a little while is save shell commands in environment that's actually already been implemented tell there but also by doing this we're going to be able to improve the logging and the error reporting and make easier to follow what's going on um so this has been implemented in framework the old functions are now deprecated so run command run command QA they will be there in easybuild 5 but if you use them they will throw out a deprecation warning um we've replaced it in framework we've placed it nearly in the test all of the test Suite uh I think there's still one use in Easy block test site which I have which I noticed I had done there we're nearly completely replaced in Easy blocks there's a linked issue there which I think lists the three or four that is still to go um there're almost everything that's still remaining to be replaced in Easy blocks is either a command QA one um and it's the two command QA ones that have list of questions that are about this long um where I've looked at them I've tried Abus and it didn't work when I tried to it so I need to just go revisit and work that one out and there's a couple more where people have contributed into easy build for uh new easy block or an updated easy block where and I noticed a couple of those are about to come through and need converting um but we're very very nearly there and the reason when we get to the number of PRS which I think Kenneth's already spoiled in his State the union talk the reason why there were so many freezy blocks is I did every single one as a separate PR um there if you have custom easy blocks at your site if you use hooks anything house that document and that document I think we still need to push a change to it for the command QA but it is actually all there the information I can't remember whether we've put it live but it is documented how to do it the other place to look and is look at the easy build p uh the easy blocks PR as that actually shows you how we've changed it in a lot of places um and we'll give you know information but that is that is one of the areas that will impact people going forward particularly if you're contributing to a um Upstream so we have put a lot of documentation into there and we will continue to do so um it will tell you what to do so I mentioned this one of the improvements save shell commands and environment I will say this isn't merged yet it's a PR it's open or I don't think it's been merged unless somebody's done that while I haven't been looking um what this does and it's sort of commonly yeah good reaction when it when commands are running it does two things creates two oh sorry creates two files an m doire a command doire um one is the environment all environment variabl there's a slight caveat on that that it's not quite but it's nearly it doesn't do I think bash functions was the comment um but it will you know file with all the environment variables in it and commanda which if you run that script it puts you into an environment where you can find out what you are running you can and you can just go back in your history and see what commands were run recently you know what command what was the last command which would be the one that failed we would assume also just go back and step through to see what's happened and that will make hopefully a lot of people's life easier you know some way of getting into an environment and going what happened what is it happening um there but as I said we've still got to merge it's not far off it's nearly complete um there and yeah one that we we we just hinted at and we've started the work on is we're trying to standardize parameter and config names um this is where we'll sort of use the word opt for options in one place and options in a separate place you know try and standardize so if you know one of you know one parameter name or config name you're probably going to know what the other one is rather than having to guess where where we might have shortened the name and where we might have um used the large but and this is our plan we're in the this is and exactly how this is going to be done is still a little bit open we're still deciding is we're going to add the new name in new names in some way is a second set of options um and then Implement methods for auto rename of them and then during the lifetime of easy build five we will then switch to use the new names in framework blocks and configs and then deprecate the old names though exactly that you know and that might be initiative process it might be a little bit more of a step we've just got to decide exactly on that but a non-destructive way that you know this is going to be going forward um just make it make everybody's life easier um and also as part of this there will be some policy document which will actually once we've written it that will actually tell you how we've decided what the names will be so if you're implenting a a variable or a parameter or a config or whatever in the future and you want to know what should I name it um we'll hopefully give you enough hints as to what you should do I'm looking the person Jasper who's in the process of implementing that and I think I've just volunteered him to write some documents um which which which would be good because it saves me writing it and I've got plenty other documents to write at the moment um one that is very very dear to me is enhancing the use of download instructions this is something that's actually already an easy build feature it's just if you there are about five easy configs actually currently use it uh in easy build 4 what this is is we all been there there are is a bunch of software that you have to manually download because there is either no direct link or you have to um agree to terms of service or you have to you know log into abacus's website and download a file um but easy build will try and currently at the moment download these sources and then fail with a message that's often a little bit cryptic to understand and tell you what it is partly because the last URL in the list it will try is our backup mirror and it will go well hand me down able to download it from there and every goes well why not it shouldn't it be there um so what we've done is gone through all the easy configs um and added that anything that needs it should now have a download instruction this will be in the easy build five um it also we've improved the we're improving the message so it will tell you that you need to download something and where you then should put it on your system um I.E what is the default place it looks for sources um and this is all about providing a better user experience you know why is this not working well I need to go and manually do something um the main reason this is well there's two reasons why I've implemented done most of this work in terms of easy build five one is because we're not supporting python 2 anymore and get to that in a minute um I'm able to use f strings which actually simplified my life a lot um and the other one is is and this is another feature that or another bit that's coming up because we're archiving older easy configs I could implement this in new easy configs and only the newer ones which simplified my life of actually finding out what the download places were for some of these bits of software um because we've because the other sort of thing we did was if we couldn't find out what the source was nobody else is going to be able to work out what the sources either um and it was in a few places where bits of software I was going I can't the the homepage no longer works the software is either so generic a name that you can't search for it and to find the software and yeah um so I simplified my life a bit there now I've put this as enhancement there's a slight breaking change bit here but hopefully not um we will be enabling our path linking by default there's already quite good documentation about what our halfs is and what the support is there and if you want more i' just say go and read that because it links out to there we are however by default going to rain Ali Library path and again if you want to know what that why that's also covered in that bit of documentation and I'm not going to go through that now but it also and that documentation also tells you if you do want to remove a the library path from your mod generated modules it that uh variable uh sorry parameter is there that allows you to do it and that documentation tells you how to do it um the one caveat and the one thing we're in the process of fixing there's currently if you're te test easy build five and you have a mixed AR Half and nonar Half Stack of built software there's an occasional failure lamps being one of the bits of software where our detection of is this software correctly our PA Mis understands whether the software is correctly araed um and it's very much uh it will work but we just need to work out why the sanity check fails on it or Implement a workaround for why the sanity check fails and if you want to read about that that's the where I say fixed that's actually a PO request and it has some documentation as to what we're doing and what's going on uh definitely in the improved user experience we're going to be turning on Trace output by default um I'll start with the bottom when talking with maintainers nearly every maintainer that's ever answered my question says yes we just use Trace by default um it's easier when you're tra so Trace um gives more output it will tell you a bit more about what you're doing or what easy build sorry is doing it stage and we'll show you some of the commands being run and uh it definitely gives you a better understanding of how easy build operates um and also if you then copy and paste the logs to us and say this is where it failed it will help us understand where you were in the process um particularly where a particular stage inside is a multi-stage process so let say the build step might do five different commands with the extra output there and we Trace enabled we'll know which of the five commands was actually being run at the time rather than just having to guess which one failed um there for python package installation all easy configs and after set two well I can't remember which way around it is one for python package one for python bundle those three have to be set to set at the moment they will be on by default so you will no longer need to set them and actually the easy config CI in Easy bu five will tell you please do not set them um there so you know just simplifying what you have to specify making everybody's in life a little bit simpler um yes is there supportal uh is there support for virtual environments virtual environments I'm in what sense sorry uh installing into virtual environments like currently no no okay um I think we need to can if you'll want the microphones are okay okay um okay so currently you're saying can no there's no support for virtual okay okay but I think maybe you need to explain what the use case is oh um that python p um I mean especially if you you know if you have different tools that perhaps use different python versions or something which can come up a lot and kind of stuff that I install then using python path can cause clashes because python path itself is not a versioned path I wish that it was but it is not uh it is it is meant to be a direct um path to site packages so um if you have different python vers different major minor python versions there will be clashes in what you're loading and using virtual environments and hash bangs that point to a python within the virtual environment that you've created that python will find its dependencies relative to its own location rather than use Python path and things like that so we've stopped using python path completely and instead start now install into python virtual environments and so we can have different tools with different um python versions different major minor versions and there's no clashes between them question in the back so well answer okay so um the issue with setting python pass is real so if you had set this python pass in the modules and then a user for example creates a virtual emphasis model loaded then he of course expected if he installs an updated version um of the package that's already installed that his package will be used but it is not and this is exactly why we introduced a hook um for writing the module file that replaced um each occurrence of python pass in the model file by a corresponding entry of EB python prefixes um and this basically solves this and the change here um basically makes this the fault so that will just work so it isn't installing into virtual NS but it plays much nicer with virtual NS and it's only the prefix there so um the script we installed to make this EB python prefix work um then takes the version of the Python you are actually using into account to add this to the sus pass so yeah all problems solved hopefully yeah and you know and that is yeah that works ongoing but yes we're aware and hopefully we'll make life just repeat what I'm saying for the for the REM people I think the only issue with this is is that everybody knows python but nobody knows so Kenneth's com comment was yes that everybody knows what python PA is but no EB you know very few people even who use easy build probably know what e python prefixes is unless you've looked under the hood of what's Happening um but part of that was probably us then documenting and making it really clear what's going on and why um there um another one enhancement and this is for people uh installing Fortran software what one excited user in the room at least good to know um uh because GCC core is a sub tool chain for uh for not just GCC based tool chains but also for Intel ones and other compilers umod files shouldn't be Fortran mod files that is shouldn't be present in GCC ore installations currently we don't actually check this um but what we're actually going to do is start what uh we are going to put we've bu put sorry we have put in detection um and by default it will warn if it fails if it finds one but actually what we're going to do in the central repository is we when we're using test Bots we will turn it make that into a failure to make sure we don't there and there's been a a few PRS recently where I've either move software out of GCC core or just pointed out to some other people can they just move it because of this um there there is an e easy config parameter that we've added skip mod file sanity check and that's for those F bits of software that you install in GCC core that aren't actually Fortran dood files are just do mod files because they use that extension for something else these two they're both sort of some sort of data file format or instruction type file instruction to that bit of software running um type file um so there is a method of skipping that there's also things such as clang itself and some of the variants which obviously have mod files but they're mod files generated by that particular in um compiler during their installation process so they're fine in there um but those items every anything I've hit so far and found it on I have fixed in easy build five or put the work around in in easy build five which is most things that have it they're you know they think is small handful um python 312 easy build uh ev5 works on python 312 well the test weite definitely runs um I'm not sure whether any of the developers any of us developing and using easy build I have actually tried doing installations yet partly because the type of system I need to be installing on I don't actually have python 312 on to test with which means I might need to just work out how um there is things like we've replaced this utils use and how easy build in stores and some other stuff um with there also as of this past weekend there's actually a PR for python 312 as an easy config um but that really important statement read the notes in it you need to do a reinstall of end cursors because yeah I'm seeing Kenneth's reaction because I'm gonna explain this one because it amused me um in EN curses we don't install the package config files currently um python 312 tries to detect en cursor tries to en cursor's W the wide character support and messes up the detection of En cursors and detects en cursors rather than n cursor W yeah see Ken's reaction um when you install the package config files it does the detection correctly gets ncurses W Linked In correctly and doesn't fail halfway through your install with something with a cryptic message of you just sitting there going well why is that causing a failure I can see the library myself software for you um like I said just just read the notes it tell it that yeah it tells you what to rebuild and it is quite a quick rebuild fortunately um and then that I think is available to work it will probably not get merged because it's adding an extra python version into an existing tool chain generation and probably won't do that but it will give just give us the basis of the next one I'm seeing a nod in the room there there the other so yeah the good reason for now is that we're not going to introduce another p in the same tool chain that's too much of a pain it's not worth the trouble either I also think that there's going to be too many projects that don't actually support 312 yet so it's still yeah from that side probably early on we've actually had a request from a user that's asking for python 3.12 it's like how hard can it be yeah we can give it to you but you're probably run into trouble very very quickly yes I mean stuff on top of it I might I I've had one request as well and I might in install it the slight complication is that's obvious in a 2023b tool chain and I haven't even deed 2023a at my to my side yet so long time but I know at least know how I would do it um there but if I do install it for a user they will literally just get python with nothing else you know it's going to be python plus setup Tools Plus W wheel plus pip I think um change module options I think I'm with this first statement that this well it will be Al mod um I think it's leis syntax only we're going to flip some defaults to switch to using module depends on by default and switch to module extensions on by default uh module depends on by default improves the loading unloading experience and module extensions on by default is a l allows your module file to know that not only does so where we say have scipi bundle um there it allows you to go well cpai bundle actually also has NP and whatever so if you do something mod I think it's when you do modular Veil numpy it'll go ah yeah but that's in cpai um there um I've linked to the alod documentation that's very much um and obviously my slides will be available later online so you know we have follow links um there both of those are in the process of being purged which is generally we're just trying to fix fix the test Suite to deal with a changed option about more outputting module files but nearly there um and here's where we hit the breaking changes but with the comment that we've been warning about this so minimum versions python 36 is now the minimum version as Kenneth showed earlier python 27 usage we're getting reported as under 5% python 3.5 usage was 1% so I'm assuming one response in survey um and by going to pyth well we were probably always going to lose that one but you know it's come Python 2.7 is end of life four years now um and Talking personally as a developer I can develop with python 33 now quite easily if I have to make sure my code works on python 2 it probably takes me 10 times as long as just doing the Python 3 because this is the one place I have to still think about python 2 um I haven't actually written has to support python 2 code other than for easy build in probably three years now I'm think some disgusting some person still has to um yeah it's uh We've managed to convert everything else we were doing locally um we put a lot of effort in because we just hit a point where we could consciously try and St support python 2 and I just know for myself it I cannot do it my head just if I don't know that something is going to fail with python 2 unless I actually run it um and find a system I can test it on um and yeah um Al mod as Kenneth said I think going to jump to 8.0.0 being the lowest supported version and team OD um read the modifier on this it is 4 this is we're still just making decision and talking to developer it's 4.3 but only if it's above four um lower versions are less than four is still there but will be deprecated um it's only you can't use between four and 4.3 um that's what the T mod developer has said is a good idea um because that's still quite an old version um there but as I said not yet merged so if you do are if you are using that to your site and have some feedback for us then please give it to us um if it is really going to be problematic um remove and sort of there anything that you're running in e in easy build 4.9.1 oh sorry going back to there that is giving you that message deprecated functionality will no longer work and then tells you what the old command is what the new command is that code's gone um if you start running easy build 5 code's gone you it will not work um it will just fail that's such as the removed applicated code that's such as the python 2 two support it's all just gone that was quite a pleasing day to me removing I I just piled up these PRS of removing deprecated code and we've got you know and it just makes our life simpler as to Developers of not having to support really old code um in there code that nobody really quite in some places even understands what it's doing anymore um it just you know has to be done at some point for some things uh new deprecations um not actually that many at the moment um environmental modules so that's as what you would set I think easy build to tell it what they are what type of environment modules you are one's called environment modules and the one called environment modules TCL one of those is the really old one and one's the even older they are Eng um which is effectively anything under version four you will get deprecation warnings for those um I haven't noticed anything I didn't I looked for our PRS and I couldn't find anything else we deprecated um I think that's it but as in the lifetime of easy other than yeah sorry and run command and run command QA yes yeah forgotten about adding those because that was um but as we've seen during the lifetime easy build 4 because easy build 4's been around how many years it was Kenneth saying five years yeah on the earlier slide and I've forgotten already um we will add some more de I would assume we'll add more deprecations over the lifetime of easy build 5 and publicize you know of things we might removing in easy build 6 whenever that happens um sometime in the future you know but based on our uh uh release rate you're going to have a couple of years you know probably more Kenneth mentioned this one as well tool chain support Pro policy um this from last year of the sort of motivation using the central repository as a reference and where possible one version of software per tool chain generation as we said as I said last year and repeating this year there are some exceptions but we try to keep them limited so you know your stack will work you know using the reference Easy config the stack should work most software together andless off of in different tool chains um information there as to how to add your own Easy configs um repository on top of that um if you want to do that and I know lots of people do documentation there so based on the feedback actually this was what we came out with as our policy and it is in the document we have there um example as of 2020 as of today 2023b is the latest generation we have released uh we will accept PRS for the most recent six tool chain Generations behind that and at the moment we have no plans of changing from two a year so that's those six he says and he looks very quickly just to check he did update the slide based yes so those six um we will then deprecate the next two so seven and eight tool chains back so at the moment that would be 2020b and 2022a would be deprecated at that point we will start closing PR and issues for software related to th in those generations and archive anything older uh 20 2019b and older in Easy bu 5 that means we have done archived everything from 2019b to chain generation and older and when I say a generation so obviously that's the underlying GCC GCC core equivalent Intel version um and any other tool chains um and going back there there were some other ones and equivalently we've also archived older system tool chain easy configs um I can't remember exactly exactly how we did that but thing of a similar AG TR older um with also those things where nobody could work out where the how to download the source code for was a few things um and also combined into that we also archived out old patch files that were no longer used by any easy config there is then a further thing we've done on top of this there is there will now be an easy build five a separate easy configs archive we've separated out with G history anything that was in the main easycon things reposit so before there was this directory inside there uh underscore uncore archives uncore uncore instead we've pulled it out into a separate repository um what we found is by doing that g operations are faster because there's less files there um speeds up the test Suite the combined of the previous slide and some work here speeds up the test Suite less easy configs to pass um it is its own repository it's already actually available but we're not going to install it by default um and they are provided as reference no support will be given um very much the idea there that it's a separate thing there just so people could reference and still have them available um there but if you do want them you can pip install them on top of your easy big um build installation we've removed a number of unused easy blocks so when I was doing and others were doing the conversion of run command and run command QA to run sh command if having followed the previous slide if we were left with no easy configs well actually sorry that slide having followed that got take arched out the older thing these are all the items where there were no easy configs left um where no easy configs left and they had either run command in them or run shell command so they were taken out if you care about some of this software it will be a case of con the bottom line contribute easy config and if it still needs an easy block an up toate easy BL uh block I yeah I've missed at least one out of this list because it's in a separate list I um archived the easy block for yeah my slides will be available online as well so we will make sure to send it around um you also archived the end cursor uh easy block and you'll go and I already talked about end cursors in this I think it was end cursors um that's the oldest easy config it was used back in nearly 10 years ago and I think it was nine years ago we switched every easy config over to actually just using I assume is config a make um and had forgotten to delete the easy block which I only found when I was trying to edit something else um but most of this software as I is quite old um and that's why the easy blocks have gone um but hopefully some of them that if they are still existing have better install methods now um I've certainly found that in some places yeah if there is any easy block that you still care about but you don't have an easy config is there a way to access them still or the only way is to just to like get go back oh well get history I mean they're still there in easy build 4.9 so you can probably get them from an old release there as well um if you really care about them then yes you you can get restore them so you could if you needed it restore it in a get branch and then submit it to as an update um but one of the things there is probably going to want you know part of the reason is that when doing the Run command to run sh command uh conver version not actually being able to test whether some of these things worked was quite important because you know some of them are quite easy commands that could be tested but some of them are quite complicated it's like I have no idea well it really BS down to if if an easy block disappeared that you were still using maybe with only in-house e configs or something then either if you know how try to restore it and try to convert it to be compatible with e bu 5 or let us know somehow like mention it in slack or open an issue do something if you need help with that we're happy to help out to pull it back from develop and revive it um and maybe I mean maybe even we can do the porting and then you do the testing that's fine as well I mean the porting effort is not the biggest issue usually it's just making sure that we didn't Overlook something when when doing the port now there may be exceptions but most of these I don't think are still relevant do do you see something that's still yeah classic is still used quite a lot well so it's not because the name is here that the software is no longer supported the easy configs may be using a generic easy block now and then it's a nonissue so but yeah take a look and I mean we'll we'll do some Hands-On today if there's something that you notice today that we still need to Port over that's fine none none of these I'm currently installing with easy build so it's not it's not keeping me from use thank you it's not keeping me from from using these These are we installed these differently but the it's very common when I talking with hel it's very common to have old software that probably was built with an old tool chain and maybe needed an easy block or something like that and no longer does or just needs to be ported over to something new y but but I can see several here that that I would want to keep around y yeah as I said we're happy to revive them just we didn't see a need and then Mak sense there's also related that this only so this happened because of the Run shell command and sorry run command run shell command and run command QA conversion there's also actually also a separate list that hasn't yet been in touch of ones that are no longer used but don't have run command or run shell command I haven't I had about four of them but I've only got about as far as D in the alphabet before I ran out of time um and we all have a similar you know we publicize that list somewhere as well then see um uh remaining tasks uh we need to decide on what the remaining breaking changes are for easy build five um and this is the maintainers or the group of us working on easy build 5 will go away and do soonish in one of our meetings these were the items that need to be merged quite soon because if they're going to break functionality we need to make sure people know about it and we're advertising it and correctly um I'm not sure whe there's anything else in there but we'll need to do um it's then deciding what remaining non-breaking changes there are um we've got more time to merge those but you know the more time we take the longer it takes us to rease easy build five um probably the biggest Step at the moment other than well is in these two it's right the documentation and yes there's quite a bit more of that to do um some of it will come out of these slides and then it's test test test which probably via a release candidate um there and then hopefully a release so there is a bit of a short-term impact that's going to happen um at some point soon we will merge the five Ox branches into develop in the easy build repositories when we do that configs that are in develop configs that are being PR may not work with easy build 4X um so the python package defaults I talked about things like use pip equal true because that's now going to be set by the easy block it won't be in the easy config but our current configs or easy configs require it um so you know it will make it slightly harder to be used develop at that point um some easy blocks in development you know in PRS or in develop will um fail because they're using run shell command which obviously isn't in easy build for Point um easy build for um so what this will mean is if you're using from PR and include easy blocks from PR they're going to be what you know you're going to might have to think about what you're doing and we will try we will advise as to what's going on and what you should do um and work around um one of the things also is this CI will change to reflect the newly deprecated items um I.E where we've said please don't is in the future you will you know we'll have the PRS going please don't do you know don't do this it will cause failure such as the mod files in Fortran that I mentioned you know we will fail so far and Kenneth mentioned this that's a number of PR 75 for framework is mainly the newer Chang and deprecated remove functionality 170 in Easy blocks which is mainly run command um that's why there's most of them and 85 and easy configs which is almost entirely archiving all of them um uh yeah I can't get to that um observation we base all our stats and you know the one real place we can get well there two places we can get one is from the survey and the other is from you know meetings like this from PR sending us messages is telling us about things going on um those are the places we can find out about because that's how we're going to guess that you know python 2 use has gone down so much python 3.5 use has gone to nearly nothing so you know we can lose support for them um so feedback on those things um to us is important where you care that something remains so if you don't care that python 2 support you know is going then you's no point feeding back to us unless you want to come join the um go away python 2 party which I might start organizing soon um yeah a I'll be there um so handson we said there was some time for handson that's why I've gone through re quickly Kenne if assures me these are the simplest instructions for setting up an easy build environment so we will believe him uh we some copy paste them into slack in a few minutes moments um and Hands-On testing there is time for you to test um start testing very much what the important thing about this is very much what it's doing is putting easy build into a virtual environment so it shouldn't impact anything else you do only when you're inside that uh virtual environment it will have the same requirements as uh well requirements you need a module tool so you know you need to be on your HBC system or on a suitable container image or something else to be able to use it um and if you do have questions please ask um obviously I'm just volunteering you know Jasper and Kenneth and Adam to answer them all and yep that's my last slide can you go back slides oh Sor right so the the breakage for front PR I only just now realized that this is going to be a bigger issue than I anticipated because anyone who's using front PR now anywhere and this may be an easy stack files which we we're doing in Easy in the easy project as soon as we collapse the 5x branches into develop that will all break well in the worst case scenario because when when you're calling from PR on a merged pull request it pulls the easy confli from develop rather than from the pull request no no no so if you do front PR on a merged PR if the pr is merged it takes it from develop so as soon as we remove use pip and all that crap because easy B 5 is going to enable it by default that's actually going to be a bigger impact than I realized if you use from commit then you can choose but you but you act actually you actively have to migrate from from from PR to from commit yeah which is only possible since two weeks essentially so maybe we need to think this over again and think about if we can come up with a better approach I'm not sure if it's possible I think the answer is it's a fairly short period between collapsing the branches and yeah we need to get easy bu five out as soon as um but there will be a period where yeah what it's painful yeah and especially for people who want to stick to an easybuild 4 version for whatever reason if they also want to keep using from PR with that version that's going to be pretty much impossible yeah yeah they it will at least impact people's workflow so maybe we need to think about whether we can do something about it I'm not sure I mean we cannot change existing easy buol 4 releases we can do a 492 release that has something extra that eases that pain a bit so as long as you jump to that you'll you may 492 at the same time as develop as we do as 5 Z yeah well probably as the same time we do I think we need to think about this because this is gonna also it's not going to be obvious what's happening people are going to do front PR like they did before and all of a sudden it will start breaking without them moving to easy bolt 5 yeah um so that's yeah that's something something we need to do a bit of homework on I think yeah okay that's also why we need I mean at some point we'll do the collapse and I guess we'll hear soon enough whether stuff is broken right people are going to tell us um so there there may be a bit of pain in the short term uh related to this and that's also why once we have something that resembles a release candidate and now if you go back to the instructions there your virtual end stuff so this kind of testing if you have stuff that you want to test out with a essentially a pre-release of easybuild 5 please do and it may be as simple as doing a simple install in your environment with your current easybuild configuration and just see what happens does it still work does it how heavily will you be impacted by so I've not tripped over anything in my hooks that have caused problems so I've been installing any PRS I've done either created myself or tested since the beginning of this year I've done with it the easy build five development um I've not yet hit anything unexpected oh apart from the lamps and Mak St PA but that was about the only so yeah we are trying to make this as easy as possible to switch but that's also where we need feedback only when we get good feedback we can actually do stuff to ease the pain a bit okay any questions for Simon to to on the stuff if you change the meaning of fromia in 4.9.2 then that should fix the issue right if you pick a specific Branch rate hard than develop yeah and I think we can do something there to ease that pain bit we can from PR can check whether it's already picking up easy configs that are for 5.0 or not like how we exactly do that is a detail but that's something we that's something we we have to implement I think in an easy build for release and then make sure that people I mean we do see very good adoption to the latest easybuild version I think that's a reason to do something extra in the 492 or a 4.99 or whatever you want to call it like the last easy bu for release ever a 4 99 and then we need another one to 4999 one I think we'll need something this to e the pain I was wondering um from PR uses the develop Branch um do we actually make use of the main branch for some something because that could be an easy way out of this if we um are not using the develop Branch for what it's usually used EG merge it into a main branch at some point we can use the main branch for 5.x and anyone using from PR with an old version will get on from develope but only if we sync easy com between them I think currently the main branch always corresponds to the latest release the main branch corresponds to the current release but is there a reason for making the main branch follow the release and not just Tech on Branch that's I don't know maybe we need to the main brch maybe we can do something that could be another yeah because as you said that might um break a lot of people's workflow that were for example adopting the easy stack fires with from PR a lot yeah and it also works with a simplification um that is intended in easy build 5 um by using the main branch and not not wondering why are there two branches what are they used for just having one branch and text on that Branch it's less confusion I think yeah have additional branches as well we have like now branch is used for the 49x releases and that's separate fromel practical reasons releas yeah for release management it makes sense for example branching off a tech for releasing a do something release okay yeah that's maybe another okay so how many people here well and this is a question for remote for remote as well sorry yeah yeah keep keep the virtual L stuff here and do do keep in mind if you have python part set you may need to override python part as well because if you're in a virtual end python part is still going to win so depending on how you're setting up easybuild currently to do installations um sorry I'm what I'm just doing I'm putting this in slack maybe so people can copy paste it yeah yeah they are now in slack and hopefully slack and between PowerPoint and slack no sorry I Haven to use PowerPoint between slack and everything house hopefully there's no funny no funny characters no looks it almost certainly be either there or and dashes I think I caught them all um okay who's planning to to do a bit of Hands-On playing with easybuild 5 right now before we call it a r okay good a bunch of people are if you're not that's fine then I guess you're free to leave or you're free to stay if you want to ask questions um free to talk and ask questions to talk and free to discuss so now it's really the the end of the official program for today at least um for the people remote you're also welcome to play around we'll try to keep an eye on slack as well um but also feel free to do this whenever um so later on if you're trying this playing around with easybuild 5 the eb5 channel in the easybuild slack is definitely the right place to ask for help if you're running into something funny when you're when you're playing around also if you open do open and get Hub isue in any of the repositories for and you are testing easy build five can you put that is about the first thing you put in a comment yeah in the description so that we know um who whichever maintain is picking it up doesn't go well that's really weird because it works perfectly fine yeah yeah make sure it's if it's specific to easybuild 5 that you notice that we'll then add the necessary label to that as well so we can easily zoom in on these issues all right do we have any more questions for Simon before we start playing around a bit no okay then yeah Simon and I will be walking around maybe Jasper too where's Jasper yeah oh yeah okay yeah screen as uh yeah sure I can do that yeah I've got some stuff uh so I've renamed the um I've got support for renaming easy config parameters templates and the constants I haven't done the configuration um yeah I've got I opened another draft pull request that actually implements it with like the um renaming added to the dictionaries that I added so you can play around with it if you check out that one uh once the one that actually adds the implementation is merged I'll rebase that one um but yeah the idea is that you can um the it it won't easy build won't complain if you use the old versions or the new versions or a mix of them as long as you don't well as long as you don't use the same one twice as in like use config Ops and configure Ops at the same time too many so that's the yeah so that's the poll request that I just opened uh there's an issue that I opened in the framework which goes through a list of all of the renamed things uh if anyone has any strong feelings on how we've decided to rename some like feel free to comment on the issue or on slack obviously um it's a pretty long list uh there is some rules for how I selected the renamed version further down in that issue but basically it's just standardizing the use of underscores whether we use Ops or options and stuff like that depths dependencies and so on yeah so that that's definitely an interesting one to go through if you're because this is going to have in the end high impact not an easybuild five because we're what we're first doing is adding support for the new names before we touch the old names but at some point we're going to deprecate the old names and it includes very common easy config parameters like config Ops we're renaming that there's a bunch of others as well that we're also renaming so you're you're bound if you have any custom easy configs custom easy blocks hooks you're bound to hit this stuff at some point so it it's better to give feedback now than after we've put this in into probably easybuild five we're going to support in new names yeah the idea is as well that we will provide some scripts that should automatically rename at least easy configs yeah uh possibly easy blocks as well um we'll see how straightforward that is I guess yeah we'll see how difficult that is the largest effort will be doing framework really I think are you considering to have at some point say strict mode so that we can check if it's going to support only the new one and stuff like that uh yeah that mean that hasn't been done yet but I think that has been talked about before in mon of our E5 meeting so yeah that would be something we would look at as well so that you can check that everything that's is there the easy way at the moment would be to manually add them all to the deprecated parameters but that's you know that easy build will spit out dication warnings at the moment a lot of them because it does one per instance it finds rather than collating them into into one nice warning yeah I mean I imagine we'll do something like we've done for the failing on fortrend mod files that you know there'll be something that will flip it from being a warning to a a crash because that's how we that's how what we'll use in the test B to make sure they're not being used at the point we want them to do mention mod files and I was just thinking I was wondering if um would we since there are other modules that use mod extensions would it not make sense to flip the sense of that so you check for mod files on option and just ignore files without an option since they would only really appear in Fortran code and they appear in so many other modules or several other modules so so at least in so it's they have to be there in an installation of theof software and so it's not just that a piece of software might call its files files it's that it has say some examples which is the clex and whatever example is it actually has some examples or inside its installation so yes they might be used in a lot more but it's quite rare that you'll find them inside um an installation um so the answer is I think that's the right way but we you know if we if we hit if we suddenly hit a whole bunch of PRS where we see a problem we might revisit that um it's quite rare yeah I mean from what I've seen I think it's predominantly for TR files had a few EXs at the moment we've gone with the naive there is a do mod file extension um we think there might be an enhancement where we actually explicitly ask some Linux utility as what is that file and we might then be able to tell even more information and if we're hitting it a lot we'll enhance the check and go it has to be some other properties as well um but at the moment I haven't hit enough of them to think it's an issue um because also you know we are only talking about GCC core in installs so that's all you know it's already minimizes the number there's likely to be [Music] is there is there anyone who had any let's say wild ideas for easy buol 5 that you were hoping to see but you haven't seen mentioned at all UV UV yes UV replaces pip install and the en so you want to have support for using UV and easy build instead of P yes yeah to me that's an additional feature that we can implement or you can Implement anytime I know so we're so this kind of stuff extra things that easybuild can do but don't affect current behavior at least by default fine I mean for me they're out of scope for easy bu 5 we're looking for stuff that we can only do when we have a new major version because that's what needs to happen now before we wrap up on easyb 5 um so I'd heard fairly recently not too recently that um Mamba was starting to be competitive with Anaconda has um has there been any mention of building support for that into easy build or I think that may already be there okay okay yeah you can you can already install Mamba with easy build that's for sure let yeah yeah don't don't steal mine I think we can install that I think there's an open PR for installing things with M other easycon things that use Mamba might be an open PR rather there's Mamba support in the cond easy block which is so if you're if you're using an easy conf file that uses the K easy block and you have Mamba as a build dependency it will use that or you can at least opt into it sometime so so I I just noticing that James is nodding which must mean we've used it somewhere again that's something that we could add any time right so not specific to easyb five at all yeah one of the most work intensive things that I found was that uh when you try to update python uh dated extensions is really painful so um R um there is this tool called easy update I think yeah it works really well with r but with python not so much so something like that would be nice in integrated in easy build itself me yeah so there have been plans is a big word ideas to integrate that into easybuild M including John Day who developed the easy update tool he has come to the user meeting couple of times already mhm and we always said okay yeah let's get our hands dirty after this and then make it happen and then it never really happened up until now now yeah it would be very useful I fully agree um it's not that easy of a task to do but maybe we I mean and his implementation is probably I don't want to say old but it's using old techniques that are there's now better apis for that okay that we could Leverage the the ecosystem stocks that's coming up on Frid on Thursday um is actually one thing we could use for this because that's like one Central API where you can talk to pipie and c and a whole bunch of other things I haven't really looked into it in detail yet but that looks very promising okay and there's there's other things like this as well that may help us a lot to at least reduce the effort significantly to be able to do something like this yeah okay but having that would be I think a killer feature for yeah yeah it'll help a lot yeah but again yeah in terms of easu 5 it's not a breaking change okay it's an extra feature that we could Implement any time fair enough yeah but that yeah I I agree that would be very good to uh integrate somehow yeah and what you're what you mainly want to do there is say start creating an easy config and let easy both figure out the dependencies that are missing the extensions that are missing or yeah perhaps you can create based on like old easy configs that already exist you just update them and then then uh it checks the dependencies if it still matches or yeah I have an ugly dirty script that does something like this but it's not integrated in Easy at all okay but it it could become a feature yeah just because yeah I think the the reason why it works fairly well with r is that uh San has some easy access easily accessible metadata on the dependencies are which pipie doesn't have so basically you need to download the packages before you know what the dependencies are need to install stuff before you know what to install yeah yeah basically and that's fairly intrusive right like you need to like you're downloading for something like pipy bundle uh python Das bundle pipy whatever it's called um there's thousands of extensions that you need to download somewhere and check what all the dependencies are and build a tree from that whereas for R it's a lot easier to determine I think with r also talking to IND is um nearly every r package has no real version lock IE if you get the latest one you just get the latest one of everything and that's probably going to work together near enough I I'm not going to say 100% but it's whereas python you more often hit need greater than this less than that you know having do environment res resolution to work out what's a consistent and applicable one of the easiest ways of figuring out for python is doing a pip install of what you want and having a look at what what what is actually installed and then pip freeze yeah check check this UV thing because it's a dependency resolver for pip so but can it do it without installing crap I am not sure I I I think the metadata just isn't there on Pipi as far as I know you have to download and look at yeah download you have to download the python package extracted and then you can look at the data um that you find out which other packages you need to download and do it over and over again yeah wait wait wait Sam Sam yeah um I contributed a script which basically does this it creates a temporary virtual environment installs the stuff and then uses the PIP de tree to figure out what you need to put into your easy config so it would be better if that's actually integrated into easybuild somehow as an option no yeah it integrates pretty much already so it's cause easy build um but it's a standalone script yeah we just push it Standalone script um which is on top of easy build so it actually calls easy build to for example get the environment in which to create the viral environment yes in the framework uh scripts folder also so who who's concerned about easy Bol five is it scaring you that you'll have to Jasper is because he's losing too much time on it I guess because we we do want to make it as easy as possible for people to make the jump what we don't want to have and we saw a little bit of that with easybuild 4 is that people like drag their feet and try to stick to the easy build three or four release whatever we we can do to avoid that we're definitely going to do so if there's ideas there they're also very welcome the thing I'm most worried about is the rpod because we don't do that yeah that's trivial because there we can just flip the default again you can just tell easy build art false and that's it so that that's only changing your configuration that that should be very easy if you're worried about that yeah but we don't want to deviate Too Much from the default because that also creates issues I guess it depend there's there's many people here that have opted into RADS a long time ago and are very happy with it and will EAS build continue testing without our part yeah probably have especially in the short term because we know that many people will probably keep it off I think also my plan is probably that at some point I you know I will have some cut where all the old older tool chains so we split out my tool chains when I go into a tool one specific tool chain generation I do all those installs with one set of config and so what I'll probably do is go all the older ones will stick with no rpath you know with ourth disabled and at some point I will just build it when I build the new stack I will do that with ourpath um specifically for that sort of reason of not not ending up in a mixed stack but also um giving myself a convenient point where I can do the you know jump um be know which also means that be I know I'll be naturally doing some testing for the next few years probably we we're likely going to I think switch in the middle of seven eight tool chains at the same time you're going to sorry switch to our path on their already existing tool chains yeah not recompiling anything no and and that should actually work yeah that's that's something we're definitely trying to make sure that it works that with easybuild five you w not have you will not have to reinstall everything you have you can just switch to easy bu 5 on top of what's already there and just go ahead and yeah whatever problems pop up we will try and fix there may be some especially if you're mixing arpot and non arpot but that should work yeah that's definitely something we're going to support but Sam made a good point that we should make sure we keep testing in the non arpot configuration like one of the Bots has to keep testing in with non rpods so that's something but yeah but I mean that's again back to the we you need to be doing more testing and understanding yeah yeah we we need to come up with a good set of test configurations we need more en yeah if anyone has questions on playing with easy Bol 5 let us know if you're trying something that doesn't work and you expect it to work we're interested to hear that now is a very good time to do that throughout this whole week by the way if you're playing with something maybe while watching talks that's allowed um you're allow to play play during night time sorry you're only allow to play during night time or night time as well yeah who sleeps this first thing tomorrow morning when you can play you know not sure who's talking first tomorrow you want to say something about this or uh that's just the demonstration of the download instructions which is you know one of those features I quite like um the other bit I'm intending but haven't yet actually added is because well I'm not sure whether Kenneth has Rich enabled in that environment probably not because there but also some use of colored or formatting via Rich to make that even nicer or more obvious um but that's not enabled in that command yet um but you know but that that's not just for that command that's for the output is you know better use of color and and um you know other features Rich has to do with making things a bit more obvious um where it's available how many people here are using Rich already in combination with easy build who's not everybody else okay then I want to show you what it does you don't like it you don't like colors everything but so if you're downloading stuff for example it's going to show you it it did show you download progress but it's why is it so quick here okay I need something bigger ah no no no wa it found it already hold on I don't want to force to download like this so it's going to show you what it's downloading how many things it needs to download and then show you progress bar on that so that's pretty nice and also when it does an um an installation it's going to show you how many things it still needs to install and in the case of extensions let me just make sure I'm not messing up something here this is just in my own personal account so I can't really break something um if it's doing an installation and we could actually add more colors not only to download instructions but also to to other stuff everything yeah you you'll just get better feedback here like how many steps is it doing and when it hits the the extensions you'll see another progress bar popping up as well so it just helps a lot question who part how do you combine rich with d Das job if you can't you can't yeah if you're not looking at your terminal it's useless yeah it it actually I think it does it disable rich I think rich is pretty intelligent in in knowing when it makes sense and it doesn't so if it's in a headless environment it probably just disables itself essentially I've never really seen it getting in the way so the question is whether we you can add an extra functionality on top that that tells you what jobs have completed and what you to mean that's a different tool that's a different yeah that's probably a good argument I mean yeah there may be tools ask Slim develop b said he had set Pyon unbuffered to monitor it during das's job he had to set Bon un buffered yeah because the buffering kills Rich it kills rich in the slur J not kills but but kills the continue use output ah okay oh we've actually seen that right empty jop output files let may be why thought we see that in test bot as well we only get the output at the end open open an issue on that fix or P to fix it okay must be the middle of his workday so yeah I'm not actually can we opt into that I'm not entirely sure well well I I I'm assuming it's uh you tell R you know you we probably just need to tell Rich to better detect we can figure it okay maybe I should show something with run shell command as well well that's using that's using it yeah some of the output here at least so I should show you what you get if you if you're not using it I guess oh now it's going another land so let's see what happens go away can hide this let me just show you the difference between easy build four and five and you can tell us whether it's better or not or if there's room for further Improvement I'll just run the exact same thing almost the exact same thing why is it not picking up the whole thing come on don't want to step this thing make sure these things don't step on each other's toes so I'll set the different prefix The Bu part that should be enough let me pause this one so I can actually compare it at the same point so the because we're using run shell command internally some of the output the trace output at least of e will change a bit as well so when it gets to the point where it's actually running the configure you'll see the difference so here this is the the new output where it tells you when it started and where it's working and where the output is so you you will always get this withu 4 you would you will not always get this and it's a bit less organized and the extra bit about the m doire and the command doire will be an extra line in that at some point once we merge I think it's an extra line at the end but um just running off to the back here in which directory are actually the N do uh shell and uh command. shell is it in the lock or is it in the build directory I have no it's in a separate temporary directory and E buil will print you the parts where you can find it so if a if a shell command fails certainly then but maybe always and Mikel knows the details um EAS will just tell you look in this directory for output of the command and scripts where you can easily jump into that that shell yeah right now it's not there because I'm not using that Branch yeah from what I remember I think the currently it says output save to blah blah blah and that gets replaced with um output and environment BL like get sa to there and then there's a separate sub directory for the uh environment scripts let me try and show it see what happens how quickly I'll shoot myself in the foot the reason I'm asking is currently the lock files in our installation go in One Directory which is actually on our login node and the build directory is St shared memory on the build node so I need to specify that directory has an environment variable on the build Noe where you build it not on the loogy Node where you yeah it's it's a temporary directory on the worker node but you can you can reconfigure reible to use a different that is the reason I'm asking it it will become clear a little bit Yeah Yeah better tomorrow but yeah you you already have the necessary configuration options to to change that uh no I don't want to do this let's see if I got this right so now I'm messing up my virtual L totally don't do this if you're not sure what you're doing and now I actually want to yeah yeah yeah we'll be wrapping up soon if anyone has questions Now's the Time all right I just want to try and do a quick demo of the the dropping into a shell thing I'm not sure I've have things set up correctly but let's see what happens yeah maybe I did where is he telling me I got it wrong you know what I'll just yeah no no you have to you have to you have to run this script to drop not counting that spe just see if I got this right what did Nel say it's only going to drop the script what he coming is you don't need to be BR you can just here yeah here it goes yeah so this this is what's going to be different yor yor here it's going to tell you where the stuff is saved if it gets to something meaningful I can stop it yeah here so I can now cancel it and check what's in here and in here you have these scripts and if you run the N script yeah no no the command script yeah this drops you in a shell where you can Arrow up and you get the configure command and this this also has all the environment variables set that eable sets so here you can easily play around yeah sorry Sam no so the command. shell uses the n. shell I think yeah you have to be a bit careful where uh see it uses this as a input to the to the B shell you can Source this directly as well and then only play with that but the the command shell also drops you into the right working directory correct working directory correct environment and you get the commands you just Arrow up and you have the correct command that that's one of the things that the new run shell command easily enables us to implement and the old run command is was no I wouldn't start with it so that's one of the reasons we're doing all this efforts to two Port away yeah