EasyBuild + EESSI UK workshop (April 2023), part 5 of 5

good so yeah questions on easy build we handled that so the the last part of the workshop is um an introduction to the easy project um and that has sort of evolved from the easy build community not directly but there's lots of people involved with easy build also involved in the easy project this is really my favorite cartoon so this shows we have some very good ideas here on what we could do in the easy project but we have a day job and we need to keep the researchers happy with what we currently have so we're like we're almost too busy using Square Wheels even though we we know that there are proper Wheels ahead of us but okay so this is let's see not progressing as fast as we would like it to but we are actively working on it and we are making progress so what is easy easy is short for the European environment for scientific software installations we pronounce it as easy that's not an accident there's a clear link with easy builds and and things are this is supposed to make things easy right so that's why we gained that a bit the the European parts we often get questions on so if I'm not in Europe does that mean I cannot join also here in the UK that's a relevant question um we could change that first e to another word that starts with an E so we're and we actually consider that at some point um it's really just to game it and to have a a project name that sounds like easy um it was also a done deliberately to some extent to get some European funding for this which eventually worked out so there are some reasons why we have the European in there but there's definitely no limitation in terms of countries who can join or people who can use it it's an it's an an open source project just like easy with this um what we want to do here is we want to build work together to build a shared repository of scientific software that's optimized like easable does optimize for specific hosts and it's a repository of installations itself and that's an important detail we're not doing software packages like RPMs we're not doing recipes like easy config files that you don't have then have to install themselves we're sharing the installations themselves and that's an important detail and we will show this when we go through the demos and the actual structure of how this is organized what that means um the main goal we have for this is avoiding duplicate work so even if people work together on easy builds you still run into weird issues that you only see on your system you have to figure those out um so even if somebody already wrote an easy company file that worked for them you'll still have to go through that installation which may take time and it may fail because of specifics of your operating system or your setup and that's what we want to get rid of we want to start working together on the software stack itself rather than tools to get that in place um we also see this as a uniform way to providing software to users so right now even if the software is installed with easy build across multiple systems things are going to look a bit different maybe the model naming scheme is a bit different the location where the modules are installed are different some systems are set up where the modules are available to use and to load directly so and some other systems you have to do all you'll use first and maybe they're using a hierarchical model naming scheme and you're not so things get confusing and things are a little bit different and different enough that the researchers get confused if we have a shared repository which you can use in multiple locations it looks the same everywhere so that's already a big hurdle that's removed for the researchers the goal of this project and that's a very ambitious goal we fully realize that is that the installations we provide should work on any Linux operating system regardless of whether Ubuntu or red hat or whatever other variants or whatever version it is we shouldn't really care um about that um it also works in in WSL so the windows subsystem for Linux you can play with those installations there as well which is really just another Linux environment it's like a VM you're in um and we're also considering supporting Mac OS so that's different enough that's going to be another big effort right now we're not working on this but the door is open to also start doing this and make that possible so the goal is really to give you a set of software installations that work on your laptop on your personal computer at work to an HPC cluster and even in the cloud AWS Azure Oracle Google whatever you want to use it should work there as well um so that's mostly in terms of operating system but also it should work on different types of CPUs all generations of Intel new generations of Intel the AMD also arm CPUs um Power we're currently still playing a bit with power 9 but we'll stop doing that because that's sort of a dead end but in the future risk 5 is coming up as well as another CPU family let's say and we want to add support for that as well so lots of operating system lots of different types of CPUs and that we're not looking yet at interconnect Infinity bands or in the cloud EFA and AWS for example um different generations of Nvidia gpus but also AMD gpus Intel gpus ideally we want to support all of those as well and that's very ambitious but because we're working together and we're sharing the software installations themselves we believe that actually becomes feasible um and we'll see how far we get with that in the next couple of years so the focus of the project is very much on performance where we want um to use this for scientific software that's used on HPC systems so performance is a very important aspect here we'll have to automate as much as we can to make this feasible so if we'll have um people manually building software for these all these types of system architectures that's not going to work so that needs to be fully automated we want to also make sure that not just the installation work but that works but that the software actually runs and runs well so that it functionally works that it performs well so we're going to be testing all of these things and we'll have to collaborate together to make this possible so there's a website there's documentation on GitHub and I'll show you the the pilot set of the proof of concept setup we have for this and explain how this works this is just to zoom in on performance I probably don't need to explain this in detail here but this gives you an idea of what the impact may be if you're not being careful about the binary that you're running on a very capable system so this graph shows you the performance that you get for grow Max um one of the the brace Benchmark inputs for gromex um all the benchmarks or all these tests are run on the same system and Intel Cascade Lake system and all we're really doing here is using a different binary for the exact same version of gromac so we didn't touch the code at all we're just basically building this with different compiler options and seeing how how it performs differently if you build it with only ssd2 instructions so it runs anywhere on any modern x86 Hardware we call it the generic binary you get a performance of about let's say one simulated nanoseconds per day so that's a measure of how fast the simulation is the larger that value the better the performance is if you start using AVX instructions as well linear boundary that goes up and to the very end we also use AVX 512 instructions which are supported by Intel casket click then you get way better performance so performance actually goes up but let's say like 70 percent by only using proper Vector instructions for for the binary so the impact can be quite big and grow Max is probably a pretty extreme example of that maybe we're more talking 10 20 but that's still significant right that could still make a big difference so that's a very important point so what do we have as major goals in the easy project well first of all we want to avoid duplicate work not only for the people who are installing this software the HTTP support teams but also for the researchers so we don't want to relearn the software stack when they jump from one system to another ideally they can use the same um the same software installations everywhere tools like easy build and spec already do some of that but they're not really sufficient because okay we're automating the installation procedure and if it works for you that's good but you're still doing the installation and if it doesn't work you'll still have to figure out why that doesn't work for you and try and fix that so there's still lots of duplicate work here so we want to go Way Beyond just sharing the build recipes we want to go to towards sharing the software installations the actual binaries themselves um we want to build a uniform software stack that runs let's say anywhere so regardless if you're running in the cloud or running on your laptop or running on an HPC cluster you're basically using the same installations it looks the same it feels the same you know how it works and you don't need to lose time um to see how how all of that works um so mobility of compute is sometime sometimes that you often hear about when when talking about containers and conda so you're just taking your software with you we're doing sort of that as well it's more like the software follows you um automatically um and we do it in a way that we're aware of this this performance issue which in containers it's usually just it people ignore this silently just run one binary everywhere and assume that will be okay well it's often not and so you shouldn't be cutting that corner and we have a better way of dealing with that um if we manage to do this this will help with HPC training we can spin up virtual slurm cluster in the cloud train the scientists on this make sure they're lamps software for example is there they can play with it and then if the then if they can get home if they also have access to easy they get the exact same installations on the laptop they move to their Institute cluster they can also have the same binaries there and they basically know how things work so making the jump from being trained to actually using it after the training becomes a lot a lot smaller uh we think this can also help developers of scientific software so um let's say open foam as an example the open form developers probably know their own code base very well but they need a bunch of dependencies and they don't want to go through the pain of having to install those dependencies or having to figure out how to build different versions of GCC or C language so they can experiment with all these compilers ideally these things are just available somewhere for them to experiment with and also in in CI environments like GitHub actions for example they can get easy access to a wide variety of compilers there it's probably going to help them to also test their code with those compilers so there's lots of things that become possible if you could get this to work all right so how is this project organized it's really a layered structure three main layers a file system layer compatibility layer and a software layer the file system layer is what's responsible for Distributing the software installations we provide and for this we rely on an existing project that was created at CERN which is called CERN vmfs certain VM file system um this was built exactly to distribute large amounts of software across the world in a very easy way and I'll explain what this does it's a bit difficult to grasp but it's actually doing because it's so different from other types of Distributing software like building packages or or downloading container images but that is a very powerful concept and we're very thankful that the the certain developers the people at sir who developed this um have been sharing this as an open source project the middle layer the compatibility layer is what we need to shield shield ourselves from the host operating system so what we're basically doing there is building our own mini Linux environment so we don't have to rely anymore on what the host operating system provides the main part in there is glip C so we have our own glipse in there and then whenever we're running on Ubuntu or or Fedora or Centos Linux we don't really care because we're not we're not going to use the Gypsy in there or any of the other libraries to the extent that we can avoid that there are technical reasons that we sometimes can't infiniband drivers GPU drivers those will still need to come from the host because there's too much they're too tight to the kernel um so that we cannot avoid but we'll have ways of detecting what's there and if needed installing missing stuff and we're playing with that actively already so that's still feasible um on top of our compact layer we'll be stolen the actual scientific applications open phone tensorflow all these things they will link to the libraries and the compatibility layer and that way they can run anywhere because we're only relying on the host for the kernel essentially and whatever code lot rifles we cannot avoid and that works and that works well so this software layer is being installed with easy build today there's actually nothing specific in here that requires easy build we could use spec or other installation tools in this top layer as well as long as we can make sure that the whatever is being built in here links directly to the compatibility layer and not to the host so we cannot pick up anything from the host otherwise it won't work as expected next to easy build we're using lmod so we're generating mod environment modules along for these installations so you can pick them up we're using an external Library here called archpec which does detection of what type of CPU you have so not only is it x86 or arm but also is it Intel Intel or AMD and which generation of Intel or AMD is it is it one that perform that supports avx2 or Apex 12 instructions or not so this time the library basically tells us what type of CPU we have and based on that we can pick installations that are optimized for that type of CPU and the whole software layer is structured in such a way that this becomes very clear that this is happening I'll show how this works so that's the three main layers we're also actively playing with reframe for testing so reframe as a software testing regression testing tool created at cscs in Switzerland um like I said we want to do a very good job at not only providing those installations but make sure they work so we're going to be testing those installations in Ubuntu in sentos on arm on Intel on AMD and make sure that that all works I will do functional uh tests to make sure it runs but also performance tests to make sure it performs well so that we're on the right end of that grow Max Benchmark put um we're looking at today we're looking at Intel AMD arm and Power 9. power down is that then we'll stop wasting time on that but we will very soon start looking at risk five as well so there's there's big European projects that are actively looking into building um accelerators that use the risk five instruction set but also very capable CPUs that user describes so I mean let's say five years from now these will be very relevant um and we want to be ready for that so when when the capable CPUs are there we'll basically have the software to run on them which is usually the other way around happens CPUs are there and then people start figuring out how to build for it we can actually prepare for that already and hit the ground running as soon as let's say the first risk five supercomputer is there okay so that's a lot of information um let's see how should I continue I'll first explain a bit more what the file system layer does so that's how we distribute the software that we provide in the easy project um this is basically what what CERN vmfs does so this is not something we created but it's to give you some idea what's what certain vmfs provides so certain vmfs allows you to build a file system something like NFS that you can mount somewhere and it's always going to be read only for the clients so the people who use the file system cannot make any changes at all in the fastest they can only consume what is there um and it's mostly targeted towards software installation some people use it to distribute data files big data sets as well that also works what happens is you're creating and what's called a cvmfs repository so a file system in a central location a central server which is called a stratum zero and you make sure there are enough mirrors of that threat and zero that have a full copy of the software stack available and they think automatically with the stratum zero so whenever software is being added to the repository this can only be done here centrally and it just syncs up to everything around the world you typically have multiple of these mirror servers to make it redundant if this guy dies if everything is fine because these I'll have a copy and as long as there's one mirror server somewhere you can still continue and consume those software installations and use them in here you could have and we eventually will have thousands of installations um these have a full copy of that and when you as a friendly user here start using that software for example on an HPC cluster or on your local laptop or in the cloud um you'll basically be mounting from one of these mirror servers you're talking to this guy and if you fire up tensorflow you're here from a cvmfs point of view you're a client computer a few fire of tensorflow it's going to check cpmfs is going to check on your local cache do I have that tensorflow binary already I don't okay so I'll ask here in this caching layer this proxy cache um or squid cache extensorflow here already if not it's going to ask the mirror server please give me the tensorflow binary it's going to come back here it's going to be copied into your cache you have the binary and you can run it that binary is probably going to need libraries so the same story happens for the libraries so it's what's basically happening um very simplified is that you're streaming your software installations it's like Netflix when you say oh tonight let's watch this movie you click the movie and it starts quickly downloading the first part of that movie so you can get started it's the same thing here when you start running tensorflow it says oh I don't have that binary I have to download it it downloads it to your cache you can run that binary and there's a small startup delay of course because it has to do that um well the next time it's going to be very quick because I have a local a local copy of that in there um so that's one thing you have this streaming um streaming idea and it's fully transparent to the end user so if they start typing module Avail what is there what module Avail means I need to know what's in the in the directory it will copy the metadata for that directory which is very quick um that look you're loading the tensorflow module or an instant module file it's going to download that so all of this happens behind the scenes and it feels like everything is local maybe with a small delay the first time you hit something but other than that you can't really tell that it's it's all streaming in in the background so cvmfs hides this all from you and thanks to these multiple caching levels you have cash here you have the cash here which could be in the network of your HPC cluster you could even have your own full mirror server next to your APC cluster to make sure to reduce that latency for downloading um stuff everything is here even if this guy dies it will just automatically migrate to the other one to download stuff from there so as long as there's one somewhere around the world everything will nicely keep working if this guy dies no problem for a couple of hours we cannot add new software but everything else has a full copy so it's all good so this it's all like this peer-to-peer Network for streaming um your software the big Advantage is wherever you are in here in the cloud on your laptop as long as you mount this easy repository the serving MFS repository you're getting the same software installations everywhere so that's how we distribute stuff but that's not enough because if you build a binary for me on Ubuntu and I'm on Centos it's hopeless writers there's no way that's going to work so that's why we need the Second Step the second layer this is the compatibility layer um we construct this with Gen 2 prefix so Gen 2 is a Linux distribution where you typically build everything from source Gen 2 has a sub project called prefix because you can install your own Linux distribution in a prefix in an installation directory that you choose yourself when you're using cvmfs everything has to go into slash cvmfs so we need to be able to build binaries that work in that prefix that's why we use Gen 2 prefix for this uh again the biggest one here is gdpc there's other libraries as well to figure out usernames and things like this um but it's pretty pretty minimal so only the stuff that we really need is there um so we'll construct the compatibility layer once for every CPU family we support once for arm one's for x86 for now once for power as well and eventually one for risk five 64-bit as well so you basically have four mini Linux installations that are included in your cvmfs Repository so that way we Shield ourselves from the operating system and we can build a binary in here that works on Ubuntu or Centos or Souza or WSL whatever any Linux distribution so it looks like this in terms of the structure all CPM faster repositories are in slash cvmfs pilot easyhpc.org is the name of our easy Repository in there we have a couple of versions because over time we'll actually do versioning across the compatibility layer as well every now and then we will rebuild this Linux environment to get a new glipse version maybe once a year or once every two years whatever makes sense and in there we have a directory for Linux because eventually we plan to support Mac OS as well and we have a subdirectory for the CPU family that you're using I can show this interactively as well to get a better View on what this does and then on top of this we have the software layer so this is very easy build and lmod kicks in which we've been playing with for let's say the last day so in here we will build our big scientific applications and all those dependencies that it needs and they will link to the compatibility layer glipse and whatever other libraries not to the libraries provided by the host of us and that way they will work as long as that compatibility layer is available we're currently using easable for those installations we could be using other tools as well so anything that we can control to only use stuff from the compact layer Works spec I think today cannot do this but it could be enhanced to also make sure it doesn't go outside of this little box that it's supposed to stick into and then Arch pack is used for detecting what type of CPU you have a small part of that is which one of these three families are you using but also way more specific than that it's going to check if it's Intel and if it's an Intel Haswell or Intel Skylake like this so that means this software layer is not one set of installations software installations it's actually one set for Intel as well one set for Intel Skylake one set for AMD Rome one set for armed graviton 2 which is an arm CPU and AWS and so on so we basically have every installation multiple times for as many CPU types as we we need to care about okay I'll get back to this later let me show you how what this looks like on a system where we have easy available and it is right there in our prepared environment so if you do cvmfs you can start looking into this yourself you will find the easy Repository right now it's a bit more messy than we would like it to so there's an old version here uh that dates from the time before we had a versions directory So eventually this part is going to disappear in versions We have currently two versions this is assembling to the other one this is our latest version which is getting quite old we're actively working on a new one and we will probably ingest that next week so we've built one in April 2304 which is going to be our next version we have a latest Sim link as well which now points to uh the 22 21.12 so let's just use this latest Link in here like this in here you have the compatibility layer and the software layer so the two layers that are included in our file system layer which is cvmfs if you have an init subdirectory as well we have some scripts to set up your environment let's take a look at the combat layer first like I said a subdirectory for Linux because eventually we hope to also support Mac OS and here we have three CPU types our power and x86 and if we look into one of them this is where you'll see something that looks like a Linux file system hierarchy binaries libraries in here delete C is somewhere I always forget where here it is so that's our our glipse that we will be linking to uh in the software that's the compound layer our software layer has a very similar structure what type of os are we using Linux or Mac OS what type of CPU do we have RM powered or x86 but then here it gets a bit more fine-grained and x86 we have AMD Intel and generic generic means any activity 6 CPU I don't really care but then the Intel and the AMD directories we get a bit more specific So currently we have Hassle and Skylake for Intel and so basically avx2 and AVX size 12. and in AMD we have sent two and send three so Rome and Milo and then in here you'll find the modules and the software directories that easy build produces when installing software so you can see we have this whole structure in the repository so that this that's absolute Madness you would never let a research or manage this by hand right they would never figure it out that's where the init script comes in so we have an init directory which has a couple of scripts the most important one is our init script here and this does some magic this will use the arch spec python library to detect what type of CPU you have and use that to set up your environment so if we Source the script this script is going to change stuff in our current Channel environments we need to Source it not run it and this is very silent why is it very silent because it's probably already done let me do it on this confuses me it's supposed to give me some outputs I guess it's because if you log in it already does that by default so let me do it on our system to show you what kind of output you should be getting you can see it drops us in an easy environment but it's not really producing the output I'm expecting there's a way to make it silent so maybe that's enabled by default in the prepared environment let's do it on our system where I know it's not going to be silent so on our systems in Ghent we already have the easy repository mounted but we're not telling anyone yet so the researchers don't really know if they would be looking for it they can find it but we're not promoting this yet because we know it's not really let's say stable and reliable so but this is doing what the script is doing is it's doing CPU detection using archpec so and it produces some output says our spec says we're on an x86 CPU AMD Zen 2 so it's an AMD row which matches our login nodes that makes sense using that information it says okay so this is the subdirectory I'm going to use in the software label and it's telling you here in cvmfs Pilot easy the current version of that in the software layer I'm going to use the Linux um subdirectory I'm going to use this particular subdirectory for the modules so it's focusing in on the software that was built for AMD Rome and it does it automatically it adds this path to your module path um it finds the helmet configuration file for that makes those changes in the environment and then you're ready to go now what does that mean if I now do a module of veil of let's say open foam I should be seeing the open foam installation that's included in the easy repository now this feels a bit sluggish right because at first system FS says okay oh you need to you need to know what's in that module's all directory I'll have to download that metadata and and cache it locally otherwise I don't know what's there so that's why the first time it takes a while the second time it should be a lot quicker yeah it should be maybe not always um one thing I didn't do is we're still picking up stuff from that we have installed in in this case in my account on the system even if you empty your module palette before you will still see the things that here in the easy repository so right now we have three versions of open foam installed in there and those should work should work fine so that's the the setup that you do to get started to get your access to those modules and now let me let me do an all news of the stuff we have in my account so I'm only really interested in the easy things like this we have open foam we have grow Max I can load one of these and again this will make cpmfs pull in some stuff in the background so all these latency aspects these are annoying but you can limit those a lot by having a proper caching setup at your site right now it's not properly set up so that way it's a bit slow initially but it does work that gives us a bunch of modules and let's see if we're now looking which python we are using so the python command has changed because there was a python dependency loaded for God knows what here something um the python binary you're using is one that's coming from slash cpmfs the easy repository and the one that's specific to amds and two that is optimal for our current CPU so that all just works automatically so from a researcher point of view let's assume easy is available it's there all they need to do is somehow initialize their environment right now we have the source script and we can have better ways of doing that so they do this then they can start loading modules and start doing their software and they should work regardless of whether they're on an HPC cluster in the cloud on their laptop should just all work fine it also detects what you have if you're on a system let's say an Intel Cascade league so we only have optimized installations for Haswell and Skylake currently so if you check in here so in the Intel directory we have Haswell and Skylake if you're on a Cascade League system that means we're in trouble right because there's no exact match Arch spec is smart enough to say okay if you just take the Scarlet binaries you'll be doing pretty good so it knows what is compatible with what and it will take the the best possible match for your CPU even if there's no exact match so that should also work fine okay any questions on this yeah let's use the mic so when the users running this on a cluster they could do the sourcing in their batch job and pull everything that's right that's the better way yeah so if you do this up front and then you submit your job at slur which passes down the environment if your login nodes are different from the cluster you're submitting to you're in trouble yeah so it's better to do this from the job script itself yeah yeah yeah so you've followed a layered approach does that mean that you could remove the file system for example in future replace another one and there is that kind of thought process that yeah things will uh okay yeah so the file system layer we're deliberately not calling it cpmfs because that's just one way of Distributing like what you what you could do and we're planning to do this at least for archiving is you could say this whole directory so everything that includes this particular version of easy I'll throw it in a container image then I can take that container image jump in that container and Source from there so this is going to be a pretty big container image right because it's going to have AMD Intel arm power everything so maybe you want to zoom in a bit on something very specific or even specific installations from there you could throw those all in a container image and then you don't need cvmfs at all anymore so there's options there yeah if you don't like cpmfs you could sync everything to an NFS file system and just Mount that it will still work fine and I also saw that there was a branch that bypassed all the layers that went straight up is that because is that because of the yeah in the diagram the reframe part yeah the host OS you go all the way up bypassing all the layer Yeah by passing the compatibility and the four systems yeah yeah that's because for some things we still need stuff from The Host like slurm we're not going to give you slurm in here anywhere that doesn't make sense the GPU drivers need to come from the host because they're too much tied to the kernel your antiniban drivers need to come from the host they're too tight to your hardware and the girl so some things need to like leak in but it leaks in in a controlled way and we take we take that into account in here so when we configure our open API library for example we're doing this with libraries like ucx which which basically also detects what you have and just use what you what's provided by the OS and that works one thing I didn't mention I should have mentioned that is one question we often get like well wait this is a good idea but this is going to work in practice well yes because compute Canada has been doing this for five six years exactly this system this layered approach they used to use something different than Gen 2 but they have now also switched to Gen 2 because it's a better option they used to use NYX for the compat layer but this idea of this layered approach and using cpmfs that's basically what they do in Canada they have one software stack that's used on all the Canadian systems and they have a team managing that Central software stack that uses they have a mix of infinity band and omnipot interconnects they have a mix of Intel and AMD they're not playing with army ads but that's a detail in this setup that's really just another CPU you don't really care too much so it works absolutely fine yeah excellent thank you yeah another question there York a stupid question but there's some understanding if my operating system is quite out of date then but a user wants a latest greatest piece of software this helps me get around that yeah as long as you can get cdmfs running on it yeah that's enough that's really enough yeah and we we will be a bit careful here what we build because there is some things that check what kind of Kernel version you have kernel headers you have and take that into account that's something we'll need to keep an eye on as well so for example if if we're building something we can actively test for that we can we can test these installations in a center six VM that has a very old kernel just to make sure things still work yeah you can run our tests in there as well and that's something we plan to do yeah so we want to we want to um make what we support here as broad as we possibly can Mac OS is the annoying one because you cannot run a Linux binary on Mac OS that doesn't work so that creates a fork in that directory structure and for now we're not paying attention to that but gentle prefix works on Mac OS so we could we could build a special gentle prefix for Mac OS and now we're going doing good again we'll probably need to do it for every major Mac OS version which we don't have to do for Linux but yeah okay if we have to go through that pane we can and if all of this is automated that's not too big of an issue okay I see it seeping in slowly what what this could enable right so that's good um so let me continue uh we have a paper an easy paper an Open Access paper which explains in detail what we want to do with all of these layers mean how they work together how this thing is designed uh what kind of use cases this could enable and so on so this this paper was was written um it was published February 22. and by then we already had our pilot repository for for a while we've been playing with this so we knew technically quite well what we were getting into it's definitely not finished work um but the IDS are there one experiment we did with the 2021.12 version of easy which is the one I was showing we have a grow Max in there and someone ran as a regular users another sesame just in a regular user account was playing with grow Max coming from easy and comparing it with the grow Max they have installed in ulik which is optimized for their system and their interconnect and he was doing assimilation up to 16 000 core so pretty big I was seeing this kind of performance so the the dotted line is is ideal scaling the black dots are what he was getting with easy with the auto detected Zen 2 so this was an AMD ROM system um and the red is the performance he was getting with the optimized system installation so higher is better you see some of these black dots hovering above the red ones that basically means here we're getting better performance than the system one but that could be in the noise so I'm not going to say our installation is better that could be they're quite close to each other but at least we're in the same ballpark right so and that's already pretty impressive because this installation knows nothing about the system nothing about the interconnect it's just doing the the detection of the CPU open API does the detection of the interconnect and it scales and works fine and that's that's the main message here The Blue Notes that you see are the generic binaries that we also have in easy so you can force easy to use other binaries by just setting an environment variable so you can basically disable the detection and say run these binaries because I want to test this or see if it works so if you force it to use the generic binaries you're getting lower performance of course but it still scales so there is a gap here in terms of performance but it still works fine and this was a very important result that's why we put it in the paper this basically shows that the idea can work and you can scale quite good and again I'm not going to overstate this but this difference in performance here could be because we're using a newer glibc so I think this was still Center seven which means an old gdpc we were using a very New Delhi in our compound layer and that could give us a performance boost as well now again it's close enough that it could be in the noise but at least it's uh it's a same ballpark definitely all right so what's what's the current status of this whole idea um in the tutorial page this morning I wrote up a small history of the project it's um it basically started after having a meeting endowed with some Dutch universities that invited me there to talk about easy build because they had some crazy idea to work together on something big and they wanted to get some funding for this and they were figuring out what to do um and we had a bit of a brainstorm at the end of that meeting and the conclusion was basically let's try and do what the Canadians do but on a bigger scale like make it a community project make the European thing even maybe we can even get some European funding for this and and let's let's see if this works that was March 2020 it was a bit of a ruse also to have lots of beers together and have the Dutch people visiting Belgium and the other way around then the world changed a bit so we couldn't really travel much but we did use that time to work out the the proof of concept this pilot repository and we've been working on this um since then we set up a GitHub organization on easy called easy we've been doing monthly meetings monthly online meetings every month since basically April the 2020 to see okay what is the next step we should take how can we tackle the automation uh who's going to work on testing so basically getting ourselves organized writing the paper happened in in that Community as well and we applied for funding on the European level as well um so we ended up with a proof of concept setup where the central server is running in Groningen in the Netherlands and we have four stratum One servers um one gruningen one in Oslo in Norway one in Azure which is I think running on the east coast in one of the data centers a modern AWS which is running if I'm not mistaken in Ireland so they have four of these mirror servers that means we have a relatively robust Network as well and we just wanted to see okay what happens like one of these mirror servers dies is just fully transparently notice we also wanted to have a good set of software in there already for now it's only CPU because supporting GPU is a little bit more tricky and we had to figure it out but we have some big things in there like open foam like tensorflow like Romex you can load those modules and it should just work in terms of targets already good set of of CPU supported Intel AMD a couple of armed ones and we're still mucking about with power 9 but nobody's really interested in that and it's that's a pain to get that to work so we're going to stop doing this the interesting news here is also um we have pretty good contacts with with both Microsoft and Azure and they were basically throwing Cloud credits at us like whatever you need tell us if you want to build binaries and test stuff in different operating systems we'll basically do that for free and that helps us a lot we can because we can get very easy access and very quick access to a big variety of CPUs which is what we need to build all those boundaries um so that's been very very helpful so it's like if you look at what we're all combining here lots of Open Source packages so all these things basically existed already we're making we're puzzling them together and the Canadians showed us how to do that we're leveraging leveraging the clouds which gives us very easy access to all these different CPU architectures there's also changes in in openmpi and there's this companion libraries like ucx and lip fabric that do auto detection of what kind of fabric you have that's definitely an enabler for us as well so it's like everything is basically there to make this possible and make it work well and we're building that puzzle and making it happen uh that's basically what's going on um okay we'll do the Hands-On and demo at the end um once you have a good idea of what's going on now one thing we're now actively working on and thinking about is we want to make this a community project we won't to basically bring the community together and work together to get installations in there now how do you do that we don't want you to come up with the binary and say Here's a binary throw it in there right that doesn't from a trust and security point of view that's a bad idea and we also want to make sure that these things are actually working we can test them and so on so what we're doing is we're setting up a way that you can essentially send us a pull request and it's a bit easy build focused but it applies to other tools as well that somehow expresses okay I will I would like to have open phone version 10 added to easy so if you use this easy config file you should be able to build that binary for all these CPUs that you support could so they make a pull request and what we want to do I'll skip ahead here a little bit because there's a lot of technical stuff here that's not that interesting um yeah this is basically what we're going for is we want you to be able to open a pull request to our software layer that says Please add open form 10 with this tool chain or if you were saying okay that makes sense um our bot will start building that on all the different CPUs we have some in Azure some and AWS maybe some on-premise we'll use in a container for this so to isolate it from the host OS as much as we can once we have some tests we'll run some tests on those builds as well if that all works good we'll put those installations in tarball so it's all nicely in its own installation directory so we can make Dar balls to easily ship them to other places um those star balls get uploaded to an S3 bucket so like a place where we can collect stuff they get copied over to the central server before we add them to the easy repository we have another step and this is more to keep control of what's actually going in and this part is automated with the Chrome job as soon as a new turbo appears on the stratum zero a pull request is opened to our staging repository that basically says Ah this is a turbo these are the files in there does that make sense to add a human test okay makes sense so they they hit OK in that pull request if we want to we can have another bot here that really runs those tests and when we're running the test we could run them in the build container so the same build environment but also in a totally different container with a different operating system and it should still work so it has to pass all those tests as a dollar looks okay they'll be added to the easy Repository and again easy repository as just a cvmfs repository it's streaming that means if open form 10 gets ingested here it will automatically appear on all the clients that mount easy they don't need to do an update at all it's just streaming in like Netflix adds a new movie you don't need to update Netflix right it just appears that's exactly what happens with all these installations as well so we're now working on on building this whole Pipeline and automating this all and making sure this works we've been quite we've been quite successful at that already we have a bot that you can tell go ahead and build all these things and it reports back whether that worked or not uh but we're improving on this the testing step is now very light to almost non-existent but that's the next iteration so build and deploy is our biggest goal and the test part we can we can enhance later so that's really the explicit goal to make this and a community project that's more text which is basically explaining what I just did so that that Bots that I talked about is a GitHub app which means when you open a pull request um an event gets sent to some python code that can decide what to do and the event could be if the reviewer hits OK on that pull request that's an event and the bot would say ah that means I can start build start the build on all these things automatically so we're basically taking out the human as much as we can we do want to still people have approved things so we're not getting anything malicious in there but that's the idea so it's automatically building that there's some slides on this as well that explain it step by step so someone opens the pull request and our our goal there is to make this an easy stack file so this new experimental feature in easy build that just says it now actually looks a bit different but basically says I want open foam built with this tool chain and these two versions the reviewer said okay that looks good um and then easy book knows how to do the installation there's an approved review the bot says okay I'll submit some slurm jobs to build all these things on different types of CPUs that gives me turbos for all that software um the builds the bot says the builds worked now what do I do and the reviewer says okay let's go ahead and test this and make sure it works in another container right maybe we're building incentives and testing a Ubuntu that should work because of the compact layer if it doesn't we overlooked something um I do could have a reframe test for that as well then the bot says tests look okay to me now what do I do and the human says okay let's go let's get it in there it looks good so this this cycle is um is what we're now building and then when the bot gets okay it just can do all these uploads and ingestion um so you basically add it to the easy repository and then you're done then everybody can start running that software Goods like I mentioned um one of the goals of this initial collaboration was this is a very good idea but nobody has time right Square reels and round wheels and all that stuff so getting funding for this was very important it took us a while it's a lot of effort to start writing this idea from scratch um do you realize that getting funding for a service like thing is not easy it's not research um so what we did is we found some researchers that actually have ambitious ideas as well that we can help so they want to develop some software to do multi-scale modeling like batteries and helicopters and all these complex things where they need lots of software it's a headache to get all of that in place and and be able to migrate between systems are very easy so we are helping them they're like one of their one of our demo one of our use cases um so we're combining forces with with Scientists from the seacam Consortium together with people that are already active and easy we're joining forces there we have a scientific uh use case and we are going to help them in in achieving that use case so that's what the multi-x scale project is about we propose that to as a euro acpc center of excellence it got accepted and January first of this year the project actually started so we now have let's say more dedicated manpower to make easy uh possible so we're going Way Beyond the spiral repository the development of this bot has really sped up a lot in the last couple of months we've made very good progress there so we're slowly working our way to make easy more reliable and and go beyond a product Repository we have a website as well where we're starting to make noise about the things we do we had a kickoff meeting and so on so demo um one demo I will do is I already did the one on our HBC you get infrastructure what I will also do and let's see if this works because I didn't test this at all this morning but it should work um oh I love this what I'll do is I'll I'll create an empty Ubuntu VM in AWS and show you how quick it is or how quickly I can get access to easy on a totally empty system so a blank operating system so I'll fire up VM somewhere in here launch a new instance it's been a while since I've done that okay easy demo let's make it Ubuntu 2204 that makes sense let's make it a bit more interesting and go for arm um we should at least have some c6g I think let's go for eight ish CPUs 16 gigabytes of memory that should be enough for a demo I have a key in here and finding where that key is is going to be interesting but I'll figure it out and the rest is pretty standard I'll make sure it has enough disk so 30 gigs of disk and the rest should all be fine so I'll fire that up that should only take a couple of minutes and then we can check if we can actually get easy working um so that's basically the top one that I'm just setting up um there's nothing there cpms will not be there that's on the standard package so we'll have to install this first but we have a tiny script for that and then we can at least show on a single VM how quickly we can get access to this of course this requires admin privileges you need to be able to install cpmfs and configure it and mount that file system um if easy is already there you don't need to do that like I did on our cluster I was just a regular user there so I can just Source the internet script and hit and start running that scientific software there is another option I'll get back to these when the VM has spun up there is another option and that one you can actually try yourself on the prepared environment or if you have a system somewhere where you have either Singularity or obtainer you can try this yourself so this only shows you part of the instructions but if you click the link to our documentation this will show you the steps that you have to do so you have to pre-create some Library some directories you have to bind Mount some paths in there because these locations this needs to be writable for cvmfs that's where it's put it's putting the cache um also this place needs to be writable our making sure we have an empty home directory and then you can basically do a singularity shell to Shell into our container so the container we're providing you here our easy client container basically only has cvmfs in there it has no software like open foam or gromax um so that's just a way to get around not having admin access to install cpmfs our mounting the repository using the fuse Mount option of Singularity or obtain so that works absolutely fine if you copy paste these things these commands on any system already have Singularity or obtainer it should work and it should give you a shell like this where you can then check that cdmfs the easy cfmfs repository is mounted you can Source the script and try something okay let's see how our VM is doing and if I can get access to it it looks like it's running so well it's still initializing but it should be close I'll need to figure out where that key is I think I have it in here somewhere and I've kicked off a VM in U West so that must mean it's Easy E West must be this one I'll need this identities only and then our front part is always what is the username and you start I think it's Ubuntu yeah okay so we're getting access to our MTVM Ubuntu 2204 there's no model command here absolutely hopeless there's no easy builds uh yeah so we have an empty operating system so what now what I will do is clone the easy demo repository that we have on GitHub if you can just get clone this so all we need is git that should be easy to install in here we have some test scripts for grow Max open foam and so on there's also a script directory which has some installation scripts and this one is for Ubuntu scripts of course so this is all you need to do to install cvmfs and the config the easy configuration for cvmfs so this whole part is cvmfs this part is our tiny configuration package which just installs and configuration file for cvmfs that tells it about the easy repository very soon this will no longer be needed because the cpmfs people are asking us to include easy in their default configuration because they see the value in this as well so this part will disappear and this creates a tiny configuration file for cpmfs where here it says I'm not using anything special in terms of proxy cache or I'm basically directly connecting to the mirror server which is not ideal for latency but it's good enough and this says you're allowed to use 10 gigabytes of cache don't go beyond that so it's like a what's it called as soon as it's long enough ago that you you've pulled something in the cache you you know or need it will be kicked out so it recycles it has a name I forgot what it was so I have student rights in this VM I can just run the script it will pull in cvmfs which will pull in some dependencies as well so that's going to take a minute or two uh but once that's done um we can get access to the easy repository and start playing with the software that is there so you can imagine even a researchers could do this right if it's only running a single script so if they're spinning up a VM you could have a pre-configured VM that has this already installed so that's what the AWS and the Azure people that we are working with are considering you could give um pretty big TM images that are tagged with easy and that way all you need to have is cvmfs in there and everything is basically pulled in as you start using it so that makes it very attractive it's going to be a small VM that gives people access to lots of stuff so that's it now let's see this looks empty so that looks wrong but cpmfs does Auto mounts so if you actually know what is there and even less on that it will mount it now it's actually mounting the easy repository and from here it's exactly the same as before latest in its Bash uh this is some arm CPU I'm not sure what and it's a relatively recent month so it does a little gets a little bit confused but it's basically an arm graviton 2 at least that's what it detects and then it does the same thing as before does the multiple views on that which means we suddenly have open foam grow Max tensorflow all available open foam tensorflow grow Max the law looks good so that means we can start uh so now I'm in the tensorflow directly there's a run script here which basically does a module load and runs the python script so very very basic and this again when you run it it's like come on let's go right but it's even FS is pulling in all that stuff on the background not only the module file but we're firing a python so it has to download that python binary all the dependencies for that the python packages but you can see it takes a couple of seconds and it starts running tensorflow I think it's very difficult to make it easier to get tensorflow running than than this for now this is CPU only there's no GPU support yet but we're working on that and that will also be possible yeah that's very important of course and this you can try yourself you can you can in the prepared environment you should be able to just clone the easy demo repository and run one of these example Scripts and it will uh as long as you have set up your environment to use Easy with the source command it should be working fine if you have obtained around your local cluster you can you can do the same thing As Long as You Follow um the instructions here so this is let's say the technical details on how to set up your environment and do the singularity shell if you look into our documentation and the easy container part in here we actually have a wrapper script as well which does all that magic for you and drops you into a container that has easy available so as a scientist as long as they have the script they can just run the script and they can do single node stuff very easily any questions on that yes thank you yeah so you showed us the pipeline where you deploy automatically the the new software and and one of these steps which is you said that it's not mentioned here but it's basically testing and I'm assuming yeah that's the reframe tick mark it's there it's not we're not actively doing that yet but we will yeah so I guess that's functional testing you're referring to but uh we could also do performance testing in there yeah but what about kind of security aspects because you know if you're taking software from another kind of unknown built by that could also be done there yeah so this this um approval step here that can be human it just says okay right or you can have security scanners in here as well that first scan the source code that you're pulling in and gonna use for the installation yeah and there's lots of tools that do this already they're usually quite cheap if you're only scanning source code it's okay you could even after the build rerun the scanners again on the binaries for the check for watermarks and or fingerprints yeah that's definitely possible yeah I should we become more and more serious about this we're we're going to do that of course and this is a one-time cost right so this you only have to build those boundaries once thoroughly check them then it's okay but we will also also do actually even after stuff is already in the repository we'll do weekly retests to make sure everything keeps working because at some point in we bring back this guy um in the compatibility layer here there's a gleep c that we build ourselves tlpc also has security issues that pop up every now and then so we will have to update our TBC in here I will have to make sure that doesn't break anything in there because Philips is supposed to be dropping replacement drop and update in practice it's not always the case right so that's something we're very careful so what we can do is in the in the sandbox environment do the delete C update and then rerun our test suite and see if something is broken and that's that's exactly what we're going to do okay I have a couple more slides on what this could enable if it's um actually working and has a bit more software so there is stuff to play with there but let's say it's in the about 100 modules eventually when we have this automation in place we'll start installing everything that easy build has because why not right as long as it's open source we could include all of that in there and then you have a collection of let's say about three thousand software packages waiting for you to be used it's like a catalog and you can just start using them if you think you need them so that's the demo did I forget something here not really so please give it a try the easiest way to try it is with um with Singularity over obtainer you don't need anything else if you have obtained installed you can pull in our client container as long as your environment is set up correctly and you can do single node tests with what is in there and easy quite easily so this enables a couple of things that haven't really been possible before so that interesting things start to happen so when you give people a uniform software stack that works everywhere um there's like new opportunities that arise and we're discussing them also in our Open Access paper so uniform access means wide variety of systems you can run anywhere you can play around on your laptop if you're confident enough that your script works or your input files are real prepared you can jump to the bigger system with a lot less effort your operating system becomes mostly irrelevant we can leverage high-speed interconnects like we showed with the grow Max Benchmark and so on and we can actually prepare ourselves and that's part of the multi-xql project we can prepare ourselves from for the time that there will be risk 5 CPUs super computers in the future so that's something Euro the EU is betting on quite heavily they want to become um self-sustaining essentially so build their own processors and become independent from China and there are risk five is is one way that they will be able to do this so there's lots of research going on currently and a small part of the multi-x scale project is to see how difficult it is to start building binaries for these CPUs already we can do this in um with emulation so we can have an emulated VM where you're building stuff for risk 5 which is going to be quite slow on your building but once you have the binary it should run you can actually start building the software before the CPUs will be there because this is very standardized instruction set and very predictable you know all the nasty details that you need to know we can we can make this happen um there's lots of software like open API like python already works fine on risk 5. so it's it's starting to get there that it's become more realistic to to do this already even though there's no very capable CPUs yet um like I showed with auto detection of the CPUs and building for different generations of CPUs you can do this without compromising on performance so very different from what you do with the container image where you build one binary that works everywhere and even that is going to slowly start changing one binary that works everywhere works as long as you stick to Intel and AMD if you go to arm it's game over you have to rebuild your your container your container image and the same thing with risk 5 so you will at least have three contains images that you need to juggle around so that's that's sort of a dead end at least to me but in this collaborative software stack you can build for different generations of CPUs different families of gpus and you can either Auto detect or you can tell it which part of the stack it should be using so that's a very to me that's a very large contrast with generic binaries that you typically see in containers um it facilitates Cloud bursting as well so if people have a job and the queue is too long on your on-premise cluster and you have some credits in the cloud just throw your job into the cloud and as long as your data is there your software will be there waiting for you bonding this also enables is using um easy NCI environments that that to me is very interesting so some scientists are definitely running tests for their code like every time they change something in the code they have some test cases that they run over and over again something that's very painful is that that means they have to compile their software I have to make sure all their dependencies are there you have to make sure that the compilers that they want to play with are there all of that could be coming from Easy because why not all you do is mount a file system load modules and Things become magically available they are streamed in as they are needed so all you really need to install a cvmfs as a package and I showed you how quick that is and everything else is streamed in as is needed file by file so not huge packages of gigabytes that you need to download and also in other environments so in in Jenkins or GitHub actions this is quite typical but it also enables you to run those same tests on your laptop in the same software environment as well now we've actually done this um we actually do this for our demos so for the tensorflow demo I was running we're running those tests in in GitHub actions as well in a a workflow um to basically make sure that our demo script still works and doesn't break and all we need to do here and this is very small we've come up with our own um GitHub app or this is uh no this is a central GitHub action for mounting cpmfs repositories in this environment so you just tell it okay I want to use this I want to mount the easy pilot repository the configuration for that can be found at this URL so this action knows that it should install this package and do the amount of that and then you're done you Source our Inlet script and you start running your tests you load modules in those tests and everything just streams in as needed so it's very very quick when you do that we're doing this in the easy demo environment I'm not sure if this link will still work because they clean up stuff in there every now and then but you can see we're running our test for tensorflow open phone grow Max and bioconductor and we do it twice I think we do it once with CPU detection and once forcing it to use the generic binaries just to make sure both of those aspects work um so that's to test our own demo script we can imagine a scientific software developer developing your own code could write tests that just load a bunch of modules for the dependencies or load different modules for different generations of GCC that you want to test with and then make sure that that code keeps working with all of that so forget about figuring out which RPMs to install it could all be coming in from Easy so that's at least very different to what's currently possible that's another that's another example where we have our own dedicated action for making easy available in in GitHub actions and there you only need to give it the version that you want to use everything else it knows that's a bit more minimal but it works just as well and here for example reloading grow Max and we're checking the version works fine yeah that and that's that uh infection like I already mentioned this also facilitates HPC training so if you're giving a training session on open foam maybe you want to also explain them how to install open foam that's going to like be day one of the tutorial I'm not sure you want to spend your time on that if you want to focus on the actual science that they do you could say for this training we're going to use the easy environment because open form is installed in there we even have two or three variants of open foam installed in there so you can pick one and when they get home they can try it on their laptop as long as they mount easy they can jump to their cluster send an email to the system and then say this easy thing can you please make it available because it makes my life a lot easier and they can hit the ground running with the stuff that they learned and uh in the tutorial also for the trainers it's very easy they can set up a slime cluster in the cloud they mount easy and as long as the software that they need for the tutorial is there the work is done when the training is done you just throw away the cluster and you set it up again for your next training session so that that helps a lot compared to making all people that are attending the training get an account on that system and going through the security or whatever whatever Administration is needed to get their account just set up your own cluster in Azure or AWS and there's lots of tools for that like cluster in the cloud or Magic Castle and there's parallel cluster for AWS there's AC hub for Azure so there's there's lots of tools that allow you to do this quite easily and integrating easy in there is very very trivial like I showed it's like five or ten lines of Bash as long as you can figure out how you can make the the tool do that you're you're good to go this I also briefly mentioned already um we think this could be a step towards software developers as well so if someone was asking are people opening it was the final I think are people opening pull requests um or easy blocks um not really because easy build even though many people are using it it's quite Niche it's just one way of installing the software but if we have easy and we go talk to the gromac Developers for example and say look you could maybe add this to your documentation this is an easy way to get access to grow Max binaries that run anywhere and that are properly tested that also makes it a platform for them that's maybe more interesting than helping out some random build tool that that only a handful of people are using um so it could alleviate them from many questions on installation if someone isn't sure as long as they can figure out how to get easy uh working they can get grow Max running and then they should be happy and that means it's maybe more attractive for developers to actively help out um validating the installations that we provide they could figure out or they could help us figure out which tests we need to run to make sure it's all functional if we're doing a performance check they could tell us one nanosecond a day on an Intel scar Lake that seems a bit low right so something must be wrong there well we have really no idea we can say we get a one and we'll try and make sure we keep that one but if the developers say you should be getting five nanoseconds a day yeah that helps us as well and we can we can figure out what we did wrong so that getting that kind of feedback I think becomes more interesting from the developer point of view because you'll probably be helping a bigger um a bigger set of people of researchers as long as they use Easy also for the developers themselves like in the CI case um maybe they can get their dependencies their compilers that they want to play with from easy and that upstairs uh their development a bit as well another thing is portable workflow so especially bioinformaticians do a lot of this they string different tools together in a pipeline um and all the tools do something small but they're genomic data or whatever they are using that's all very good but if you need a thousand tools to get your research done that becomes a bit painful to get all those installations in place if they are a part of easy a big pane of that goes out the door and they can be running their snake bank or next flow or whatever um on top of easy that's very different from the the container approach that some of these tools take when you're running next to all you're pulling in containers which includes a whole bunch of binaries that you're probably never going to run right well with tvmfs only the stuff that you're actually using is being pulled in so we're downloading way way less things than you would be downloading full container images yeah all right so that sort of wraps up all the the content I've prepared there's our Open Access paper there's our website we're still doing a mailing list for easy which is only used to announce our monthly meetings really there's very very little activity there but we do have a very active slack Channel where people jump in all the time um and ask us some questions we have documentation which could definitely be improved but it's not that bad we're on GitHub we have a Twitter account we have a YouTube channel just like we do for easy build where we're posting any any talks we do or we had a community meeting last uh year in September in Amsterdam where we also explain easy from scratch and a bit slower paced than I did today but also talking a bit more in depth about the use cases talking a bit about um the Bots that we're building and so on and we have monthly meetings which are really open to anyone to join yeah that's what I have in terms of content um yeah I guess so I think I'm I'm pretty much done in terms of content so we can still answer some questions of course but yeah if you can I think that makes sense to see if as soon as they can start the coffee break it's okay yeah so I think the model is that there's essential repository of all the easy software and then there's all these mirrors yeah so I mean potentially if you have many many users you're going to need lots of neighbors so who's managing the mirrors who's owning the mirrors who's paying for the mirrors you know software mirrors for the mirrors yeah okay that's a good question um so from a naive point of view it looks like the more mirrors the better that's actually not true so there's like the cvmfs people tell us go for a dozen mirrors or so spread around the world but don't do many more than that because every time cvmfs starts or or need software it's going to check like oh what's my best option here which is the close it's just geolocation of the mirror so based on the IP address it figures out which one is the closest one it should be talking to to minimize the the latency and the delay in downloading stuff the more options it has to choose to the more things it needs to bring to see if they're still alive and it's actually going to slow things down so we're looking at the order of like a dozen or so mirrors um and what we envision is that the the let's say the maintainers of the project um also maintain the mirrors at least the public mirrors because then if something goes wrong with the mirror if it's somehow not syncing anymore that would be bad because that means new software is not streaming in there if you're coming in through that so we're looking at a at like say a core team of how many maintainers we will need um that also keep an eye on those mirrors set up monitoring and make sure everything works as expected so we think like let's say 10 dozen mirrors a couple in the US several in Europe one in Australia one in Asia to be close to those people if they want to use it as well but the focus is going to be on Euro probably um and that that should be enough yeah so there will be a team managing that there's um as a part of the multi-xcal project there's the idea of setting up a rotation among five or six of the partners that are involved in the technical parts of the of the project so they will be keeping an eye on on the mirrors they also we're also set up a support portal for easy well people can ask questions or say I tried running this and it didn't work and then we can either figure out this is the problem with the system is the problem with easy do we need to talk to the software developers like pull in the right people and try to figure out what's going on and make it possible um one thing I didn't mention which is also interesting the ones I'm showing here are like the public mirrors so which are part of the easy Network let's say um if you install our easy configuration packets those are the mirrors that cfmfs will know about you can install your own mirror server as well you don't have to ask anyone it's all documented on how to do this you could put it next to your HPC cluster in the same network so you're reducing the latency and that will automatically sync with the public servers of the network so that's like your own mirror only for your use case and that will not be serving anyone else so as long as you do that and you maintain your own absolutely fine yeah so that's just one or maybe two more servers that you add to the list and the configuration and then cvmfs will also bring them but not all the other ones that people have at their own cluster so you can in your network you can and that helps as well because maybe your your cluster nodes don't have access to the to the web so they could be offline as long as your your mirror server is in the HPC Network and you can ping that that's actually okay that's all you need because your mirror server has a full copy of everything another thing a full copy sounds scary that sounds huge right maybe hundreds of terabytes yeah not at all so what compute Canada does is they have thousands of software packages installed and last time I checked they were still under a terabyte of disk space with cvmfs because cpmfs does deduplication it will never store the same binary twice so it has object storage in the back end it's like git where all these hashes and so it has only the every file once on the on the disk and it also does compression on disk and and when sending the data over the network so compute Canada can can serve all the software all the Scientifics of the researchers need across the whole of Canada with a terabyte of disk so yeah setting up a mirror server is not scary in terms of resources you need it could be a two or four core VM with one or two terabyte of SSD disk then you're good oh yeah coffee yeah okay well I don't need no no I guess no yeah I don't think we need to ask for more I'm happy to take more questions but other than that it's sort of a wrap right um so it's it's up to you if you can uh if you're ready for the weekend I'm totally fine with that I'm also happy to answer any any questions I'll show you something more specific more technical uh up to you so yeah if you can I would say take a look play around with it maybe on a VM maybe through the container and just see if it doesn't work for you we'd really like to hear from you because now now is the time that's also clear in our documentation so we're whenever we talk about using easy there's a big fat warning here right so this is not ready for production our pilot setup is you can play with it and there's been very little instances where it doesn't work but please don't make this the only software that you provide on your cluster today that we don't have strong guarantees there we're going to move to a different domain we're going to make it easy.io that becomes a bit more neutral not HTTP specific because we're going to rebuild our ccmfs network with dedicated Hardware that's securely set up with ubikey so only a couple of people can can access it and stuff like this so and when we're going through that exercise now once we do that and once we have the Automation in place with the bot to set up that whole contribution workflow then we'll start saying okay now we think you can start relying on this and assume this will work today not yet but I think by the end of this year we'll be a lot closer to that if not already there so that's an explicit goal and the first year of the multi-xql project is making easy ready for production which means it's stable it's properly set up and from that point almost start expanding with more and more software testing uh making noise about it attracting attention getting developers involved and so on okay if there's no more questions I'll I'll stop the recording yeah all right yeah thank you