nf-core/bytesize: nf-meta

Hello everyone. Maxim is the by size. Today Julian Julian Frisch will be presenting us NF meta is working at Cubic. And if like [clears throat] me you're very interesting into like workflow chaining and you really hope to see my dream of killing Sarek. I think this is the talk that we were all waiting for. So as usual you'll be able to unmute yourself at the end of the talk to ask your question. You can also ask them on Slack later on. And so now over to you Julian and I will give you the spot. >> Yes, thanks for the introduction. Welcome everyone to this bite size. Today's topic as discussed will be NF meta work where I'm the lead developer on that's been cooking for the last month. And yes, I'm quite happy to show you what it's all about and give you a brief overview here. First let's get ourselves up to speed. First of all as I meant this is a bit of an ongoing work still and it's become quite stable in in in the format it is proposing but technically we're still in a better version so everything you hear might change but the best place to always check is to go on the Git and the pip page and to check the docs. And also we're getting close to a first release. In the near future. So first of all sorry for that, let's recap what we mean by pipeline chaining. I think this might be clear for all of the people that are interested in this talk and are here but just in general really simply we're trying to run run several instances of pipelines of distinct workflows that are from NF core for example and we basically want to combine these pipelines into something bigger where the output of the first pipeline for example serves as the input to the next and so on. And by that we can achieve some analysis that is more broad and covers generates some deeper insight than any single pipeline could. And in this example for example we're trying to chain MHC quant and HLA typing and epitope prediction for some peptide analysis. In general also why do we need this? There already are sorry like a lot of awesome really cool pipelines developed by really devoted developers here at NF-core and these are already reproducible in itself. But as I just showed with this first initial example maybe you always want to run some chain of pipelines together and it makes sense for you that way. For example the example of RNA seq and differential abundance has often been quoted in this in this position. Or maybe you you're writing a paper and you want to be able to run or you need to run five pipelines and then some auxiliary scripts to produce some some output and you want your readers to be able to reproduce it exactly and and that's all something that's maybe not covered and that can be a pain as all of us know. And maybe you might want to do some things that Nextflow cannot do yet and we're getting to that at the end. And um also very very the current thing that people run is so maybe you want to give a lot of your work or or try to organize your work or build workflows with an agent and have some help there. But then you might need some some harnessing for what it's about to do without spending too many tokens. And you also might want some provenance of what was done and maybe be able to be be able [clears throat] to even rerun that provenance. And after your agent created something. And and so the goal for NFMeta in general is like we want to avoid the this for applications this untracked I talk scripting. We want to reuse the beautiful patterns that we have as building blocks. And we try to not make pipelines more complicated than they are. They are already sometimes can be complicated but still try to figure out ways to combine them. And this is where NFMeta comes in. Our new tool like built in Python and it's shipped and it's installable via UV or pip. And it introduces at its core um a YAML file format that might remind some of you of GitHub Actions or something where you declare um uh things that going on in an order and you define uh where stuff is coming from. And uh as I just mentioned over here you can see like uh installable via UV um so it's quite simple to set up. And then there's >> [snorts] >> um these are the core functionalities already at a glance. And at uh a more deeper glance in general there is a representation there's three things that this entails three uh base units in sort of say it's core elements. Um the most important thing as I mentioned is it introduces a YAML file format. What does it mean? It means like it uses YAML which is already well-defined format but we have uh certain keys um that make um a YAML file a pipeline chain usable with NFMeta. And um this this allows us to have a single artifact that describes the larger logic as we saw in the introductory slides of um running many NFCore pipelines or non-NFCore pipelines. And there's also already a schema that knows this representation and can be used in the editor. Of course I mentioned it's a Python tool there's also a CLI component to this for for validation and you you even access the internal representation that we have, which is in itself a graph, via um via Python and via its API. The second pillar, you might say, or core component of this is obviously runner. We have a representation, we want to execute it. And we are able to execute these pipelines again via a CLI command. And currently there is one shipped with NF-core that's wrapping everything in in in Python. But um and and perhaps again nextflow runs. But it's extensible via plugins. And there will be more details on this in the docs. Because it's just just a rough rundown of of of what is capable here, but um you will be able to and we already planning maybe to build runners that are fully in nextflow or um when when the when the new nextflow language version drops or you could think of having a runner that that ships everything to platform based on your YAML artifact. So that's achievable via plugins. And then the third pillar is the editor that we also shipping here. It's browser-based and it's interactive. It's using a popular graph library to to give you that interactivity and the nice touch and feel. And the um the edit and it has neat features like an edit history and helps you with your params and again it allows the easy access to the validation. Now this is rather abstract and I want to um I want to create briefly, sorry for the ordering here. I want to briefly give you an example of what this looks like and what this is best representation is. And we'll have a deeper look in a second then. Um So in general I already quoted this popular fetch and digest to differentiate abundance example. And just for those who may not know, we were trying to combine a few different things here that are not itself covered by one pipeline. We obtaining RNA sequencing reads in the first step. We are trying to run uh quality controlling filtering steps and then also quantification of those of those reads which gives us a nice tables as we said output for example. And then we try to run some statistical tests in the end that are already also covered in a differentially balanced pipeline. And just to show you how this looks as a format as a first hand representation in the end of meta format would be this 40 line YAML. And you can kind of already see and go into a bit more detail in a second, but this there's these these these keys that describe that are described by this representation. And the most important one here is the workflows keys. You'll see it's keyed by IDs itself and you will recognize these names here and of and of course NGS and of course RNA-Seq. And then also differentially abundance. And you'll see it's tied to versions here which is really key for the reproducibility I mentioned and it's parameterizing everything in a way. And um and this is already um this is already like tying what what everything you need for this run together into one artifact. This again is then version controllable for example if you want to someone else to be able to reproduce exactly what you have here. You will also see one thing that's this brown node in this case which is just indicating that it's a non nf-core workflow. And in this case I called it the DAP because it's serving as a bit of kind of glue code between these two steps that allows translating or adding some some columns to to a CSV here and to make to process to prepare some input for the final step. Now I already teased you with this demo and I'll try to quickly go into my terminal here. I hope you can still see this and try to um uh show you this uh this is part of the showcases repository that um that I um that I will show in a second again and where this example is already also located. And in this environment I already have kind of meta installed. Uh you can see it should be the most recent version. Very perfect. And and then I also mentioned already that it's um we have this validation functionality for example. If we if we if we take one step actually back and we take this out, we can see there's already one of these um YAML files, I call it meta config YAML, um uh in here. That's that's the showcase. And um we can as a first thing we could for example um call a linting step here and run uh validate. And it will quickly uh show us uh the most important thing is that the config is valid. And the second thing is already a an artifact an artifact of what I mentioned that is um this is a beta and we kind of shifted some things. But you will also see that this is kind of nicely handled and it's uh taken care of you for you. Um so it seems this uh this this config is valid and the next um most logical thing is to try to open it in an in the editor. So I'll just go ahead and do this and it should bring us um into our browser. Now we're in my browser, just so you know, and um we see this is exactly the example from the slides I just show showed um and you can see that uh it shows the same flow here. Um and we can go into into details on some of these steps by double clicking. We can we can open it which opens them in the right here and it shows us the parametrization here and yeah. And we can also see that the the this brown step here it points to some it's it's an analytic core pipeline and um it points us to this this um uh >> [clears throat] >> this repository that's hosted in our organization in the biomedical data science um um, repo organization. Yes. And so, and yes, and if you want to edit those, you will be dropped into you can, for example, also think about um editing a new step or editing something in here, and you will be able to let's maybe cancel this one and edit a new one. Um you'll be able to select from from what we have in NF-Core, for example, and it it allows you to um to, for example, I mean, it doesn't make much biological sense, but for just for the for the sake of this demonstration, you'll be able to select from the from known pipelines here and be able to select uh pipeline versions. You'll be able to select whole param files if you don't want to use this and config files, but they're not really necessary here. You um you'll be able to resurface everything that you need for for running Nextflow in the in the CLI. That's something that let's just try to practice. Um And let's try to save these changes, and we should see that um we get some errors in here or the Well, if this is not the presentation. [clears throat] Anyway. Something that not service, but let's just maybe try this again, and I'm really sorry. Mhm. Let's try again. You don't need param files, but we want to And we want to maybe uh We also would need an input file, and we have to provide this um in our in our um in our in our uh in our directory, but uh we can see that this is how we we we we create uh could create a new um a a a um a note here for running, and then connecting it to another workflow is as simple as dragging uh an arrow like the the connector here and edge, which is created. And I mean and then um since the the magic of this is always in in uh extracting the correct files from the the previous uh from the previous step that we want to uh that you want to get it from, we can we are able to select um to use this dollar notation here to define references, which is really neat, and it's already filled it in with the correct um um ID, and it subsets to params, which is the only it's just a limitation right now that it's only using uh variables from the param section, but for example, we can say uh we want to reference the out dir here. Nice. And then get the sample sheet, which uh which French NGS produces. And you can save this. And you'll be you will you will see that this basically how how it's done here. Um Yes. All right, but let's uh stop this for now. We can we can save this, but I mean this was not really uh a meaningful addition to this to this pipeline, since this was um this has nothing to do with the uh sequencing analysis we have done before, so we just remove it by uh using the um rebar tool. And um we'll then save this again and go back to to our console. And the last step here, of course, is then to to um run this. And um let's first inspect what we can do here. There is uh different runners that you can use. Currently, it's shipping with one that's this this Python runner, um which is uh basically, as I mentioned, a wrapper around Nextflow built in Python, so it's uh accessible from the CLI. But, as I also mentioned, and this will uh be in the docs later, but there is um with the class documentation, there is a way to um extend this these runners and to have your um own runners be dropped in here. I already mentioned you could be able to you could you could think of um using uh or building a Seqera platform runner here, for example, that ships all these steps to Seqera, or something similar. And uh as as the next the Nextflow language version already plans uh these um foreign uh imports of of workflows, this is something that could be a become a new runner as well. So, this is just a minimal runner that that enables um us to to um uh run these. And um running it would be as simple as um using run, and then we select the Python runner, and select our meta config. Um And before running this, something neat about this runner is that it's it also uh surfaces um uh right now only some, but um this is uh subject to to extension, and I'm thinking about some more general form, but it's surfacing some important uh Nextflow run um uh options that are essential and that would um you wouldn't want in your meta config, in your in your chain um representation, um as they are really specific to your single run. So I will for example the run profile docker or something. And I will only show you briefly how it looks like. You will then be since Um And then you should be able to use this. And we are not since the validation fails [clears throat] for It seems in this environment there's a wrong version installed. Sorry, um I don't know if this and we run this. It's and I think we have we have a limitation with our with our next version. Okay, that's uh Um let's just quickly try to circumvent this by going into our I this is actually also a good time for me to show you the um how the representation looks like. And I would like to um So here we have the same representation again and it just complained about it next version error which is defined since we set the global next version as as pipeline developers you know that sometimes the with new versions things new constructor introduced and things might break. So this is something that's really required for the reproducibility of the whole meta chain um and so we are able to fix fix to a specific version over here. And while this will break the um the actual this will not work for the versions of the parameters specified. This can still just see how how this would work. Um and I'll make sure to cancel this um and we will see that kind kind of we have these it shows us steps that it runs through and um it uh then creates a nextflow command for for each of these steps that's printed here for for better visibility of what's going on. Um and what create um um also a cache of what what's happened locally here that summarizes the parameters that we use and is able to be used for rerunning. All right. Those are the the core things about this and let's also we've seen um this example right there in a bit more depth um and um there is there is a small program more showcases that we want to build for in the near future. One of them um um is is almost done. It's about um combining uh different arms of um um immuno immunopeptidomics analysis starting from both pep raw peptides and as well as TCR sequences. So, sequencing and and and protein analysis and [clears throat] um it will combine it into um uh a by combined binding prediction for those tumor antigens. But, this is only a teaser of what's what's what's coming and that brings us to the next things that are coming. I would already mention like this immunopeptidomics work right now. Um and something that we also working on is to um ex- use this for for imaging and we want to especially highlight there how we can use this um outside the CLI and and using the Python API directly which enables us to do things we're not able to do in nextflow. That's what I said in the beginning as well. Um one of those things is for example the ability to uh run training loops where we want to maybe optimize over something and that's something that's just intrinsically and for good reason right now I'm not not something we want to do. Yes. And the most important thing of what's planned though is um that there's docs coming and they also will launch launch tomorrow morning. So keep your eyes open for those. And and watch out for those. Um they will we will explain there in more detail how the whole plugin logic works, how you can extend it, and also um give you more some examples of what to do. Some other general things is um um we're planning work on the editor again. Uh For example, some some better parameter reviews and and highlighting of input files that are uh fixed and needed for the uh for the meta program run. And then also I mentioned the using the API directly. I also mentioned the new Nextflow language features. So there will be more output feature formats. Right now it's only YAML and my idea would be to to have it as a as a core representation but I'd be able to translate to um to an older Nextflow representation uh such as I had at some point introduced with NF Cascade and also to um for example Jupiter outputs or whatever we want to see there. Um and also something we've seen with the validation is that the um um the the I glossed over it in the demo a bit but they they it's based right now heavily on the um you saw that one pipeline was missing this but it's based heavily on the on the NF core metadata that we uh curate and that the developers provide. And um while this can be generated nicely now with with tooling and also um with the help of gigantic tools and it does require upkeep and keeping it in sync with the pipeline code. So, this is something that could improve with static typing and this is something we want to also move towards there. And with this I thank you already for the attention and want to specially specially thank all the beta testers and feedback givers and initial users that are already trying to apply some of this um in in in real use. Over here down here are the are the most important links for the showcases that we mentioned and the and the um PyPI page. Thank you very much. >> Thank you so much. That was very very very interesting. Uh I think other people will have some question. I don't see anything in the chat yet, but I'm really hoping for some question. Uh Beatrice, let's go first. >> Uh hey, I do have some questions. Thank you. This this was really interesting and the the visualization tools are just cherries on top. So, thanks for showing that. Uh but I have a question more on the technical side of the sequentiality of the runs. So, um does does NF-meta ensure that each step is run strictly sequential? Uh so, for example, in the example that you showed, the step that you added was dependent on an an input file that was the output of the previous step. So, how does this chaining works? Is is the directory watched so that as soon as that file is created, the second step can start or how does it work? >> Really good question. Thank you very much and that's maybe something I glossed over a bit, but let's look at exactly again. There are I mean you just picture it a bit blurry, but there are these are arrows here the edges. So they are directed and so internally this is a graph and also in the file it's already a graph. We we have these declarations, but then there's this important from keyword here and it shows this this there is something coming from this node and this already even though it's really a slim format I would say, but this already finds a directed graph. And this is what's used for the execution as well. So there is definitely there is strict order to how these are run and this is kind of the the whole point of it I would say. So I fetch in this example here fetch and js would always be run first and only then would RNA seek we start. And yeah. I hope this answers it yes or >> Yes, it did. I do have more questions. So should I should I ask a follow up one? >> Go for it. >> That's great. Yeah, cuz this is exactly something that I struggled with because if you work with channels in nextflow, even if the channel's not populated, it technically exists. So it's if the if an output channel is then provided as an input, then the second step can start with an empty channel. Which I struggled with. So this fixes that. But then on the reverse of the question, let's say that you want to optimize in time and you would like to start the second step before the first step has fully finished. Let's say that you want to use just an intermediary file of the pipeline and you don't need all the files to be there to start the second one. Is it possible to run things in in this case not exactly in parallel because they wouldn't start in parallel but then at some point they would run in parallel. Would it be possible with this implementation? >> With this implementation it depends on the pipelines. It depends on if you're able to already stop or I mean one thing you're already able to do if you have this definition you can define a from and to. So if you already have intermediate files from this and this already read at some point you can try to resume and and say start from here and go to here. And you can also try this without resume but it is assumes that there's files there obviously it will fail otherwise. And for the for the stopping early I mean if in RNA-Seq you're already interested only interested in the filtering step or something and don't want to do the mapping that that is something that RNA-Seq would need to parameterize right now. And so it would need it would wait until they they have run. So there is yes. [snorts] Um Yeah. So you would run until the would run the whole pipeline and then go to the next step. This the fact you can run stuff in parallel in in in principle and that's where the run plugin runners play into again. I'm just not I wasn't willing so far to commit to that because it's just just for for for like what we're adding here we're not trying to be a workflow engine a full we're just trying to be a this descriptive layer on top of existing workflow languages for now nextflow. Um and we're trying to you you have them do the um do the orchestration and do the and do the parallelization and do the and do the and and and and and and we're just connecting the the existing steps. So the a bit of blurring but to answer like if it's if the pipeline allows you can do early stopping otherwise it runs it waits until the command use finished and picks up with the next step. Yeah. >> Uh we had a interesting question on the chat by Adam. Can you execute code in between next flow runs? >> Yes. Um and no. The thing is right now it's So right now in this this example, unfortunately, doesn't cover it because it's quite new thing, but right now what you run are only nodes and and nodes are either next flow pipelines defined by this or they are modules given that there's now a next flow module run. And with a disclaimer that modules for for now only work as leaf nodes as their output formats are not quite nicely trackable. But the idea so far was that you have um you can represent everything as as a next flow script itself. And in this case it's this is like this this orange colored node, but this is the Excel itself just a pipeline. And if you want to have a look this um this this is just a one um I'll open this in a sec, but but this is just a like couple of lines of of next flow main script and nothing extra. Yes. And um yes, you can kind of uh represent everything as this as this and and you can also with a module running directly, you can also wrap any arbitrary code into a container and and run it as a module. Um we strictly don't want like code in here directly as it's like loosening the whole reproducibility and it's like introducing issues. Um so everything is kind of wrapped into into a workflow right now. Yeah. Or into a module. But that being said, like I mentioned, I want other output formats than this as well. And one of the output formats that some people have already thought about was something that you can can interact with Python, for example. And so it's something that's not coming in the next weeks, but maybe in in a bit more time could be the uh to have this there's already the Python API, but to have this directly output into some kind of Jupiter notebook or something for example, or where you can then between the notes do some some some cool stuff and and publish the whole thing and and maybe have it create the plots you want for your publication or something else, yeah. So all code is run inside nodes and nodes are modules are next flow right now, but this yeah. >> Mhm. Sounds good. Yeah, I had similar question to Beatrice about like being able to run stuff in parallel. Uh Simon had the comment, but I think it's we can keep it like aside for the moment. Uh I think we're good for today for the recording, so I will stop recording now and that way we can continue talking later. So uh let's keep in touch. I'm just stopping the recording. Uh thank you everyone for assisting that and see you next week. >> See you next week. Thank you. >> Thank you so much.