Video summary
The video features Nicolas from the Center for Medical Genetics Ghent presenting a guide on implementing parameter typing within Nextflow's strict syntax parser. He explains that while older parsers treated all parameters as strings regardless of their intended data type, the new strict syntax requires explicit declarations to ensure correct types such as integers or booleans are recognized. To resolve this transition without manual errors in every workflow file, Nicolas introduces a solution where users can define parameter blocks directly above or below the main workflow block. This approach allows developers to specify the exact name and expected data type for each parameter, along with optional default values, ensuring that the pipeline runs correctly under strict mode by automatically handling necessary conversions like wrapping strings into files using Nextflow's built-in functions.
The presentation then shifts to a practical demonstration of converting an existing NF-Core demo pipeline to utilize these new typing features. Nicolas walks through the process of generating static type definitions from JSON schema files, highlighting important edge cases such as distinguishing between file paths and simple strings for output directories that may not yet exist at runtime. During this live conversion attempt, he encounters common issues like stack overflow errors caused by outdated subworkflows or incorrect Nextflow versions, emphasizing the necessity of updating to version 26.04.0 or higher where these features are fully stable. He also addresses complications with nested parameters in configuration files, showing how converting them into map structures resolves recognition failures and ensures that optional values like logos or quality control configs do not break the pipeline if missing.
In addition to technical fixes, Nicolas discusses best practices for managing parameter definitions across different parts of a Nextflow project, specifically addressing redundancy between `main.nf` and configuration files. He advises keeping parameters in the main block unless they are strictly needed as defaults elsewhere or used within closures that resolve at runtime, which helps streamline maintenance. The session concludes with insights into future developments, noting that while current tooling like linting may still flag errors due to incomplete support for typed schemas, these issues are expected to be resolved soon. Nicolas encourages the audience to adopt this new syntax immediately despite minor hiccups during migration and directs them to upcoming blog posts and Slack channels for further assistance with conversions or troubleshooting specific pipeline behaviors related to path handling and schema validation.
Read the full video transcript
Uh, hello everyone. Maxim here, hosting
bite size. Today, Nicolas
uh, working at
Sorry, where are you working already?
>> At CMGG in Ghent.
>> Yeah, but like what does CMGG mean?
>> Center for Medical Genetics Ghent.
>> Okay, that's better. Okay, so Nicolas
here working at the CMGG.
Uh, he will present
balance
typing with the new nextflow syntax. And
I know for me for sure this is something
that I encountered like recently because
I I had parameters that
was a number and is now a string and I
was all like, "What do I need to do to
fix that?" And so I was really hoping
that Nicolas will help us like to answer
that question about that. And as usual,
you'll be able to unmute yourself at the
end of the talk to ask your question,
but you can also ask them on Slack later
on. So, now
to you, Nicolas, and thanks thanks again
for presenting that.
>> Okay, thank you for the introduction.
Uh, okay.
Uh, so first of all, I'm going to go a
little bit over the problem. Uh, so
I have a little pipeline here set up
with some parameters and it outputs the
parameter
value with the type.
So, for example, we have here with the
syntax parser version one,
I just give it a couple of parameters
like this. And as you can see, the first
one is a string, the second one is an
integer, and the third one is a boolean.
But at the second syntax parser parser,
so the strict one strict one,
this doesn't happen anymore and
everything is a string.
Yeah, you probably can see that this can
cause some issues in pipelines. But
luckily, they provided a solution to do
this, which is parameter types. As you
can see in this block here,
and I'll just comment these two.
You can just specify a params block now
in above or below the workflow uh, with
the name of the parameter, the type of
the parameter, and then if default value
if it's needed.
If I were to run the pipeline again with
the strict syntax parser,
then we should see that everything has
the correct type again. So,
string, integer, boolean.
If I for example wanted to make the age
also a string,
then I will just
change it there and then we will also
see it as a string value.
So, that's the easy explanation of how
to fix it.
Uh but there are of course a little bit
of as as usual, there are a lot a lot of
edge cases in these situations.
So, for this I will also go over the
conversion of the NF-Core demo pipeline
and you can see the conversion done on a
real pipeline.
So,
here it is.
We have the demo pipeline. Um
it's not really important what the
pipeline does for this case, so I'll
just go ahead and convert everything
already.
Um so, it's pretty easy to do the first
conversion so to make the parameters
block if you have an Excel schema JSON
file in your pipeline.
You can do this in VS Code by pressing
control shift B or control command B for
the Mac users.
Then you can
select this convert script script to
static types option if you have if you
are in the main.nf file.
So, if I do this,
then you will see that the parameters
block is created
with the name of the parameters and the
expected type derived from the
parameter schema.json file.
So, we'll of course have to check if
everything is correct. For example, for
the inputs, we say that we see that it's
a string, but we maybe would
want this to be path for example because
it's a file.
For the output directory, we would also
say let's make it a path, but path
type checking also doesn't exist check
Which the output directory does not
always exist.
So, for this case we will would like to
keep it as a string and convert it to a
file using the file function in
Nextflow.
So, if we set a path type for a
parameter, for example, then the file
function is automatically applied on the
parameter.
So, then
the email should also be a string, but
it can't be set type to I'll just
quickly go over this. So, the faster
should also be a path.
Uh,
this looks good.
This also looks good. These are all just
string values, boolean values, which are
also fine.
Um,
so, then we also have the multi QC
config here. But, may this is an
optional value, so we'll also put a
question mark behind this to make this
optional. Otherwise, Nextflow will
complain if the file has not been given
to the pipeline.
So, same thing for the multi QC logo.
Yeah, this is a bit of tedious work,
but it's important, of course, that it's
done correctly.
Um, then we have this.
So,
everything should be fine right now.
Maybe we should first make some others
parameters here optional, too.
Otherwise, it will fail. You should keep
genome required because it's a required
parameter,
and so on. So, let's try to run the
pipeline now.
Hopefully, something will break in a
live demo. I'm counting on this.
Um,
let's make it up to 10 to make it
quicker.
So, fingers crossed.
So, faster file isn't being given.
We
would expect it to be given by the
test.config file.
And it is not given. Okay, so that's
just an issue with the pipeline, I
guess.
Um
let's just give it a random faster file.
That
it doesn't have to be a real file
because, yeah, of course, the
uh
it's a step test, so we don't actually
analyze any data.
Then you see the stack overflow error.
This is a common error which we'll
encounter if you convert your parameter
to any type, especially uh if you did
the path type checking here.
Uh this is an issue in uh certain
subworkflow and in NF schema, so you
need to update those those two things,
and it should work again.
So, if you go to the next load of config
file,
go to the bottom, you should check that
NF schema is at least 2.7.2, which is
already the case in this pipeline, so
it's fine.
So, then we just need to update
the
So, with the
NF core tooling subworkflow update utils
nextflow
pipeline, it's this one that is breaking
uh with the stack overflow error in this
case.
So, let's wait for it to update.
Should all be fine, and then it should
hopefully all work.
And as you can see, the pipeline starts
running.
So,
it's actually pretty easy to convert
your pipeline to parameter types. So, I
advise anyone to really do it. But the
real caveat and for this is that you
also need to set the minimum nextflow
version
would at least be 26.04.0
because this is a new feature in uh this
version, And it's not really working at
25.10. It's It's already in that
version, but it's still broken and has
some weird quirks. So, I'd advise you to
to set the minimum version to 26.04.0.
And then everything should start working
again.
The nice thing of having parameter types
inside of the workflow block, of course,
is that you can also remove a lot of
these.
Should also just make sure that the
that the parameters that aren't used to
set the defaults of other parameters or
aren't used in config files
um are removed because everything that
is used in config files or for the
defaults of other parameters should
still be in here
uh cuz they need to be set at that
point. So, for example, the genome file
genome value we used to set the default
of other parameters like the fasta file,
for example. So, this still needs to be
in here because it needs to be set.
So, we can then just
remove all these multi species
things.
Uh
skip trimming is also
okay.
These are all used in configuration.
These two and the validate parameters
isn't used.
You should just check this for your own
pipeline, of course.
It doesn't hurt if you leave something
in by accident here. Uh
the other way around this. So, but your
pipeline will fail in that case.
Um all right. So, let's try this again.
Then we just need to update two more
little things and I think we're done
with the conversion of the demo
pipeline.
So, let's see if it still works. It does
still work.
So, um
next up we also need to update the
iGenomes.config file because as you can
see this uses some nested parameters.
So, this is params.genome.grch37.fasta.
Um but nested parameters don't work
anymore with the parameter types because
they aren't recognized as a special type
for example.
So, we just need to convert this to a
map for example. So, if I make
params.genomes
here
and set this to equal
a map of values.
We can just convert it using some
search and replace options like this.
Just try to quickly do this.
All right. That is should also be
updated.
I've done it a couple of times already.
But, still something is broke. So, there
should also be a comma here for for all
the values, but I won't do it again
right now.
So, it just needs to be a map structure
like this to actually start working
again.
This is yeah, just because the types
aren't correctly set. So, the parameter
will be null in this case and
I don't really know why it's happening,
but converting it to a map fixes the
issue.
And then last but not least, the inner
core pipelines lint will also start
failing in this case.
Because yeah, parameter types aren't
officially supported by the inner core
tooling yet. Yeah, of course I just
broke this file right now. So,
everything is broken here, but doesn't
matter too much right now.
We just need to
exclude some things from the linting
here.
But, we type linting
or lint, I don't by heart, actually.
Next though,
config,
set it to false, and then the same thing
for schema errors.
These two will fail if you convert the
parameter types because it doesn't uh in
the code tooling doesn't account for it
yet, but it should hopefully be added
sometime in the near future. So, keep
your eye on it.
Of course, see if it still fails before
doing the conversion.
Okay, and I think we have a semi-working
pipeline now except for the iGenomes
thing here,
which I uh
don't want to fix right now because I
will just make a mess of it.
Uh so, that's it. Thank you for
listening. Uh um Any questions?
>> Oh, [snorts] no, that was super nice,
definitely. Thank you so much.
Uh
yes, I had a couple of questions
Matthias answered partially already, and
he has a couple of questions as well
in the chat.
Uh first, I will cover my first
question, uh which was regarding the
iGenomes path and the pipeline test data
base path. They are like [snorts] noted
as string and not path. Uh Matthias
answered that because they are URLs.
URLs definitely for the pipeline test
data path, but iGenomes is usually not
an URL. I know that it can be an URL,
but most of the time it's an S3 bucket.
So,
could we put it as a path, or is it fine
to leave it as a string?
>> Uh
depends a bit, of course. Uh you just
need to
keep in mind that every time you put a
path there, then exist check will
happen.
Okay. Um so, for example, if you have
the iGenomes base in there,
then you shouldn't set it set it as a
path because not all systems will have
access to the iGenomes base that is the
default of the pipeline.
Um so, it's better to just keep it as a
string and then just check the actual
files that are derived from it, like the
faster file, for example.
>> Yes, then anyway, we will check the
individual file, so that's what is more
important. Okay, all right, good. Thank
you.
That's nice.
>> Yeah, just takes a bit of testing and
seeing what works and what doesn't.
>> Yeah, but just had a question. Do we
need to update the type in the schema to
get the string path conversion
correctly?
>> Uh
no, as far [clears throat] as I
the type path is not supported by the
schema, so there's
as far as I know, no real way to do
this. Maybe if the converse converter
gets updated and that it checks the
format of the
uh tools, so if I go to
the next tool schema of JSON, you will
also see that some
that every path does have format file
path in here. If the converter can get
updated to also check this, then it
could also
do the path conversion automatically,
but yeah, I don't know
how to do that or
yeah, if they want to do that.
>> Yeah, okay, good.
Uh
>> [snorts]
>> is there some tooling that shows
unnecessary duplicates of params, like
in main.nf and nextflow.config, like you
showed? Uh like next rolling or
something else?
>> Um
I don't think they show the duplicates.
Um I just know that the defaults
supplied here will take precedence
precedence over the ones defined here.
So, it's just better to keep as much of
your file parameters just in there and
only keep the ones that are really
necessary in the config files in here.
Uh you could also
Oh. If you don't want to
have those in there, you could also just
use closures like this, for example.
Um the curly brackets at the start and
the end here uh to tell nextflow to only
um
resolve this when the pipeline is
actually running and then when it has
access to the parameters which are in
the params block.
Uh which is something I personally do in
my pipelines now because it just makes
my life easier.
>> Yeah, yeah, yeah, I do the same as well.
Uh [snorts] Matthias replies that he we
can add the past type to the allowed to
the allowed types. So, that's a good
one. And that's the
in his opinion like the language server
script should check the the format.
>> Yeah.
>> Which I agree with as well.
No, all good. Uh does anyone else has
any other questions there?
Okay, I feel we are good in terms of the
question. So, if we have any particular
question on on on like helping getting
some help into converting a pipeline, we
can pester you on Slack.
>> Yes, of course. I You can also read the
blog post on the NFCore website which
also contains all the information I said
today.
>> Uh you have even your guest on. That
that's the same information. Yeah, I
really should have read the blog post
because otherwise like
I think that would have been like quite
helpful, but yeah. No, no, thank you. I
think I think this will be good.
Uh okay.
>> Yeah, of course. Feel free to pester me
as much as you want.
>> It was in the newsletter. Yes, Matthias
said that it was in the newsletter,
hasn't come out yet, right?
>> Was that also in the first one?
>> Yeah, I don't know. We'll see.
>> [laughter]
>> That's okay.
>> Okay, no, thank you very much for this
nice
bite-size. I do think it was very
interesting and I will convert right
away in my one of my pipelines. So,
stay tuned. And I will definitely pester
you if I encounter like any issue.
>> All right.
>> Yeah, I think given your presentation
and given the blog post that I haven't
read, I'm pretty sure I will like solve
it myself. So,
thank you so much. It was nice to see
you.
Yeah, and okay, and Matthias said that
he will see how easy it is to support in
the pipeline linking. Yeah. Yeah,
because I'm guessing also the main step
is like to change all of that in the
template as well.
>> Yeah, I've al- I've already updated the
IGEL config and the templates, but I
don't think it's a released version yet.
So, it'll be available in the next one,
probably.
>> Uh I don't know how Matthias has planned
for the coming release. I know that he
wants to put in probably the containers
before doing anything else, but yeah,
that's more discussion for another day
anyway.
Uh
Uh yes, Matthias uh so, Matthias this
Matthias, the other Matthias,
uh says that we have to keep in mind
that output paths cannot be path types.
They must be strings.
>> Yeah.
>> I'm assuming because they don't exist at
the time, right?
>> Because of the exist check, yeah. That's
why I kept it a string here. Yeah.
>> That does make complete sense. You
cannot check if a path exists if it
hasn't been created yet.
>> Yeah.
Um yeah, you you should you could also
just remove like for example file params
inputs the function you could just make
say params input right now. So, it's it
just makes it a lot easier to have your
paths in here, but it won't won't work
for OT of course. Uh you still need to
do file params OT here.
>> Okay.
Then, thank you very much.
Um
see you very soon. And um
goodbye everyone, and see you next week.
>> Bye.