Video summary
The video features a presentation by Louie from nf-test introducing version 1.1.1 of the nf-core/nf-test-utils plugin, which significantly enhances how unit tests handle output snapshots in Nextflow pipelines. Historically, nf-core relied on legacy Python testing methods before migrating to nf-test, but early snapshot approaches created files twice as large as necessary by duplicating outputs and struggled with unstable data like log headers or VCF/BAM files. The new release addresses these issues by introducing "sanitized output," a feature that removes numbered content from snapshots to reduce their size by half while allowing users to selectively disable MD5 checksums for specific keys, thereby balancing stability with the need to verify file integrity.
A major improvement in this update is the ability to handle folders containing multiple files correctly, ensuring that all contents are captured rather than just folder names. The plugin now supports "ignore keys" and "unstable patterns," which allow users to completely exclude channels like logs with timestamps or specific file extensions from the snapshot, leaving only their filenames. Furthermore, the tool introduces specialized arguments for "reads MD5" and "variance MD5," leveraging external plugins like nf-bam and nf-vcf to replace standard checksums with more robust metrics that ignore unstable headers. This simplifies test configurations by allowing users to specify a single process output channel and automatically apply different validation rules, such as checking only filenames for certain files while verifying read counts or variance data for others.
During the session, the presenter demonstrated how these features streamline testing for common scenarios, such as handling CRAM files where reference fasta information is required for accurate reads MD5 calculation. The audience discussed practical applications, including a significant Pull Request that replaced numerous manual checks in the module repository with automated nf-test-utils logic, reducing complex multi-line assertions into single, clean statements. While there are some technical nuances regarding the loading order of plugins to ensure compatibility with BAM and VCF tools, the consensus was that these enhancements greatly simplify pipeline maintenance. The presentation concluded with an open discussion on future possibilities, such as adding checks for CSV files or handling floating-point number variations between different container environments, highlighting the community's commitment to making Nextflow pipelines more reproducible and easier to validate.
Read the full video transcript
Hi, Maxim here hosting the byte size.
I'd like to welcome Louie from in
breast. Today he will present the latest
release of NFD utils. As usual, you'll
be able to unmute yourself at the end of
the talk to ask your question, but you
can also ask them on sec later on. So
now over to you Luis.
>> Yeah. Hi everyone. So as Maxim said um I
will just yeah present a brief overview
of NF test in NFO modules and and I will
yeah mostly
take yeah more time to describe how we
get the snapshot and why it's important
to get yeah stable output. So first
sorry the yeah so in NF core we for the
unit testing we yeah the legacy was
using uh uh piest python test but we
migrateed to NF test and a first
approach we had was to simply get a
snapshot of all the output of the
process and match it. This created um a
snapshot files uh at the in inside the
test folder. It works well but it has a
few issues. The main one that that we
you encounter is that it duplicate each
output
by either using a number to access it or
the name of the output. So it the
snapshot is twice as long as you need
it. Also all the files that you that are
outputed are checked with the MD5 sum.
It works for many files but if they are
unstable
uh for log files for example then the
files need to be taken out and it's also
not suitable for VCF for bum files where
the header is unstable.
So that was one problem that we have and
then in NFT utils the seven release we
introduce sanitized output that enable
us to first taken out taking out all the
numbered uh content. So that way you get
only the name of the output channel and
its content. So that reduced by tw by
two the size of the snapshot and it also
gave you give you the opportunity to uh
select some of the keys of the name of
the output and disable the MD5 some
check. So that way you get only the file
name and for most of the time it's it
works well. But if you want to ensure
that the VCF or the BAM files isn't
empty or it does contains relevant
information, but while keeping a stable
snapshot, you needed to get the variance
MD5 by using the NFTBAM plug-in or the
NFT VCF and it was yeah it complexify
quite uh the sanitiz output. put well
the snapshot uh length of the the
expiration length. So yeah, it was not
ideal. So what we want and also the
unstable keys at the time and even in
1.0 zero didn't work well with folders.
When you have multiple files in the same
channel by giving just the folder names
then it would output only the name of
the folder without its contents. So it
wasn't really nice. So in the new
version of NFT utils, we fixed first uh
for unstable keys the folder. Sorry for
the misspelled and now all the content
of the folder is currently correctly
outputed and you have the file names for
everything inside.
We also added ignore keys where when
there's an entire output channel that
you don't want to be part of the
snapshot, for example, when the log file
name isn't stable when you have a time
stamp, for example, in the log file
name. So that way you just put ignore
keys log for the output channel uh key
and then it disappear completely from
the snapshot. You all I've also added
unstable patterns and ignore patterns
that works the same way the same way as
the they key the key about but for those
one you just need to provide a globe
patterns. So it's
um it's not really reax patterns. It's a
bit different and simplified and it was
what was used in all the NFT utils
plug-in. And that way you can specify,
for example, a specific um extension of
a file or um a specific folder that you
want to
ignore or if you want all the files u
matching this pattern to not have the
MD5. So for example, if you have yeah
log file, error files or stout uh
printed inside a file, you can
uncheck them or ignore them completely.
And the only caveat uh on those two
parameters is that they doesn't work on
key provided in other uh arguments. So
if example you've provided
uh the band channel uh in unstable keys
then the unstable pattern won't check on
this uh channel. So that's the only
thing you need to worry about. And then
most more interestingly and that was
that's the main part I've worked on and
was a bit tricky but I managed to make
it work was to add reads MD5 keys and
variance MD5 keys. So these two
arguments uh allow to replace the simple
MD5 some uh for the snapshot and replace
this MD5 by the reads MD5s or the
variance MD5 for that uh it use NFT bum
and NFT VCF. So you need to have them
loaded before NFT utils in the NF
test.config. config and this way the MD5
sum will be replaced by for example for
reads it will be replaced by ban uh
files get reads MD5 and similarly for
variance MD5s.
So that way you get on your the file
name
uh two points the MD5 reads and the MD5
sum of only the reads without the
header. And similarly for variance you
get only the MD5s of the variance. That
way the snapshot is clean. You don't
have to specify really uh to yeah split
all the your different channel outputs
and say that you want the file for this
particular channel you want to get the
file name and the variance MD5. You can
now simply for example here
you replace all the different process
out phase varant collect get the meta
separate the VCF for the VCF you get the
name and the variance MD5 you simply
said I want to sanitize the process
output and for phase variance replace
the MD5 sum by the variance MD5.
You can do the same for for reads and it
also work for chrome files but for
chrome files you need to uh provide the
reference faster.
So for example here in uh I guess it was
a a snapshot from uh some tools view uh
you only the crown uh bi y and uh ca are
unstable so l empty I guess. So that's
why we put it in unstable in uh this
yeah in NF core modules because we don't
want uh empty uh MD5 and for unselected
crumb then you provide it in the re MD5
keys provide the reference faster. So
unfortunately for the moment it doesn't
work with params as we do in the when um
directive of the NF test. So that's
something maybe we should we could work
on. But yeah it simplifies quite a lot
the output.
And also for unstable pattern and ignore
patterns. It's really useful when you
have a folder with many many files and
some just a few of them are unstable or
some of them has uh for example here um
it has just some of the log files have a
time stamp in the name. So you can't put
them in the snapshot because it might
work for the day but the next time uh
the yeah the next day someone will rerun
the the checks and it will fade. So that
way you just ignore the r error and out
files. it doesn't appear in the snapshot
and for the PDF and the LO files for
example in this case you just put them
as unstable and then your snapshot all
the def all the other files are stable
and you get the MD5 and for the PDF it
just ignore them and you just get the
file name. So it does quite simplify the
snapshot.
So here I was just going to share with
you a few modification.
So if we take uh here
modules.
So if I I look for example in modules uh
I can take one that I'll put a chrome
file
and
take
in the snap
and one that has a chrome file output.
So there is plenty.
So for example test crumble and
[clears throat]
so this one for example. Yeah. So here
as you can can increase a bit the size
to make it readable.
So here typically
um what they encounter is that the
chrome files are unstable as you do not
have yeah
I don't have the faster file so I don't
know how they generate that so here the
you just need to sanitize replace all
those line by sanitize output process
out and you just need to say reads
uh MD5
keys
and just say chrome.
So that that was uh the key spot wasn't
something that I yeah get the first time
I use sanitized output. I was thinking
it was the the extension or it wasn't
quite clear what keys was meant to be
but it's the name of the output channel.
So in this case it's uh the crown
channel.
So yeah, here
I will just put a tag
right and so that that won't work as it
is because as it's a crown file
um it expect the faster for to read the
pair end
but and I don't see any where it's Yeah.
Yeah. So for this one it won't work. I
need to get a bit more where it work. So
yeah. So if I take this example maybe it
might be easier.
Yeah. So here it's typically what we've
done before. We get the ban. We get the
name to check the file limits table and
then get the reads and all that is
simplified by
what I done just before
can copy paste.
And of course and it's
bio
done
just
Yeah. So why it's running? I don't know
if anyone has some cation.
>> No, definitely it's looking good and
it's looking better. No, I really like
that. I've been using it like since you
made the release
and I guess we we have already some
uh
some yeah some uh
PRs with uh this.
>> Yeah, I think I I created a PR I think
we merged it already for
for I set two but not that one. I think
that was the one before.
Uh yeah, this uh I don't know one of the
P I made for ISO 2. This one seems like
way too way too small for that.
>> Yeah.
>> Uh maybe that was the first one for I
said too. Can you check the You see the
one you haven't
Oh, yeah, that sounds like it.
Yeah.
>> Yeah. So here Yeah. You just Yeah. And
many many modules were asserting that
just the output uh file existed and now
you can just said it's unstable and
getting
>> I prefer saying I prefer saying uh I
prefer snapshotting the file name that
just saying it's uh that just checking
it exist.
>> So yeah. So if we take for example this
so it simplifies it and now you get
instead of just having the test band
then the the reads MD5 you nicely get
the name of the channel the meta map the
file name and it explicit said that it's
a specific MD5 and and similarly for
variance instead of reads you get
variance here and you get then the
amplify. So everything is
>> yeah nicely normalized now and it it
will be easier in the future.
>> Definitely it's a thing of beauty and
congrats on making that.
Yeah, it was a bit tricky to make the
plug-in depends on all the plugins that
aren't necessarily
um yeah uh loaded or available when you
launch the test. So that's a bit tricky
to get but I guess in the future we
might have if NF test come under the
umbrella of next floor maybe they will
automatically
yeah
merge
>> or or maybe we could have a way to uh to
load a plugin within a plug-in.
Yeah, maybe. I don't know. But
for now, so you just need in the
>> So if I go back here,
>> it is an excluded config.
>> Yeah.
>> So you need to load it after NFT VCF and
NFT bump. I don't know why when you load
uh in a different order every NFT utils
will crash. No, it won't crash, but it
won't output the MD5 reads because I put
a sanity check. And if it doesn't uh
have an uh VCF or BAM plug-in loaded, it
will output the regular MD5 and it will
give you a warning. So that way it's
still stable. But I don't know why when
you load it before it's Yeah, it's some
Java magic. I don't yet quite quite yet
grasp on
>> but it works.
>> I think it's I think it's more something
we need to try to figure out like at the
at the NF test at the NF test level
first and then we'll figure it out. But
no no definitely I think that's amazing
and it works well. Uh [snorts] I made
like another P on the model repo for
snap aligner.
>> Snap aligner.
>> Yeah. Uh we can see it already. That's
the one where I use the
um
pattern
>> and definitely it works well. I was I
was super happy with that.
>> Yeah. So here the difference is that you
output uh a folder where there is
multiple files in it and then you said
that in this folder these files that the
the files matching this pattern uh have
unstable output the MD5 sum shouldn't be
checked and then instead of just saying
that all the files are unstable you just
yeah say that those one shouldn't get
the MD5s but the the other one gets. So
that's really that's nice because you
can more fine grain tuned and still
check that the results have yeah
interesting results or aren't at least
not empty.
>> Yeah, definitely like for for me that's
perfect and that quickly simplify like
uh what we're doing already in the
module repo. M uh
but yes as more example in another PR
the
12790
ah yeah [laughter] that was a big one so
here uh Matias did a huge work by
replacing all the one he find in found
with bum files and then use reads and
d5s. And so you get yeah
[clears throat]
likes to to make big PS
>> yeah quite a lot but yeah you simplify
five lines in in only one. So that's
that's really nice.
>> Don't know if I guess
you should be able to unmute yourself if
you want now.
Yeah. So some of them was already using
unstable keys but yeah it's better to
have read MD5 keys but because like that
you ensure that it's not empty at least.
>> Yeah but yeah I know that for sometime I
have some BM files that are like
unstable even if I use the read MD5
keys.
>> Yeah it can happen.
>> Yeah. Do we have like any like I'm
guessing we can always like update and
add more stuff that we want to. I'm
guessing uh sometime I we with bump file
I check the summary.
>> Ah yeah but I'm guessing it's just a
question of no like uh having fields
that we want to add and so on. No that's
good. Yeah,
>> that will just create yeah additional
arguments. But I I saw sometimes yeah uh
some check was getting the summary of
the reads instead of the MD5 reads. I
don't know what's the maybe it's just
because they want to check for example
if you do uh streaming or something like
that you were you want to be sure that
you have less read that you started with
or when you collect uh different VCF you
want to be sure that you have multiple
chromosomes or multiple individuals.
So I can understand that maybe sometimes
only the reads MD5s or the viance MD5s
aren't what you are looking for for your
specific unit test.
But in this case you can simply add a
new line with the just getting the
summary of this one.
[laughter]
>> I see. I see. Uh do you have any more or
can we open the floor for more
questions?
Yeah, I'm I'm open to cation.
>> Okay, good. Uh [clears throat]
M ask you do you have any more question
about the what you're doing or are you
happy with everything that you have done
in your PR already?
>> Well, I'm quite happy with the the
current changes like you said. Uh
something like check some for the
summary would be nice uh with regards to
the bomb or the PCF. Um,
but I think we can update as we go along
and use and then we'll just see where we
end up, I guess.
>> Yeah, I'm quite open to suggestion now
that I Yeah, I now have a better grasp
on what is going on behind
the scene in NFT utils. So now that the
link with NFT bomb and NFT VCF is done
and stable, it's quite easy if we want
to add other fields or other checks.
>> Yeah. I'm guessing what I would love to
have like is
checking for CSV file.
>> Yeah. Yeah. And rounding numbers and uh
>> Yeah. It's super nice
>> because you like this has been like an
issue and also some specific content in
some text file and so on. I know that in
in some pipeline I have like a function
just for that and yes it's slightly bit
bit of a mess.
>> Yeah. And the thing is that uh having um
a function that do that.
Yeah. The the other issue I find is
often is that cond and singularity often
get uh different uh floating number u
values
but it's stable between singularity and
docker and most of the time is simply a
library or a version that isn't uh
properly set up inside the environment.
YAML.
So yeah,
yeah, having something that take the
risk out uh from just yeah debugging a
bit more and putting the right version
in it to get out to get rid of the
variation in the floating number.
Yeah, that that can be nice sometimes,
but sometime it's necessary to go look a
bit deeper and find what is causing the
variation instead of just Yeah. getting
rid of the variation in in the way you
get the test.
>> Oh yeah. No, no, I agree with you. Like
for me, it's not a question of being
like purely reproducible. We want to
make sure that what we're doing is
actually repeatable and that we have
like similar enough like result.
>> I know. Oh, thank you so much. Uh yes,
so Sophia has left already. Uh [snorts]
do we have any other question from uh
people joining in? I don't see any
question on on Slack at the moment.
Uh I would have a question because at
the moment I know that the way we deal
with uh pipeline level test is fairly
different from the way we deal with
module and sub workflow test because
from the subway the module we just grab
the
okay we just grab the the channel and
from there we can make assertion on the
channel in the pipeline we wait until
the pipeline is done and then we look at
the files that are being produced. So
for the M solution was like to create a
specific like lib in the pipeline to
deal with that. Uh I do need to try to
move that away from pipeline into the
plug-in but yes it's slightly too
complex at the moment to to deal with
that. Uh but do you have any idea into
how we could try to move some of that
logic that we are doing with uh this uh
this function into a pipeline test?
I guess sanitized output should works on
just giving the file name
because sanitized output doesn't expect
the yeah the process out to be uh a
channel specifically I guess
[snorts and clears throat]
>> uh I did I did try process like for
example process outbam and have all of
that in a sanitizer put and I notice
that it doesn't work
>> but because I don't think it's expected.
I do think that it works on a specific
like type of u
of
>> yeah I don't remember what was what was
the type of the of the of the of the
channel there. Uh so I'm not sure it
will work on fail right away but maybe
it's something that we can think of and
try to improve there
>> because yeah most of the logic of
sanitized output is just yeah um
iteratively check the what is inside
a channel or a folder or files and
getting the different uh it just process
everything in how you
as by iterative. Yeah.
>> Yeah.
>> So yeah, even if it's nested, it it
works well. So
that should work at some point. We we
can get the sanitized output to uh to
work with the channels or just fold the
name. That that should work.
Okay, then uh let's like let's try to
have the talk like another day for that.
But no, thank you very much for the
presentation. That was super nice. I
really like appreciate that someone else
has put in like the work on this
plug-in. That's super good. And for me
that's what is this plug-in is all about
like it's it has been created like a
while back and we just like many people
have been contributing to it and like
adding whatever they they are interested
into and I really like this kind of
contribution on such a project.
>> Yeah that will that will simplify quite
a lot the snapshot.
>> Oh yeah know so much. Uh thank you very
much Lux. Uh
>> and thank you everyone for attending.
Uh, see you soon then.
>> See you.