INCF Software WG recordings: HED and SpikeInterface sessions

um so thank you for joining the software highlight on head which is hierarchical event descriptors I know some of you some of you haven't met yet um but I'm today I'm going to talk about what head is and what it uh brings to Fair data sets now head is completely uh open source and everything is online and I'll also share my slides afterwards um for the the meeting so I'm going to address these questions um first what is head and what problem it solves for for fair then I'm going to talk about how head is integrated into bids bids stands for brain Imaging data structure and I'm going to talk about tools that support head annotation validation and Analysis and why they're useful and finally I'll address the work that's ongoing and planned for the future so uh there's a data sharing Gap uh Fair requirements generally focus on highlevel standardization uh at the data set level and data set search is generally at the uh publisher of the for the Publishers is generally at the data set level unfortunately that's not the granularity that's actually needed for analyzing neuroimaging time ser iies um they need detailed unambiguous event descriptions um ideally data search would be based on experimental design tasks and cognitive processes and uh events need a context um which is what else is going on um while these events occur so had which is hierarchical vent descriptors Plus bids uh we contend gives a practical path to machine actionable data sharing and I'll talk about what that means uh shortly so head annotations um are comma separated lists of head tags so for example sensory event visual presentation and Q are examples of head tags and head tags come from specific vocabularies that are organized in a tree structure so on the left you see a snapshot of our schema viewer um which is partially open some of these elements are collapsed um so they have items under them and you can uh reference reference the head tag by its schema path so for example sensory event could be addressed as event SL sensory event when you give an annotation but the individual terms in the head schema are unique and so you could just use sensor event um you don't have to provide the full path now head is a subtag system and what that means is that it consists of a group of hierarchically organized terms and in each subtree the nodes uh satisfy and is a relationship so sensory event is a event and uh this is important because it allows for search generalizability so if I search for event I'm going to get um all the data that's been tagged with uh children of that as well as tagged with events so I'll get data features or sensory events as well um so had um well so if you look at the uh uh schema uh let me just point out a couple of things structurally which are useful in The annotation uh there's basically three kinds of subt trees here uh there's the noun trees like event agent and item and then there are verb um Tre action and finally there are descriptive tags um which are under the property and relationship so head supports Advanced annotation features such such as the representation of ongoing events and searching annotations based on experimental design now the head vocabulary viewer is available and uh the schemas are on GitHub so everything is open so head um is well integrated into bids and as part of the data set validation for bids it uh head is validated in order to use head with bids you simply need to specify the head version you're using in the data script the data set description Json file and uh then you can go ahead and annotate your data set and we're going to go through some of the tools that allow you to easily annotate a bid's data set so there are basically two use cases in annotating head in bids the first use case is that you annotate each individual event and this is very useful for artifact annotation or um in the case of clinical neurologists um they may want to annotate seizure features in an EEG signal and so those are unique to the particular point in the data set they're not something that's generally applicable across the data sets so use case case one is IND annotate each event an individually but there's a second use case that's also useful and that's um to recognize that most experiments use unique lab specific event codes that um are applicable across all of the Imaging data in a particular data set unfortunately those unique codes are not applicable across multiple data sets and that's part of what had brings um to the table we take those unique event codes that are lab specific and provide um a matching annotation that then can be used across data sets to do um multi- dat set analysis so the Second Use case which uses um which annotates the unique event codes uh uses Json side cards which is part of the bid's data set standard and you can use combinations of these approaches so there are tools for supporting annotation using either approach and I'm going to demonstrate those now and once you do that um the annotations can be used directly by dat data analysis tools to do things like extracting the particular sections of data where particular sensory events have been uh presented or responses have been made or much more sophisticated uh types of queries so let's take a look at how you actually do annotation in bids um bids event files are represented in tab separated value files and we see an excerpt of a typical event file on the left each row represents an event marker in the underly neuroimaging data set so the what the events give you is information about what is happening as the experiment uh unfolds and this is critical to being able to analyze um the cognitive processes that are going on now in the event file each column represents specific information about that event and in bids uh all event file must have an onset column and a duration column the onset column gives the time relative to the offset of the matching data Imaging data set that this event marker occurred and the duration uh column is a little tricky and isn't always uh it's not always obvious what duration is it's referring to and head can solve some of the problems of am ambiguity in this so so for use case one which is annotating the individual events um you can use an optional head column and if you use a head column in your event file then that can only contain head annotations but you don't have to do it there and um so to annotate individual events you simply Place head annotations uh next to the rows of the events that you want to annotate so use case one is directly annotating each event in the events file now the second approach uh uses sidecars and this relies on the fact that um many of the event codes are going to be repeated so there are a lot fewer event codes than there actually are events in the data set so you start with an events data set and you create a dictionary of the unique values and we can have uh we have a tool for doing that and once you have that much smaller number of unique values you simply fill in the head annotations for those values that's all you need to do to annotate the data set and bids actually has an inheritance principle which means that for most data sets you only have to provide these annotations in a single Json file at the top level because they're applicable across all the recordings in the event file now you can do specific things for individual files but mostly you're going to be producing a single Json side car file with the head annotations for The Unique event codes in the data set so you fill in the head annotations and then at analysis time there are tools that actually look up these annotations and produce a an assembled annotation for each event marker in the data set and those can be fed directly into there's strings that can be fed directly into analysis tools and we have a number of analysis tools that use these uh event annotations for example you can filter based on head tags and other information to get a reduced um event file that focuses on the exactly the event markers that you need for your your data set so that brings us to the problem of actually getting the head annotations into the data set and we have tools for that also so our task here is to fill in head annotations in this dictionary of the unique event codes in the data set and we're going to do that by using a um a recipe and we have a recipe for sort of basic simple annotation and there's a caveat here that um head provides some much more sophisticated capabilities for annotation but basic annotation is really useful and it's fairly easy to do so you get a lot of benefit from that first um insertion of annotation it will provide a lot of usefulness for your data so the recipe is very simple and um the table is online in our documentation you ask what kind of event it is and um in this case maybe sensory event and then if you picked a sensory event you add um the supporting tags so if it's a sensory event then you might say well what kind of modality is it is it a visual presentation or an auditory presentation or something else um one of the things that we've recognized is that a lot of sensory events are not that occur during an experiment are not exper enal stimulus there are instructionally uh things and other things that go on and they all cause um neuroimaging responses and so if you don't annotate all those things that aren't exactly experimentally uh stimuli then you can't regress out the interference that these other U sensory events occur so it's very important to report all your sensory events and then to distinguish what kind of sensory events there are and so you could follow on with saying okay well this is an experimental stimulus or this is instructional or a Q and then you can say well what kind of thing was presented in there and head provides tags for that so for example this non WM code is a sensory event it's a visual presentation it's instructional and a Q and it's a dash black and it says get ready basically um think about what um remember what you saw in the um previous seven presentations of letters uh the correct event code goes is annotated as a sensory event it's an auditory presentation and it's a beep which provides feedback so these very simple annotations can be done fairly quickly and you can follow the recipe to get a very basic um annotation in and then recognize that annotation and restructuring of event files is an iterative process and as you proceed through analysis and other phases using the data set you may need other kinds of information and the idea is that it's easy to reannotate if you need it simply by changing the Json sidecar so uh B basic information is straightforward and we've got some supporting tools for that and I'm uh going to summarize some of the tools that we have and then go into a little more detail so our main code base is in Python and those head tools are released on piie so you can um automatically download them as part of your requirements when you run python we've implemented a lot of our online tools um a lot of our Tools in online format and we have both a web-based form so you can fill in a menu and and do something and we also have them implemented as web services and these are very useful for doing an initial annotation or analysis using a single file um they're not good for annotating for doing something over the entire data set but they're great for debugging and um doing a basic annotation they're implo deployed in Docker and the docker containers are available on GitHub if you want we also have um a userfriendly standalone guey for annotation which is cager and um in bids bids doesn't actually have head annotation directly in in it instead um they have a library they call on which is the Head validator Library which is available on npm and and when bids recognizes that there's head annotation in the data set it calls on our head validator to check to make sure that those head tags are correct um we've also been working on some matlb tools um and Matt these mat lab tools are well integrated into EEG lab um and we have uh head services and other direct calls in matlb as well so mat lab users can use these as well now I'd like to uh give a demo of what the online menu tools look like for doing annotation of the Json side car that I talked about in use case 2 so that so on the left is an example of the online menu for the event file uh for processing event files and you'll see that there four possible actions here validate assembling annotations generating a sidecar template or Remodeling and we're going to generate a sidecar template here from an events file so it's going to read in an events file and actually generate a Json side card that we can fill in with head anotations so we browse all we have to do is provide a single event file and once we do that the server reads the event file and lists the columns that are in the the event file the column headers and it also uh processes the contents of the events file and lists the number of unique event codes in each uh column in parentheses so you see onset has got 350 different unique values in that column and and that is basically the length of the event file but many of the columns have a much smaller number uh the event type the task role um and the memory condition have um very small number of unique codes letter and trial have a larger number now I'm going to annotate the event type the task roll the letter and the trial column and there are two kinds of annotation here one is you annotate each unique value in a column and the second one is you just annot the column as a whole and I'm going to demonstrate those two uh by designating the event type and the task rule as categorical columns so they're going to be each unique value is going to be annotated individually and I'm not going to designate letter and trial as categorical so they'll be annotated using a single annotation once I've selected The Columns I want to annotate I press process and a Json side car is generated it's a legitimate validated Jason sidecar so it if you put it in your bids data set um your bid's data set will will validate the annotations aren't all that useful they're mainly labels and descriptions but the idea is that you then fill them in so um you'll see here that the event type which I selected as a categorical column has individual entries for each unique value in that column and there are seven listed there are seven here the uh on the other hand the letter which was not designated as a uh annotation a categorical annotation just has one entry and you'll notice it has a number sign and the idea is you provide a head annotation and somewhere in there is a number sign placeholder when The annotation is assembled the value in the column will be substituted for this number sign placeholder now bids also allows descriptions in the levels and it's very important to provide P full text descriptions of your events in addition to the Head annotations so the idea is you simply fill in these um annotation template sections using the recipe and you're done so it's a fairly quick process and um I actually made a YouTube video which is on our YouTube CH they had YouTube channel that goes through this process and does The annotation um using these tools so I'm all next going to just briefly show our cager tool for annotation and you'll notice that on the left so this is a gooey on the left are the unique vent um codes that were in the event type column which is um at the top and in order to annotate I click the show head schema and that provides an additional window with the events that I need to um the event tags that I want to select then I simply uh click on the different tags as I want to annotate and my annotation is assembled and I'm done so cager is available as a standalone tool and it's a plugin that's also integrated into mat lab and EEG lab so the next kind of tool that I'm going to introduce are the head remodeling tools and the head remodeling tools are a platform for implementing a lot of our analysis as well as validation and other kinds of tools so it's a an interface basically which allows um user access to the tool Suites that are provided in head tools so the remodeling tools have two types of information of operations one are transformations and transformation operations take an event file and they do something with it to produce a new event file and we have a large number of operations um available there but the the idea is they actually change the event file they don't store any state but they um make a transformation the second kind of um operation in the remodeling tools or summaries and they take also as input an event file they may also require the Json side car with the head annotations if it's a head operation and a head schema if that's needed and they do not change the event file in any way but they analyze it and save the information about that event file in um a global context file Global context structure and then after you've processed an entire data set uh the idea is that you can then request the output um of these summaries and either as an overall data set summary or summarizing individual results so the advantage of having this sort of um tool set is is that you specify the operations in adjacent file this is very reminiscent reminiscent of the bid stats models and we're hoping that some of these will be um actually incorporated into the bid stats Transformations but the idea is that you don't need any programming and you can pick your operations from a series of templates and string them together to to transform event files in various ways and the Json file with the remodeling instructions gives you a record of what you've done so you can repeat it um and you have Providence for the operations and most of the head tool functionality is now available through this interface that's not the only way that the functionality is available but this is very convenient um for nitty-gritty data analysis and for data curation so I'm going to uh give you a few examples of what you can do with the remodeling tools and one of the examples is to in a single file so the the example here is we want to do something on a single file and we the online tools will allow you to do this without any um information so you start with an events file and and each operation has h a format which is basically operation description and then parameters and most people use this this by going to the documentation and um cutting the template out put pasting it in their file and then filling the queries in in the remodeling file give it to Yes um we I mean I lost you for a bit in the middle would you maybe repeat a little bit of the slide please uh sure so I did get a flash up that said my internet connection was unable so um so in the let me just start over on this one um so the idea of the remodeling tools is that you provide the instructions in a Json file and this allows you to perform the operations without any programming and you have a record of what you did so you can repeat it um so you start with your events file if you need a Json side car uh because you have head annotations you could provide that and you give it the Json um remodeling file and in and all of the instructions are in the same format you list operation you have want you should give a description of what you were aiming for when you gave this operation and then you provide parameters um and in this case the only relevant one is what queries you're going to ask of the head tags associated with these event markers um to pick out the rows in the event file that satisfy the queries so um we have correct action and incorrect action which are really simple queries uh but again the emphasis is that head supports very complex queries um that allow you to do very uh sophisticated searching so you could run this since it's on a single file you could run it online um and test it and see what the results are and the process is that the head annotations for each row are assembled and then it computes factors based on the results of the query so the idea is that it if it produces a column and the answer to the first query which is is it a correct action is labeled in a column called correct it produces a column of zeros and ones indicating whether the individual rows satisfied that query and so you can produce an arbitrary number of factors you can have any number of queries here and and um one of the things is you can actually since head supports quering the design Matrix you could produce a a design Matrix automatically if your head annotation has um the information about the design Matrix and one of the important things is that queries can be very complex so let me just show you an example of how head tags have been used in this kind of um factoring this is a study that we did a couple years ago we used 18 head tag EEG studies and um we used ersps uh we calc for each individual data recording and there were over a thousand and recordings in this uh set of 18 studies um we computed uh the ersps for um each of the channels in the data set using linear um regression and uh got a vector uh which is shown here as the spectrogram but yeah you know we it's linearized as a vector to go into TS and E which which is a uh Dimension reducing algorithm designed to take very high dimensional data and produce nice pictures in two or three dimensions and what you can see here is that the if you color the projections based on the dominant head tags that were in that that annotation there are well-defined clusters even though uh these data sets came from different kinds of paradigms different headsets completely different subject pools different numbers of channels um there still is a commonality among the head tags um and the idea is that by Common producing common tagging across many studies you can combine the results to increase the statistical power of your studies so I want to demonstrate also um the experimental condition summary um so this is an example of the summary operation and this would have been run from the command line because I wanted to do the entire data set so I start by um giving the Json remodeling file and this is for conditioned variables so this is something which allows us to tag um and summarize the design Matrix or the condition variables in a data set so I start with the remodeling file and it's very simple the only thing that I have to provide is the tag type that I'm going to use and in this type it's condition variable and since I'm running this from the command line I'd have to produce some arguments and I'll show you those very shortly and once I do that um it'll go through the whole data set and produce a summary of the condition variables or the design Matrix um in the data so this is an excerpt from the overall summary the the all of the summary tools produce an overall summary of the complete data set as well as individual summaries of each um event file uh separately and you can select what you want out of that um when you do the setup with the arguments so if you look at this overall summary under this data set line You'll see that this particular data set had three condition variables a total of 63,000 events and there are 126 event files now um I'm only showing part of this and so I'm only actually showing one condition variable and then the rest is is down the the line and that conditioned variable was key assignment and key assignment actually had two levels um there was a right symmetry condition and a left symmetry condition I'm only showing the right one and what the summary says after going through the whole data set is that the right symmetry condition had uh 35,000 events in 60 files and you'll notice that the key assignment um only appeared in 108 of the total 126 event files and the reason that that was the case is that this data set also had some behavior of files that didn't have that information in it and then the summary will provide um a description if you gave a description at which is always encouraged and it'll tell you what the uh head tags is associated with this summary were so these can be run automatically um on data sets very simply now let's look at how they might be called so one example is to call the these remodeling tools from Python and in both cases whether you call it from python or the command line you use the Run remodel uh module and the to run it from python which could be a Jupiter notebook you simply call um the main program of run remodel with an argument list and the argument list if you look down below um there's a data root which is the root of the data set that you want to analyze so you have a choice with these tools of ass assuming it's a bid's data set and using that information to gather all the um events or if you if it's not a bids data set then giving the data set root will cause the tools to find all the event files in that directory tree and analyze the data set um using that information so in this particular example um I have the data set root and you also have to give the model path which is the the Json remodeling file with the instructions and I'm using in here A minus B option which is bids say okay it's a bids data set and that will GA allow me to gather all the information like the Json side cars um it'll use the data set description and other things to find the information now the final um argument here was a minus X and that's a list of the directories to not use when doing the analysis so I mentioned that there were these behavioral event files in the data set so I'm excluding the be directory because um I don't want to analyze those Behavioral Studies and I'm incl excluding the derivatives directory if you're familiar with bids because that has doesn't have the event files in it so I can run it either from the command line or from Python and um I can also run it from Matt lab and we are providing wrappers for uh a lot of things but the Run remodel will allow you to run everything um that's available through the remodeling tools um using just one command and giving the arguments in a Cell array instead of a a square bracket list but other than that um it runs the same way so let step back a little bit and talk about what the workflow for event files is for most people so an experiment starts with the experimental logs and from those experimental logs uh the researchers have to generate the event files and this is often a very arduous process um and very often from our experience multiple iterations are required to actually turn the information from logs uh into event files so that was part of the impetus for developing the remodeling tools we curated a lot of data and the process of converting experimental logs to generate event files and CH and curating them caused us to do the same operations over and over again and this was a motivation for making a tool set that would make this easy and repeatable uh then you're going to want to validate the data and this will include head validation if you've put in head data sets and then you want to do an analysis and what we F also found from art experience is that often event files need to be restructured to support a particular kind of analysis when you curate the data initially you can't possibly anticipate all possible uses of the data and so another place where we ended up doing a lot of refactoring of our event files um depending on what was needed turn was in the analysis phas so the remodeling tools um are used because we found that we were repeating the same operations over and over for multiple data sets now I want to talk a little bit more about the process of remodeling with and without backups I haven't mentioned backups and the remodeling but one of the problems with transforming event files is that it changes the event files and if you do multiple Transformations it's very easy to lose lose track of where you are and to lose the information from your original event files so we recommend that you actually use the head remodeling tools to do a backup and the the backup um produces a directory tree of the original event files and the derivatives directory um and that's the so you've got your you know the starting point at which you um curated the data once you've done that you you don't have to think about the backups anymore and each time you want to do remodeling when you call the remodeler um it copies the original event files back into to the data set uh and applies the Json remodeling file to these copies so you always know what the state of the event files are when you do the Transformations um after you do the Transformations the remodeling writes overwrites the event files in your data set but remember you've got a backup copy and you can always restore them at any time and the consequence of this for the Json files is that generally you just you take have a Jason transformation file and you and the operations so it always starts from the original you can also run this without backups um and for summaries that's you don't need backups at all because the event files are never changed so I'd like to step back and just um give you an overview of of head past and present head's been around for well more than a decade and it started out um as Nema bdal shamlo PhD thesis under Scott MC and the design of the head hierarchy was a single event hierarchy it had cogp as a subset and it was used as the basis for uh one of the first EEG repository open repositories um that we came available this was head it which is uh still in operation and it had 12 studies in the S format which is the EEG study format um and this had initial vocabulary um had some problems which as we went to annotate more data sets were detected um and so second H generation head was born um as part of an army project um which was a multi-institution project to uh instrument the brain and body at work and EEG was the neuroimaging uh format used in this study in this uh Consortium thing and one of the important deliverables of this was an open repository um with well curated uh EEG studies they had it had ey tracking and lots of other information um like the the experimental logs um and and the result with 30 head annotated data sets um and as part of that using um the data caused a redesign and this is when head became a subtag system so U things were separated out into aor orthogonal sub hierarchies um during this time it was incorporated into bids so had is bed in bids for a long time and um validators and the online tools were created so in about n 2019 the head working group was formed and um the goal was to move forward with much more sophisticated annotations um that were allowed a better representation of events their inter relationships and um the context in which these events occurred and so from 2019 on a group of us have been working um and head Generation 3 the first version of which was 8.0.0 was released in August of 200 uh 21 and it has a significant number of advances one of which was the single tag annotation which allows you to just put the tag individual tags rather than the Full Pass which is a great Boon to annotators um but another thing that was a real advancement is the introduction of Library schema so as we the group was working to try to to clean up the the hierarchy and um really review each term that was in it uh it became a clear clear that the hierarchy could easily um suffer from bloating because each individual Paradigm had its own kind of tags that were needed that were only applicable to that Paradigm so for example the Army um repository had a lot of driving uh studies in it and that required information about the steering angle and and things like that which were not really generally applicable across data sets so the idea of additional vocabularies that would be field specific was introduced and our first uh Library schema was just introduced this is the score schema uh which is based on the score standard for neurological clinical neurological annotation so this is something that clinical neurologists use used to annotate things like seizures and sleep um problems in EEG recordings and uh so score was just released and examples have been incorporated into vids another advancement has been the events of temporal extent so as mentioned head has lots of advanced features which I haven't talked about today um one of the important things is this events of temporal extent um so the idea is events are real events not event markers but real events are actually things with a start and end and they may have intermediate points that is they unfold over a period of time and head has a mechanism for allowing annotation of this in a fairly straightforward way I didn't cover it today but um we do annotate data sets that way and it's very useful in tools and so an example of why you might want to do that is a seizure extends over a period of time and you want to Mark the start and the end but you want this to be an event and have intermediate points marking uh unusual features another example is annotation of a reaching motion and in motion capture they're going to be several phases as somebody reaches to grab or touch um and another uh item that we've given is the event context so they're tools for finding events that are ongoing at a time in the data set so you might look at an individual marker and it says you're producing a stimulus but you might also want to know what else is happening at that time and the head tools allow you to extract that event context to see what other events are ongoing um at a point in time and head also supports uh sophisticated searching so under development we were developing a number of specialized Library vocabularies uh some for movies and language and we're working on encoding the relationships between events and tasks and between different relationships within event files and uh an important item is integrating better with experimental Control software so that head tags can be inserted fairly early in the process now one of the things that I often get asked is well why isn't head in ontology and how is it what it's um what's its relationship to ontologies and um my response is well head is complimentary to formal ontologies and it's really a front-facing user technology uh and by frontf facing I mean that mere Mor mortals can annotate and if you've ever tried to annotate something in detail using ontologies you'll know that um it's meant for machine annotation and it's fairly hard to annotate um you know as a user um another is that head is meant to operate at a very fine grain and most of the formal ontologies that are in existence um are more high level so they're for tasks or diseases or uh items but it's not about unfolding over time so head is primarily concerned about what happens during an experiment whereas informal ontologies mostly are not um however uh head can be embedded into a formal antology and the idea is that the subclass relationship um gives the uh the graphical structure for the ontology and we do have plans to embed head in in antology and then um provide more complex graphical relationships between the nodes for Downstream um more sophisticated processing but for The annotation end and for doing um the more straightforward analysis viewing had in its um implementation as a subtag system is much more usable so um with that I'm going to end I'd like to acknowledge the members of the headwork group particularly um young Tren who who is working on cager Monique Dennison who's worked on the remodeling T pal Alia who's um done the score Library schema um and Ian Kellan and uh Alexander Jones who've worked on the python and JavaScript toolbase all of our stuff is online we have a very good resources page with lots of very detailed documentation and we have a YouTube channel with some of the um demos online so with that I'm going to stop sharing and uh bring it back to you thanks very much K that was very very very very interesting and useful um I