HPC kitchen [data-management]: Data management is as important as data storage

Duration: 9:13 1647 transcript words EN summary

Video summary

Data management is a critical aspect of high-performance computing environments that extends far beyond simple data storage or merely finding physical space for files. While many assume the primary concern is only where to store information, effective management focuses on organization and control, which become increasingly vital as projects grow in scale and complexity. The video uses an engaging kitchen analogy to illustrate these concepts, describing raw ingredients as data and cookbooks as code; just as a single person can easily track items in their own home kitchen, larger teams with multiple members require structured systems to prevent chaos when people come and go or collaborate on various projects. Without proper management, the sheer volume of files stored on computers quickly overwhelms human cognitive abilities designed for three-dimensional spaces rather than complex file system hierarchies, leading to disorganized environments where essential information becomes impossible to locate. Several specific challenges arise in unmanaged data ecosystems that mirror common household problems but have severe consequences in research and science. For instance, if team members store files on their own isolated computers without shared access protocols, collaboration suffers because others cannot easily see or utilize what colleagues are working with unless manually transferred item by item. Furthermore, mixing private secrets like family recipes with public data prevents sharing valuable resources, while unlabeled containers of unknown origin create uncertainty about whether materials can be legally or ethically used in new studies. Standardization is equally important; just as a recipe from another country might use different terminology that confuses the cook, research data lacking clear documentation on how it was collected or what specific formats were used renders it unusable for generating new scientific insights, effectively wasting potential knowledge. Beyond organization and standardization, practical issues like expiration dates and reproducibility highlight why proactive management is essential to avoid waste and ensure scientific integrity. In a kitchen, letting food expire due to poor sorting leads to unnecessary waste, whereas in research, failing to document code or data collection methods means that results cannot be reproduced by others, which fundamentally undermines the validity of science since non-reproducible work cannot be published. Additionally, relying solely on local files risks losing valuable assets if a researcher leaves without archiving their work publicly; storing key findings as open data ensures they are preserved for future use and accessible to the broader community. The video emphasizes that while these management tasks require time and effort—resources often scarce in academia where researchers rush to publish—they must be prioritized over immediate output because neglecting them creates a backlog of disorganized, unusable data that hampers long-term progress. Ultimately, establishing robust data management practices requires creating shared storage spaces with clear labels for different types of content, defining explicit ownership and usage rights, and enforcing standardized protocols across all team members. Although implementing these structures takes time to set up initially, the effort prevents future complications where lost or misunderstood data could derail entire projects. Supervisors play a crucial role in ensuring that researchers have adequate time allocated specifically for organizing their workspaces rather than rushing through tasks without proper documentation. By delegating responsibilities and fostering an environment where taking care of data is expected rather than optional, research teams can maintain high standards of organization, facilitate better collaboration, and ensure that valuable scientific contributions are preserved accurately for the future.

Read the full video transcript

Hello and welcome to another lowquality HPC kitchen video. Today we're talking about data management. And as you can see, data management is different from data storage. So people may think that the only question of data is where do you put it and do you have enough space? But it's much more than that. So data management is more about the organization and the control of it. This becomes much more important as you get larger. So when you're alone, maybe you can keep track of your data or what's in a kitchen. But as you get more people involved and people are coming and going and there's more projects and more data, it can become a lot harder. So let's talk about what this can be. So just as a reminder, some of our jargon is data is the ingredients that go into cooking. There's also a different type of data which is code or in this case cookbooks. It's like data because it takes up space, but it's managed differently because it's actually instructions and not just something to be used. Okay, let's look at some common problems we have. So, first off, stuff can be disorganized and you can't find it. So, for example, have you ever had someone else come in your kitchen and use it and then they're trying to be nice and put stuff away and clean it up, but you can't find things anymore? And that's uhoh, it's cats is trying to eat some data. Um, and you can't find stuff anymore. So, this is a bit of a problem. As you're alone, it's easy. As the number of people grows, it becomes a little bit harder. And computers can be much worse because you can have far more files on a computer than you can have things in a kitchen. and our minds are designed to think about 3D spaces like this and not about file system hierarchies. Okay, next up, maybe we can't share things. So, for example, imagine if you were working with people for cooking a meal, but everyone had their own space and it's locked away. And the only way you could share something is if you asked them to give it to you and then they had to pick it up and hand it to you. So this is what happens when everyone's files are on their own's computer or own home directory or things like that. So basically it makes it harder to know what others are doing and well to work together overall. So it's important to have these shared spaces where we store things and there's rules for how things are used. Okay, cats is still very data hungry right now. Okay. Um, let's see. So, that's one thing, but also things can be so that way you can't uh No. Okay. Yeah. Things can be so that way maybe you can't share it because of other reasons. Let's take for example our own private cookbooks. So if I add my family recipe, my secret family recipes into my own cookbooks, then I can't share the rest of my recipes with someone if they ask for it because well, it has secret data mixed in with data that should be sharable. And this is also the same for data. So it's important to keep stuff organized early on so that way you can um share later. And it's always easier to do the organization earlier than later. Next up, maybe you can't interpret things. So, for example, there's different unlabeled containers here. So, this is not a chemistry laboratory. So, this is not the worst possible thing, but and I can like look in and see what it is there, but do I know am I allowed to use it? So, is this being saved for some other project? Or if I was doing a research project and there's something that looks useful, but I don't know the full story behind where it came from and how it was collected. Maybe I can't use it in my articles to actually produce something new. And that's not good. Next up, things can have different names or otherwise be non-standardized. Have you ever had a recipe that comes from a different country, different continent, whatever, and even if it's in a language you understand, the names they use for some things are slightly different or some of the standards they use are slightly different and you can't interpret it in your own work. So, this can happen all the time in research. someone can have data. It's like nice CSV file that has whatever is needed or whatever format it may be, but since you don't know the details about what's going on with it, you can't really use it and you Yeah. And this is not good. So, this standardization and you know um reporting documentation is really important. Next up, and this is mainly a kitchen thing, but food can expire because you forget about you have it and lose it and it makes food waste. So, my solution is whenever I get new food in the fridge, I sort the new food and the old food by expiration date. So, that way all the food that's expiring soonest is in the front. And then, well, first off, it's easy to cook because I don't have to decide what to do. But, I start at the beginning and just work my way through. and I hardly ever had food go bad anymore. And well, I don't know what the metaphor is here, but the idea is that you can't just will your way into better data usage. You need structures and things like that to arrange the system so that it promotes the outcome you want, which for me is not wasting food. Okay, another thing. Let's say you have roommates or something and they move out and they leave stuff behind. Who knows what that stuff is? I mean, if it's food, it's labeled. But let's say you have these unlabeled things or whatever. You might get in a case where you don't know what it is or you don't know if you're allowed to use it because you don't know who it belongs to. And then, well, it becomes a waste. The same can very often happen with research data where someone goes and works and collects a lot of data, but they don't document it well and then they leave and then it's sort of lost. Another big thing can be code not being reproducible. If you cook something really nice, but you don't write it down, then well, maybe you can't do it again. And well, for food, that's fine. Maybe it's a nice mystery meal. I like this kind of thing. But for research, if you can't reproduce what you've done, that's really bad because that means you can't publish on it and it's not science if it's not reproducible. So, you really need to keep track of this really well. Maybe another final thing is archiving things. Let's say you have a really good recipe and you want to not lose it. You can write it down and leave it in your kitchen, but it might get lost. But if you would say write it on a blog somewhere or submit it to a recipe site, it's much less likely to get lost ever. It might even be easier for you to find it in the future when you can't find the paper you had written it down on or the file on your computer or whatever it may be. And the lesson here is that sharing data and archiving it publicly u as open data is really useful to make sure that even you don't lose data but also makes it useful to other people. So those are some of the main problems and how it relates to research and what are some of the main patterns we saw. So first off there's the shared storage space and standard organization is really important. There needs to be different places where stuff belongs. For example, have you ever seen a commercial kitchen and all the different cabinets have labels of what should go in there? There's lots of people working there. So, they need to have the structure in order to keep it organized. There should be clear ownership and usage rights for everything so that way you can use it if you need need to or know who to ask about it if you need more information. Someone should tell everyone the standards and everyone must know that they're expected to actually follow it. Many of these things aren't really hard, but they need time. And time is something that's always um in short supply in academia. So people are always rushing to complete their next paper and maybe they say, "Well, I don't have time to do this properly right now." Or someone may always be rushed. Okay, I need to make my next meal. don't have time to clean the kitchen before I start and get stuff organized again. So in the kitchen that's not a big deal perhaps, but in research it can really pile up and you can make things much worse for yourself in the future. So I think the lesson for us all is that we need to have time to do this. Supervisors need to make sure people have time and have the expectation to do this properly. delegate it where it's necessary and then we need to make sure that we know what we should do and to ask and be given time if it's needed. So with that being said, thank you very much and I might see you in the next one. Bye.