Scaling Reparative Metadata Assessment with MaRMAT

Welcome so much to our session scaling reparative metadata with Marmot the Marriott reparative metadata assessment tool. We're so appreciative of your time and I know there's some other great sessions going on concurrent to this. So I really appreciate you selecting to come to ours our presentation. So it's to start with some introductions. I'm Rachel Whitman. I'm the metadata strategies librarian at the J. Willard Marriott Library which is here at the not here but Salt Lake here the University of Utah and I oversee the metadata unit that creates and remediates descriptive metadata for our digital collections. I am here with my project colleague Dr. Kaylee Alexander who's a research data librarian at the J. Willard Marriott Library and Kaylee led the development of our Marmot beta tool which allowed us to get a proof of concept out and secure some very critical internal grant funding which allowed us to hire Aiden DeBoer who has been our programming assistant for about a year and a couple months. He is not here because he is finishing his junior year at the University of Utah and he's a data science student and he's done a tremendous amount of work bringing this tool to the much improved tool that is available today. So I'm just going to connect the dots really quick in case it needs to be done. The Marriott reparative metadata tool acronym Marmot is pretty close to Marmot which is why we chose a little animal as our mascot and considering the emotional toll that working with reparative metadata can cause we seized on an opportunity as a moment of levity to to bring some a cute animal into the work and also yellow-bellied marmots are common in Utah's high elevation meadows in northern mountains as this diorama from the Utah Natural History Museum depicts, which I highly recommend visiting if you're looking for things to do around Salt Lake City. Um so to give you the high high overview of what Marmot is, we first to plug we have a fantastic website marmotproject.org which hosts a ton of information and points you to the latest release. But Marmot is a open-source freely available tool that operates off of Python scripts and it was created to support metadata practitioners and the bulk assessment of their metadata for harmful, outdated, and otherwise problematic language and subject headings. Marmot functions on a Windows, Mac, and Linux and there's a lot of information about how to download and install it on we have a the website and also points to our GitHub where we have a ton of information. And I'm not going to get into installation because this is not you know working session or workshop, but we have an executable file for Windows so you don't need to install anything, but for Mac and Linux you'll need to install Python and some dependencies. It's not hard, I swear. We have a lot of documentation, but I'm going to back up a little bit before we jump too much into the tool itself and just talk about the Marriott Library's digital collections and some reparative metadata projects that we undertook before the creation of Marmot to just kind of give some context of how this work was going before an assessment tool like this. So we have a number of distinctive content or sorry collection areas that need extra care for reparative needs and looking at legacy metadata and those include our indigenous people and culture collections documenting the Japanese American incarceration during World War II, women in Utah and other cultural groups um within Utah. And so, how we were approaching this work um previously before Marmot was developed was really looking for specific outdated offensive egregious terms by keyword searching in our collections and our metadata and then doing remediation projects based on isolating those keyword searches. We also did a number of changes for Library of Congress subject headings, which you know, there've been a number of throughout the years and we don't have um automated um authority control, so those are done through a similar process of isolating them and repairing them. And we also published a harmful language statement on our digital library special collections finding aids and catalog that acknowledge the presence of harmful language in our catalog records and a call to action to let us know um like where the patrons or the publics would find them. Um I think it's important to talk about what we mean uh when we say reparative metadata. And this isn't a definition that I really love. It comes through the Inclusive Metadata Toolkit, which I was actually um lucky to work with as part of the Digital Library Federation's um Cultural Assessment Working Group. And to boil this down, we're assessing existing metadata, whether it was created in the recent or distant past, and remediating where needed to uphold inclusive metadata principles, which seek to employ a wider range of perspectives that have been traditionally excluded from and silenced in descriptive practices in order to more accurately recognize, represent, and respect the breadth of human experiences. And this work um with the Inclusive Metadata Toolkit, I was really trying to immerse myself on um targeting areas of reparative work that I found in the our metadata in our collections. And it was through this work with the Inclusive Metadata Toolkit that I found out about a tool from Duke University that worked similarly and was the inspiration for our Marmot tool, but it only works with EAD and Mark XML records. So, I wanted to It was my mission to make a tool that would work similarly to do bulk assessment of metadata regardless of what schema it's in or format. >> Um yeah, so this is Duke University's tool. It was also freely available on GitHub. Um because I work largely with data and a lot of tabular data, Rachel came to me and said, "I've got this great tool. How can we modify this code in some way uh to work with what we need to do in the digital library?" So, our initial question was, "Can we modify Duke's code to work with other digital collections metadata, specifically Dublin Core, which is what we use um for digital library?" Um so, I did my due diligence. I wrote to the Duke folks and said, "Hey, what do you think we could do with this?" And they sort of gave me their blessing and said, "Maybe AI can help you." Um so, we took that um and started thinking through some problems because uh I work with data a lot, but I'm no means a programmer. I know very little Python, though now a little bit more. Um we were also uh looking at the lexicons that they provided with their tool, which were really robust and based on a lot of research. Um but the relevance wasn't always going to be directly correlated to what we needed to do in the digital library. There was also some tweaks to formatting that we wanted to uh think about um working with tabular metadata in a more uh reasonable way. And then we wanted to focus on scalability and accessibility as well. So, Duke's tool was really fantastic for finding aids, and I still highly recommend it. Um but uh it was only able to operate uh on Windows through the graphical user interface, and we wanted to create something that was a little bit more accessible to different types of users. And that maybe required a little bit less technological prowess going in as well. Um so, first and foremost, Maruit is not an AI tool. Um there has been some rumors that it is, but it is not. Um we did embrace uh Chat GPT as a way of sort of prototyping and testing what we might do uh with a tool like this to create one. Um so, we were working in Chat GPT and I was very naive. So, the first thing I did was drop Duke's code in and say, "Hey, make this work with DC metadata." And it said, "Sure, here's this great application. Use it now." And it didn't even open on my computer. Um so, we really started thinking about how are we going to do this? What are the steps that we as librarians take in wanting to repair metadata? Um and how can we enlarge that? So, we started iteratively iteratively prompting Chat GPT, what would we do as a librarian and how can we change our terminology to make it actually create a tool that does those things that we want to do? Um so, we focused on iterative changes and improvements. Chat GPT also provides great documentation ongoing. So, when I inevitably broke the tool, um I was able to go back and fix the code base. Um but then, you know, we very quickly hit some limitations both in our prompt engineering. This is first time I was really using Chat GPT. Um and then the code generation. This was Chat GPT-3. It was a lot less robust than current versions of Chat GPT. Um so, we really needed to uh kind of pick things up from there. Um what we did get from this collaboration was our beta version of the tool, which you can see here. It's very high-tech. Um it gives you, you know, a rundown of these five easy steps you have to follow through our user interface, um prompting you to upload the metadata you want to look for harmful terms in, upload a lexicon with those terms that you want to search for, and then kind of compare the two files against each other. You could select which uh metadata fields you wanted to look into, knowing that some fields just can't be changed. Um and then you could also select categories of terms from uh the lexicon as well. It would send you a report and give you an idea of things you might need to tweak. Um, it had a lot of issues. It also didn't work with Mac, so that was a problem. Um, but we were still focusing on this human in the loop thing. So, thinking about um, you know, when you're working with an AI system, what can the human bring to the AI to improve the the um machine learning model, and we wanted to keep that kind of mentality going into Marmit as well. So, we did that in the in the prep of the beta tool, but then we also kept it an integral part of how we use the tool moving forward. Uh human review is essential. It's a human-central tool. Um, we want you to still be uh working with the descriptive metadata professionals who know what they're doing, who can identify those terms and determine whether or not that term is appropriate to be used, is it correct in context or not. Um, so that remediation process still needs to happen with the metadata professionals in the repository itself. Um, so a little bit of our evolution. Um, we started developing this in the summer of 2024. By October, we were able to present it at the DLF Forum uh with our our beta version, and we got a lot of really fantastic feedback, which allowed us to go forth with this application for an internal grant. Um, so we were awarded $10,000 for that, um which we were allowed to uh then take forward and make significant improvements um with the help of our superstar student Aidan here. Um, so as uh Rachel mentioned, we've had him for about a a year now, and he's done some really fantastic improvements, uh completely rewritten the code base, uh and integrated some features that we hadn't even conceived of at the start of this. Um, so this small internal grant we are forever grateful for, um cuz it allowed us to really scale this project in an important way. >> Yeah. >> And I'll also mention it was a great time to get a small internal grant. >> Yeah. >> Um So now I'm going to take you through the operation of the tool as it the the new and improved available tool today um and go through a quick demonstration and then talk about the lexicons a little bit which are really you know, I could talk forever about the lexicons. I'm sure we can have a whole thing, but you know, I'll just give a high-level view. So this is what the tool looks like now. If you remember what um you know, Kaylee demoed in the beta version, I think this is a lot better. Um we have the code was re-written in PyQt6. Um so this is what it'll look like when you open up now or this if your computer is in dark mode because we now have dark mode capabilities which is really a really exciting the most exciting thing. So now I'm actually going to go through a recorded demonstration in less than 3 minutes of uh just the basic steps to so I can show you how easy this is to use um and assess a spreadsheet of metadata. So um you'll get a pop-up when it starts with some instructions. You can also set your folders of where you want, you know, your always to push and pull to. Um the red buttons lead you to the next thing. So I clicked next to run Marmot and then I'm going to select the delimiter of my metadata spreadsheet. We use tab-separated values at the Marriott. So I'm going to do tab and I'm going to load my metadata and here I'm going to select a collection from all you Oliver Early bird photos cuz I know it's a very spicy collection and I get a preview of all the metadata on the screen. The red button leads me to the next and this is where I'm going to load my lexicon. We provide three. I'll get into those in a minute and I'm going to provide um select the reparative metadata lexicon. You get a preview. Again, click the red next button. This is the configure analysis page which is probably really hard to see from where you are, but you get to select an ID. You always have to have an identifier, and now you can include a collection title if you wish. So, if you're doing a large set of metadata with hundreds of collection, you can get that returned in your results. And then you can select which metadata columns you want to analyze, so you don't have to analyze your whole spreadsheet. You could if you wanted to, it might take a while, but you can pick which one. So, I click title and description. And then I can pick which categories of the lexicon I want to use. I'm unchecking some of these British English ones here, and then you click next. Select where you want your file to save, cuz from a spreadsheet, you're going to get a spreadsheet. That's how it works. Um but uh click okay, and then hit the perform analysis, and it usually takes like a second as all those um scripts are configuring, and then you'll get a red bar that'll zoop just kind of, you know, running through the process, which is a fantastic sight to see. That was a lot to get installed, actually. Um and then you can click okay. And so, what we did as of yesterday in version 2.7.0, we now have um a dashboard with the results where it'll give you the flagged instances, um just a high-level dashboard of your results, also some data visualizations. We're calling this the data visualization version. Very exciting stuff. Um and then that's like the high-level load statistics, but the real, you know, output is this uh spreadsheet uh report, which flags and I will open it up here. It's not fun to look at, so I apologize for that, but it, you know, it's a spreadsheet, but it flags the term that was flagged, the category from the lexicon that that term came from, the column metadata, and the original text. So, from here you could do a lot of filtering on terms, filtering on categories, and then maybe even doing some revision and inputting it into your system. Just going to check time. So, that's it in a nutshell and we've done it on some pretty large sets of metadata well over 300,000 items and it's it's pretty awesome. So, I think now we're going to talk about the lexicons since they need to be mentioned this is really the heart of the reparative component of the tool and these are directional. We provide three key pre-curated lexicons. Users can create their own lexicon and they can also edit these lexicons before they upload them. They're CSV files. So, it's pretty low-tech but the three lexicons that we have are reparative metadata lexicon which is really offensive. There's you know over 630 terms and these include some very offensive terminology but also some more nuanced terminology as well. Please open it up and take a look but take caution when you do. We have the LCSH lexicon which is going to flag subject headings that have been recently changed or canceled that relate to people and culture within the last few years from the Library of Congress but also those deemed problematic by the community and the cataloging lab. And then lastly, we have a sensitive content lexicon and again, these are all directional but if it has a number of categories to kind of call potentially things you might want to take a second review at to either remove from online or put on a graphic content warning. And we've done a lot of work. There's an indigenous sensitive one as well that we've been working to hopefully flag things that may be in violation with NAGPRA. So, checking my time. Okay, so reparative metadata lexicon this these are the the list of categories to kind of give you an idea of how robust the different categories are all the way from you know, a grandisement terms, slavery terms. It's a lot Um and I think this category is one really good example of using existing um work from the community on reparative metadata. So, um I you know, as I mentioned earlier, one of our areas uh sensitive content areas is documentation of the Japanese Americans during World War II. Um and I used a very robust list of outdated terminology that was available through Yale's reparative archival description um finding aid to create a very basic list of terms and it's been really helpful in reviewing our collection and some flagging terms uh within our collections on this topic. Um the LCSH lexicon I don't know how much I can say here, but you know, it's helpful when you don't have automated authority control to go and look at your subject element and run it against this category and then this again the sensitive content lexicon. And again, this is directional. I know every institution has its own policies and guidelines about what to do with different types of material. Um so, this is just again, directionally, I think you'll get a lot of false positives, but it's more like a filter through all your metadata and then a human as we mentioned is in the loop and reviews it. Um yeah, and then future directions. >> So, just uh some very brief future directions. Um we have to publish an article on this because that's what we do. Um we will eventually get to that. I'm on parental leave, so after that. Um and then uh we're really interested in increasing community engagement. So, it's just been the you know, the two of us and Aiden. We really want to get buy-in from other uh people who are doing this work, community engagement specifically to identify terms that we're just not aware of. Um you know, a lot of the great work that we've done, it mitigates that disaggregated term that we all know is bad and we can eliminate that and helps you find those more nuanced things, but then we also need to expand our our knowledge base as well. Um we are seeking some grant funding uh as always in bad times, but you know, hopeful uh to host uh what we hope can be a Marmot Summit here in Salt Lake where we can bring these kind of community members together to review the tool, critique the tool, and hopefully get some more input. Um and then we're also really interested in seeing what natural language processing might bring to the table here, too. Uh Rachel mentioned the high instance of false positives because there are certain words that are totally fine in some contexts and definitely not in the other. Um we think uh integrating some NLP uh capabilities into the tool will help kind of analyze the context for those terms and help mitigate those those false positives as well. So, that's some of the stuff that we really hope to do moving forward. Uh there's our contact information, and it looks like we have about 9 minutes for Q&A. We're on time. >> Thank you. Thank you. >> The real uh very interesting project. Uh one of the things I was thinking about as I was watching this uh your presentation was what else this could be used for besides this reparative metadata? Maybe just cleaning up metadata somehow. Now, have you thought about about other uses for besides just the reparative metadata that you can use it? >> Yeah. Yeah, absolutely. Um that's why we wanted to really create this skin diagnostic tool that just functions on basic tabular metadata CSV or TSV. Um and uh you know, we really see it as something that helps with this because this is the way we, you know, saw the initial use case. Um but really any sort of like collections as data project can benefit from this as well. If you create a custom lexicon and you want to search, you know, for instances within uh your collection that maybe isn't being picked up on traditional search terms as well. Um the bulk assessment for other needs, too. How robust are your metadata and where do things need to be filled in? So, we see it as having really broad application, which is also why we want to have that community buy-in and and input as well. Yeah. >> Yeah. We we recently did a project where you had to clean up uh basically the history of our school. Like we like the different department names. Oh, it was let's just say a mess uh to put it nicely. Uh and that was a huge project for our for our digital services team to do that. And I think a modified version of this tool probably could have helped. >> Or even just to get the output and put it in Open Refine and then, yeah. Sure. Yeah, you wouldn't even you could just use this on Sure. It's a Yeah, you you could use the tool as it is. You would just create a list of terms for your custom lexicon and just upload it as the lexicon and then do the analysis. So. Yeah, the instructions are on the website, but I would love to hear more instances outside of reparative that it could be used for as well. So. It's basically a giant VLOOKUP with Python scripts that'll give you assess a huge amount of metadata and give you a report based on, you know, what you your list of terms in your lexicon. >> Thank you. >> Yeah. >> Um, well, I know this has been out for a while. I was wondering if you could share some examples of um folks in the field who are using it and how they've started using it um other than how we've used it at the University of Utah. >> Um, sure. So, at the I have heard I've heard rumors of um folks using it for their collections um at the I think it's at Louisville University. They were using exports from ArchiveSpace into a CSV file and then being then we're able to assess finding aid data metadata from like the archives export to CSV. Um and we've we've heard from a number of people who've used it. Oregon Digital really gave a lot of input on the um revised lexicon uh for the reparative metadata lexicon. Um and I've gotten some feedback too for this last iteration of the lexicon that went up. We've heard from the British Library who assessed 6 million records >> On the first >> on our beta version. So, we haven't told them about, you know, the latest one, but yeah, I would love to hear more. So, if you're working somewhere and you know that this tool's being used or if you can suggest it, but we'd love to hear any kind of feedback is so helpful for us. So. >> Go ahead. >> Um Justin >> Group up in the sun. >> Yeah, here we go. Uh Justin from the University of Michigan Library. And one of the things we've been looking at with our metadata is trying to understand what um what level of accessibility might already exist within things within our collection. We looked at a specific mark field and it was like less than 1% actually had something um like catalog. And so, I'm wondering about how we could utilize this to get a better understanding of that and whether or not there actually terms that could search and try to get a better understanding of where those might be or help us prioritize things cuz of title too. >> Mhm. Um next. >> Yeah. Yeah, I mean, I think the hardest part is the identification of the terms that would be kind of cues for you um where you're missing things, but um you know, I can imagine maybe we can chat more, but you know, um doing some sort of bulk analysis on like a qualitative analysis of the terms that you have and seeing if there are certain patterns in the data that would give you maybe an indication of those terms that could form a lexicon and then find the gaps. Yeah. The lexicon building is the hardest part. >> But >> Also fun. I don't know, can be. >> Yeah, and so yeah. >> This is the thing about language. >> Yeah. >> Okay. Any other questions? >> Well, if not, there are 4 minutes that you're able to regain if you like. We'll hang out here if you'd prefer to not talk in front of a crowd and we do have very cute marmot stickers if you would like one.