Planning and Executing an In-House Repository Migration: Opportunities, Challenges, and Successes

I'm Sean Murphy and I serve as the head of systems and open infrastructure at Illinois Tech's Paul Vgan Library. And I'm Adam Stro, the library's director of distinctive collections and digital strategy. Today, we're here to talk to you about our team's experiences conducting an in-house institutional repository migration. Out of necessity, this project was carried out without assistance from a vendor, and we hope folks in our peer institutions who share our constraints will find this video helpful. This presentation builds on one given at the spring 2025 CNI meeting, which outlined where we stood prior to this migration, and we'll touch on some of the same issues and questions today with some updates on our work and some notes on where questions remain. In this talk, we'll briefly touch on all aspects of the project so far, and we will try to highlight aspects that we wished we had known along the way and had to learn the hard way. We want to cover the history of our repository across its iterations and how they have informed the current migration project. And this will flow into a closer look at the impetus for the current migration and why we're doing it inhouse. Then we'll talk about the migration workflow itself. We want to cover this topic broadly. We'll include only major highlights and maybe some triumphs that we're particularly proud of. We'll wrap up as these talks often do with a reflection on our challenges and opportunities and our aspirations for the next steps. Illinois Tech's institutional repository began as a dspace deployment 2010. In addition to serving as a traditional IR, it also became a sort of unofficial digital asset management system for Illinois Tech with a complex collection hierarchy, complicated access permission models that required ongoing maintenance, and a not insignificant number of repository objects that were inaccessible to most of the university community. The DSpace self-deposit model added the extra complication of uncontrolled and inconsistent user submitted metadata across the repository. When we move to Islandora in 2019, we hope to mitigate these issues, streamline repository organization and open up access across the repository as much as possible. Islandora's integration with Drupal helped to simplify permissions and our team moved to a mediated ingest model to help bring more order to our metadata. We also began hosting digitized collections material from the university archives in Islandora alongside other repository content which required a pivot in our collection structure and some additional considerations when it came to object metadata and access control. So why are we migrating again? To start we should clear up some terminology for those who are less familiar with Islandora. The version that we deployed in 2019 was built on Drupal 7. So it's commonly referred to as Islandora 7. Just as we finished that migration, Islandora released an architectural rewrite that was radical enough to refer to the system as Islandora 2 going forward. This rewrite prevented us from simply upgrading our repository incrementally. Moving to Islandora 2 would require a full-blown data migration. Furthermore, Island 7 was locked to server infrastructure that was becoming increasingly outdated, enforcing a ticking clock for the project. We had limited server space which had begun to impact our ability to do things like large-scale digitization projects, work with audiovisisual content or accept deposits of large data sets. Happily, Island 2 deepens the Drupal integration which offered some distinct advantages. Control over metadata fields and taxonomies is much more flexible. We're also able to deploy the site using Docker, offering significant operational advantages for deployment, upgrading, and backups. Islandor 2 also lays the groundwork for future enhancement projects like link data enrichment, which we'll talk about later. A project like this is often done with the support of an external vendor, but this option was unfortunately unavailable to us at our smallish institution. We therefore proceeded with migration planning inhouse with a bit of grit and naivity. This project has been a team effort with contributions from a group of five librarians. This also meant that there was no one person for whom the project was their sole focus and work on the migration took place amidst all the other things we were also working on. After initial planning, we had to secure server infrastructure and conduct the baseline islandora 2 installation. Since this was essentially a clean migration between repositories, we had to follow a full ETL procedure. We extracted metadata and media from Islandora 7, transformed it to conform with the newora 2 data framework and loaded using the auxiliary programora workbench. User interface design of the new repository was conducted throughout the project, including user experience and accessibility audits. We plan to do a final review and user testing before we go fully live. Finally, we plan to incorporate some exciting enhancements to our repository both prior to our go live date and in the following months. At the time of this recording, the migration process is not quite complete. We have done significant theming on the site and have moved most of our objects over. There is light at the end of the tunnel though and we will discuss our next steps in a few minutes. Planning for this migration necessarily started with some research so we would know what we were getting ourselves into. Some of us attended Islandoracon and we studied the migration experiences of peer institutions. Luckily documentation for Islandora 2 is robust and we reviewed it carefully. In contrast with Islandora 7, Islandora 2 uses Drupal fields for object metadata. We were previously constrained by using mods fields. This meant, however, that we had to create a map to reimagine our metadata model from the ground up, including transforming hierarchical mods elements into simpler Drupal fields, identifying which fields ought to be made into Drupal taxonomies, moving values from more general mods fields into more specific taxonomies, and identifying opportunities to normalize existing metadata. We also decided to use a hybrid approach to taxonomies with uncontrolled user supplied terms in their own taxonomies and a plan for authority controlled terms to be added after we launch. And we'll talk about that a little later in the presentation. We needed to apply the existing university web theme and specification in a way that was appropriate to an island repository. The migration team included the library's UX librarian Aen Lopez. So their expertise was applied to the new repositories appearance, theming, and accessibility and helped to highlight subsequent iterative work we'll continue to do on these fronts after launch. Many specific Islandora features such as compound object relationships, access control, and solar search integration are now more streamlined into existing Drupal functionality. This was ultimately a positive, but we needed to do a lot of thinking and experimenting to find ways to satisfy our existing functionality and data needs in this new system. Stepping back further, we had to consider the repository as a whole when determining the order of ingest for existing objects. For example, a collection in Islandora 2 is indistinguishable from a parent object with many children. So, we made sure to ingest all collection objects first. Before data migration could even begin, we had to have a destination for it. The migration team worked with Campus IT to secure an Iuntu server VM for development. For data storage, Campus IT set us up with a two TBTE AWS network file system and they connected it to our server. We then configured Fedora's file system to use that NFS. Following along with documentation, we installed the Islandora 2 Docker image onto the server. We found distinct advantages with this method of deployment. Each component of Islandora 2 runs in a separate container along with all the microservices. This includes Drupal, Solar for search indexing, Maria DB for the database, Fedora for the repository itself, and other services like OCR generation and media display. The setup is easy to deploy and to keep updated. Debugging is also a bit easier as each container's log file can be accessed quickly and individual containers can be restarted as necessary without restarting the whole image. Our repository's codebase can also be backed up onto DockerHub. The Docker config file for Islandora 2 contains profiles for a development version and a production version of the repository. When in dev mode, live theming and content edits appear on the site. Switching to the prod profile starts the site in readonly mode for security. We've been developing our site using both dev and prod profiles on the same server which deviates somewhat from what the documentation seems to expect. However, when we go live, we expect to maintain a separate production server alongside the development server. Another advantage that the Dockerbased structure offers. Both our development and production servers will connect to the same NFS for access to the Fedora data. Actually getting our data from Islandora 7 into Islandora 2 represented a large amount of the work for the migration project. As you'd expect, we built some huge solar queries to extract object metadata from Islandor 7 into JSON. And we used Islandor 7's built-in data stream export module to gather media in batches. We generally extracted both metadata and media on a perolction basis. As we mentioned, it was necessary to map old mods fields to new Drupal fields, and we had to normalize, disambiguate, and coordinate values across a messy data set. We wrote a series of data transformation functions in Python to accomplish this. Taking raw JSON exports as input, the Python scripts cleaned up the data and field names, exporting the results in formatted CSV files. For some repetitive and simple data transformations, we used Mercury, an open platform for publishing web applets as Jupyter notebooks. File name preservation was a speed bump for us. When exporting media from Islandora 7, all file names were overwritten with a normalized Islandora schema. The file names were luckily mostly preserved in an obscure metadata field, but additional work was needed to properly perform the actual renaming of files. Thanks to the user supplied nature of some of the files, we had a lot of duplicates. Imagine how many students submit their thesis PDFs as simply thesis.pdf. We had to figure out how to handle all this in a way that ibuntu as the root file system, Fedora as the backend repository, andora/rupal as the front end could agree on. We usedora workbench to load all of our object metadata and media intoora 2. Written in Python, this standalone program saved us a whole lot of time in anguish. It connects to Islandora using the Drupal API and it streamlines most of that ingest process. The CSV metadata files that we generated using Python were tailored for Islandor Workbench according to its robust documentation. Thanks to Workbench, the actual ingest step was the least painful part of the whole process. While we were able to preserve some of the basic theming between Islandora versions, we had to adapt our theme to Islandora's new ways of presenting objects, media, and metadata. It wasn't as simple as dropping an existing Drupal theme onto a new site. Having Drupal as a foundation provides us with access to its large library of open-source modules. Some site functions that will be achieved this way include integration with campus SSO URI generation using handle and a framework for handling embargoed objects. The specific needs of an Islandora repository, however, are not always simply met by some of the more general Drupal modules. Our repositories content also demands a specific access permission structure. One example is the thesis and dissertations collection. We want the thesis themselves to be accessible to campus affiliates with authenticated access, but we needed to expose their descriptive metadata to anonymous users. Archival objects, on the other hand, might restrict accessing original files to all but repository administrators while allowing free download of derivative media. Figuring out the proper recipe to achieve our various access and permissions requirements has been a challenge that has required more than one Drupal module and a lot of thinking and testing. We are also planning to incorporate accessibility standards as a matter of practice rather than remediation. Our UX librarian has been auditing the site as it develops and their recommendations are iteratively implemented as is feasible. For example, we're exploring ways to expose an object's extracted text on the page to make it more available to users with screen readers. The main goal of this migration was simply to ensure the persistence of the repository, but of course, we're seeking to incorporate enhancements as they seem possible. The project isn't quite over the finish line yet. However, there are a few things that we're excited to incorporate as enhancements when the migration is done. The migration team includes Quinn McKisok, Galvan libraries cataloging and link data librarian who has been working to enrich wiki data with terms pertaining to Illinois Tech, its history and its affiliates. This will form the groundwork for more robust link data enrichment for the IR going forward. Like our approach to accessibility, we hope to make link data practice rather than remediation. We're also planning the creation of faculty pages inspired by the University of Prince Edward Island's Island Scholar Repository using Drupal taxonomies to create a one-stop shop for a scholar's bio publications and more. Possibly even eliminating the need for some faculty members to manage their own web pages. Finally, as previously mentioned, we're looking forward to a true dev prod's theming and content without having to put it into maintenance mode for long periods of time. Alongside this, we plan to leverage version control systems like GitHub and DockerHub to back up our site data and further smooth out server downtime in the future. Of course, when the migration is complete, we'll need to develop strategies for promotion of the new IR outreach to encourage new adoption and integration of Islandora into campus scholarship and research. Today though, we want to talk about a few of the more technical considerations that we're still working through as we near the end of the migration process. Site performance, especially with regard to searches, can be slow and inconsistent. We have some ideas to help ameliate these issues, but we'll also need to cross our fingers a bit and hope that further tuning will be successful. Broadly, we've been able to impose the access control paradigms that we needed. Our multimodal approach, however, means that we still need to identify and remediate cracks in the system where two modules may not naturally interoperate or more manual work may be necessary. We've had luck identifying existing Drupal modules to meet the majority of our needs. The relatively small Islandora user base within the larger Drupal ecosystem, however, means that we're sometimes forced to adopt Drupal modules in creative ways. To date, there's still a few features for which no reliable or broadly available Drupal modules exist. Finally, user accessibility needs are not always easily met by Drupal and Islandora as they work out of the box, and we'll continue to explore new options and rely on the expertise of our fellowor users to make headway in this area. We would like to thank our colleagues, past and present, who contributed work to this project. If you have any questions or advice, please feel free to reach out to us via email. Thank you.