Over two weeks, at the end of April and start of May, Museum Data Service and Collections Trust hosted a student placement with UCL, where Masters students as part of the Digital Humanities programme worked on a discreet project with a host organisation. As part of my placement work, I focused on a data enrichment project, using data from the Mapping Museums Lab (MML) at Birkbeck, and comparing it with collection-level records within the MDS. The main aim was to identify museum records that appeared in MML dataset but were not currently found in MDS, and to assess whether any of them could become potential new MDS records.
The Mapping Museum Lab is a culmination of multiple research projects, looking at collecting and analysing data about museums, their governance and changing patterns of openings and closures since the 1960s. They have a fantastic open data set available on their website – https://mapping-museums.bbk.ac.uk/about-mapping-museums-database/
The comparative work was not a simple one-step matching task. Museum records can vary across datasets because names, identifiers, accreditation numbers and closure years are not always recorded in the same way. Some museums have clear Wikidata IDs or accreditation numbers, while others require more detailed manual checking. Because of this, working with the Arran, the MDS’ Museum Data Manager, I used a combination of Excel, OpenRefine, Wikidata reconciliation and manual online verification.
- Cleaning and comparing the datasets in Excel
I began by backing up the original MDS export and MML files, so that I would not edit the source data directly. The next step was the do some data cleaning work. In both datasets, I cleaned the museum name fields to reduce small formatting differences. This included standardising capitalisation, removing extra spaces, and making small changes such as standardising the use of “&” and “and”, as well as different apostrophe styles. I then brought the cleaned versions into one working file, museum_comparison_working, with two working sheets: MM_cleaned and MDS_cleaned.
For the first round of comparison, I added helper columns in MM_cleaned (see Figure 1). I used the Excel XLOOKUP function to compare MM_cleaned with MDS_cleaned and to check whether each Mapping Museums record could be found in MDS. I matched the records using the most reliable fields first: Wikidata ID, accreditation number, and cleaned museum name.
Figure 1 MM_cleaned after adding helper columns and performing the first comparison.
This helped me divide the MML records into three broad groups: records found in MDS, possible matches, and records not found in MDS.
After the first round of matching, I focused on the records that could not be found in MDS by Wikidata ID, accreditation number, or cleaned museum name. I then used the closure year information in the MML data to separate these unmatched records into different groups (see Figure 2).
Figure 2 Excel sheets used to separate unmatched Mapping Museums records by closure status.
Records with no closure year recorded were marked as Not found in MDS and exported as the main group for further checking in OpenRefine. These were the records most likely to include potential new MDS records, but they still needed to be checked manually before any decision could be made about inclusion.
Records with a closure year were placed in the Closed_not_in_MDS sheet, while records with conflicting closure details were placed in the Closure_conflict_not_in_MDS sheet. This kept the OpenRefine stage focused on records with no recorded closure year, rather than mixing them with records that were already closed or had uncertain closure information.
- Using OpenRefine for reconciliation and manual checking
After the first Excel comparison, I exported the records with no closure year recorded and continued the work in OpenRefine. I used OpenRefine’s reconciliation function to try to match museum names with Wikidata IDs automatically.
This was useful, but it did not solve most of the problem. In practice, only a small number of records could be matched directly. Many museum names were too ambiguous, too local, or not represented clearly enough in Wikidata. Because of this, most of the remaining records still needed to be checked manually.
To make this checking process clearer, I added four columns: reconcile_name, entity_level, Wikidata_review_notes and verification_url (see Figure 3).
Figure 3 Added OpenRefine fields for Wikidata reconciliation and manual verification.
The reconcile_name column records the name that seemed most useful for searching. Sometimes this was slightly different from the name in the original dataset. The entity_level column records what kind of entity I thought I was looking at, for example a museum, a branch, a collection, a gallery, or something that did not seem to be a museum.
The Wikidata_review_notes column records my judgement about the record, such as whether it might need a new Wikidata item or whether it should not be treated as a separate entity. The verification_url column records the source I used to support the decision, usually the museum’s official website.
The Elizabeth Garrett Anderson Gallery is one example of how the manual checking process worked. It was not automatically matched to a Wikidata ID through OpenRefine, so I reviewed it manually.
I first searched for the gallery online and found its official website. I then compared the website with the MML record, especially the name and address, to make sure that I was looking at the same institution. After confirming this, I considered whether it could reasonably be treated as a museum or museum-like record, rather than only as a venue or temporary exhibition.
In this case, I judged that it could be included as a museum-like entity because it is a public gallery and permanent exhibition within the UNISON Centre. I also noted that I could not find a reliable Wikidata match. I recorded this judgement in Wikidata_review_notes and added the official website to verification_url as supporting evidence.
This example shows the wider process I followed for records that OpenRefine could not match automatically: checking the identity of the institution, assessing the type of entity, and recording evidence for the final decision.
- Cross-checking and validation
Once I had completed this stage, I sent the results to the Museum Data Manager at MDS. They reviewed the file and approved some of the records. I then carried out a second round of checking, responded to the comments, and revised the data where needed.
Another requirement for MDS was that each museum needed latitude and longitude information. To add this, I used https://www.freemaptools.com/convert-uk-postcode-to-lat-lng.htm to convert UK postcodes into coordinates (see Figure 4). These coordinates were then added to the final version of the dataset.
Figure 4 Screenshot of a website that converts UK postcodes into latitude and longitude.
- What I learned
This work showed me that comparing museum datasets is not just a technical matching exercise. Excel and OpenRefine were very useful for narrowing the records down, but they could not make the final decisions for me. I still needed to check official websites, compare addresses, and think carefully about whether a record represented a museum, a branch, a collection, a gallery, or something else.
It also made me realise why recording evidence matters. For many records, the decision was not simply “match” or “no match”. There was judgement calls involved, especially where a record was museum-like but not a traditional museum. Adding review notes and verification URLs made those decisions easier to check, discuss and revise later.
Post author: Haixin Huang