This process is called reconciliation. One of the most used tools to actually connect to external services and reconcile data this way is OpenRefine. But to do so, OpenRefine needs to be able to communicate with such services. The open standard by which they communicate is the Reconciliation API specification. Depending on how completely the API has been implemented, the consuming software may offer additional features such as offering alternatives, suggesting entries or displaying previews of entries made available through the API.

Since this weekend, museum-digital offers such APIs for different types of data.

In md:term, reconciliation APIs were added for each of the available controlled vocabularies. Besides the controlled vocabularies of museum-digital (tags, actors, places, times) this means that users of OpenRefine can now reconcile their data with select other vocabularies such as the Oberbegriffsdatei or the hessische Systematik. Besides implementing exact and partial matching of requested terms, this implementation features previews.

Two additional points are noteworthy in terms of md:term’s implementation of the reconciliation API specification.

1) Similar to museum-digital’s import tool, md:term’s reconciliation service attempts to clean up entered terms before reconciling. As such times are automatically rewritten (“1912 bis 1914” [1912 through 1914] becomes “1912-1914”), question marks and other markers of uncertainty are removed, and – whereever known – autocorrections stored in the city (Frankfurt a.M. > Frankfurt am Main) are applied to the terms.
2) Multilinguality: Whereas the current version of the regular reconciliation API does not consider multilinguality, the above-mentioned auto-corrections and rewriting are language-dependent. And so are previews and names of entities in md:term for museum-digital’s vocabularies. Hence, an additional parameter for the requested language can be supplied.

Reconciliation APIs for md:term may be found on the download page linked in the sidebar. Going this way, the language parameter is automatically set to the user’s current language.

Reconciliation by Inventory Number

To identify objects available published through museum-digital, a reconciliation API now exists in the frontend as well. It can be found at /reconcile/invno/ on each instance of museum-digital (e.g. here for the Hessian instance.
It reconciles entries by their inventory number. An additional number may be added to the URL to search only within a single institution. Where https://hessen.museum-digital.de/reconcile/invno/ allows matching objects by inventory numbers across all institutions publishing their objects on the Hessian instance of museum-digital, the https://hessen.museum-digital.de/reconcile/invno/1 will thus only search within the collections of the Freies Deutsches Hochstift / Frankfurter Goethe-Museum (the institution of the ID 1).

Reconciling GLAM data using the MD:term reconciliation API: an example

As they have much more experience with OpenRefine, I asked the colleagues from digiS whether they could provide a description of how to actually use the reconciliation API from a user perspective. This section is theirs:

Using the reconciliation API in OpenRefine is straightforward (cf. the OpenRefine documentation for a general introduction). Let’s walk through a typical workflow using GLAM data and the MD:term API!

Our sample dataset (adapted from actual data provided by the Sportmuseum Berlin) features, among others, columns on material/technique, object type and places.

These elements are just strings and consequently not linked to any authority file. Strings are not guaranteed to be unequivocal, nor are they particularly suitable for retrieval, programmatic reuse or multilingual contexts. In order for the data to be as useful as possible, let’s reconcile them with an authority file, in our case the MD:term vocabularies now readily accessible also via a reconciliation API.

If you have never used the MD:term reconciliation service you first have to add it to OpenRefine. Click on the small triangle in the header of the column you want to reconcile. Select Reconcile → Start reconciling.

In the following window, add the API endpoint you want to use (cf. https://term.museum-digital.de/downloads for a list of MD:term endpoints).

Now you can select the service for reconciliation:

In the following window, the only really important decision you have to take is whether or not to automatically match “candidates with high confidence”. Auto-matching speeds up the whole process, but you have less control over what happens with your data.

Hit “Start reconciling” to trigger the reconciliation process. You can now see that the strings have become links. In fact, they have been linked to possible entries in MD:term. If you choose the auto-match option, the most probable candidates have been selected automatically. If not, you have to select the candidate you want to match manually by clicking the arrow to the left. One arrow matches the value in the selected row only, two arrow match all identical values in the entire columns automatically.

Once you’re done with the reconciliation you can, for example, write the IDs of the reconciled values into a new column. Click once again on the small triangle on the left side of the top row of the column you’re working on. Go to Reconcile and Add entity identifiers column.

Name the column that is to be added and you will have a new column that contains the MD:term IDs.

Along with this, we can gladly annouce a new, updated and much more performant search API for md:term (see the API documentation for now).

You can access the documentation here:

• md:term
• frontend (via the global portal)

musdb

New fields

Other page

Batch updating object information

Instituion-wide settings and adding new objects

Notifications

In other news

• The calendar feature (accessible under the puzzle symbol in the main navigation) can now display tasks or make them subscribable via WebCal (thus implementing a “reminder” as had often been requested)
• PDF of all linked information is now in A4 and uses a two-column layout
• Ukrainian translation
• “Simple A5 PDF” now covers inventorization fields on rear side

md:term

• If two actors have been joined and one has an old links to the page of the actor now deleted, one is now referred onwards to the new, single actor entry. The same works with transferrals between vocabularies (an actor that was transformed into a tag, etc.).
• Ukrainian Translation

Frontend

• Ukrainian Translation
• JSON-based settings for specific institution pages have been removed
• Bulk download of object images
  • An overlay with a message from the museum can be displayed during batch downloads (see above)

CSVXML

• Almost completely rewritten
• The served page now is completely static and all checks and conversions run directly in the browser. This way, no uploads actually happen and the application is completely uncritical to the server’s security. On the other hand, this allows for installing CSVXML as a progressive web app and using it offline.
• We also added some explanatory texts did small updates to the design of the page. A footer now links to the source code and offers to refresh all cached contents of the page (this may be useful when visiting the page after a long time, as the whole application is cached in the browser for offline use).
• A new check also checks for the file encoding. A warning is provided if the data does not appear to be UTF-8-encoded.
• CSVXML is now released open source under the AGPL.

Importer

While the individual parsers for the different export formats are updated very often, the core scripts of the importer are very stable. November 2022 however came with a large update to these core sections, as more categories of data that had not before been covered by the importer (many of them new) can now be imported:

• Contact information (e.g. for object owners; loan partner institution)
• Object’s movement log
• Minimum and maximum temperature, humidity, and lux of an object
• Loans
• Events / Appointments

In terms of the parsers, we extended the LIDO parser to cover the new fields suggested by the upcoming EODEM standard for exchanging loan object information.

The probably most important part of md:term is its API, which is also – for example – used by the “graph navigation” and is available for external use as well. Since we – like anywhere else – do not require authentication for accessing the md:term API, we don’t have a full list of API consumers which we could contact about API changes. Hence, we hope that this blog post will also alert them to make necessary changes.

How then, is md:term made multilingual? And how does it express itself in md:term?

Months ago, we implemented the multilinguality in our controlled vocabularies by adding an additional translation table, which means that there is the base entry, which is entered by the museums. The “norm data editors” then enrich the data and also attempt to add translations to a separate table by fetching data from Wikidata. If translations in the user’s language are available, they replace the “base entry” when outputting the data.

In the API, the base entries are replaced as well. If they are used, it hence makes sense to explicitly state the wanted language of the entry. To fetch German language information on Berlin easily, one should thus query https://term.museum-digital.de/md-de/place/61/json?lang=de instead of https://term.museum-digital.de/md-de/place/61/json (which, before multilinguality, would have been the appropriate URL).

Alternatively, there is a new section in the JSON API: langs. All available translations can be found there listed by language.

This content is licensed under a Creative Commons Attribution 4.0 International license.