To that end, there have traditionally been two primary ways for data retrieval. In musdb, one could run batch exports and receive a ZIP with some form of XML files. One per object, with the objects matching the results of any given object search.

On the other hand, there is the public API. Using URL manipulation, one can access the (primary) contents of each page in a machine-readable way. To access the JSON representation of an object’s published metadata, where the object’s ID is 7141 in the Hesse instance of museum-digital (URL: https://hessen.museum-digital.de/object/7141), one simply has to insert json to the path: https://hessen.museum-digital.de/json/object/7141.

Next to the default JSON output, additional APIs are offered wherever suitable for a given data type. For objects, the primary additional output method is a LIDO API.

Thus far, the main limitation of the public API was that it only allowed one object (or institution, collection, etc.) to be queried at a time.

Querying Object Metadata in Batches

After a significant refactoring of the code to load object data for object pages – primarily to improve caching and allow for parallelized requests to the database – we are now finally able to offer APIs for querying object metadata in batch. Thanks to grouped database queries, performance and resource usage scale nicely. Taking simply the currently most recent objects in the Germany-wide instance of museum-digital: Loading all object data of one object and presenting it in JSON takes 0.0087 seconds and loading and generating the JSON for the 100 most recent objects takes 0.197 seconds (or 0.00197 per object). Note that not all queries for all aspects of an object’s metadata are grouped yet, performance may thus get even better over time. This does also not yet account for the overhead of the many HTTPs requests one would previously need to execute to get each object’s metadata one by one – real performance improvements are thus even greater.

Now, how to access object metadata in batches?

The batch access is linked with the search API and reuses its main query parameter (“s”). Say, if one is searching for objects related to Berlin (a.k.a. the place of the ID 61), the URL of the respective search page would be https://global.museum-digital.org/objects?s=place%3A61. The corresponding API for retrieving all of the objects’ published metadata would then be https://global.museum-digital.org/export/json/place:61?limit=24&offset=0. Like the search page itself and its primary API (/json/objects), the full batch export API is paginated with a maximum of currently 100 objects per page being returned.

Currently, the batch export of full object metadata is available for JSON and LIDO (XML) representations of the object data. More can rather easily be added later, should a demand arise.

OAI

Implementing a performant way to export full object metadata in bulk was one of the two main missing components for the long-missing implementation of an OAI-PMH API.

OAI is a standard tailored towards data harvesting. Say, an external service like the German Digital Library or Worldcat wants to do something with external data from diverse sources, e.g. to also display objects or implement a search across the different collections / libraries to find which one has which object / book. To be able to do so, they need to be able to access the respective data in some way. Ideally, using a common standard that describes how to query data, helps to identify any data sets that need to be updated or added (or deleted), and finally presents a uniform way to access the data periodically. That, exactly, is OAI-PMH.

In a nutshell: OAI-PMH allows other services to copy all (published) data from another service in a maschine-readable way and can thus significantly improve reuse in aggregation. Of course this only applies to technical questions; legally, potential re-users need to comply with the metadata license applied by the initial data provider regardless of the (technical) means of access.

Since last week, museum-digital now provides a OAI-PMH API at /oai respective to a given subdomain. E.g.: https://hessen.museum-digital.de/oai. As of now, the OAI-PMH API provides access to the objects’ metadata using LIDO (XML) and the mandatory OAI-DC format.

Note that there are some caveats remaining for now: First, the LIDO representation of object metadata is not (and can by definition not be) as complete and fine-grained as the JSON API. It is also not exactly similar to the LIDO as returned by exports from musdb (one is formed natively in PHP, the other using XSLT, leading to divergent development paths). Also, the LIDO output lists different identifiers from the ones used by the OAI-PMH API and the OAI-DC representations otherwise.

Finally, the OAI-PMH API at museum-digital does not implement OAI-PMH data sets to group collections. Instead, it follows the existing search logic (essentially providing a new endpoint per query). Example searches via OAI might thus look as follows:

• All objects from the Agrargeschichte instance of museum-digital, represented in OAI-DC: https://agrargeschichte.museum-digital.de/oai?verb=ListRecords&metadataPrefix=oai_dc
• All objects linked to Berlin (place #61), in the Berlin instance of museum-digital, represented in LIDO: https://berlin.museum-digital.de/oai/place:61?verb=ListRecords&metadataPrefix=lido
• All objects of the Freies Deutsches Hochstift, Frankfurt am Main, represented in LIDO: https://hessen.museum-digital.de/oai/institution:1?verb=ListRecords&metadataPrefix=lido
• All objects from Baranya with only their identifiers: https://ba.hu.museum-digital.org/oai?verb=ListIdentifiers&metadataPrefix=lido

Credits

Credits where credit is due: the Städel Museum deserves praise for their implementation of OAI-PMH. For me personally, seeing the Städel’s API was the first time I saw a visibly stateless OAI-PMH implementation, enabled by the ingenious idea of using machine-readable, JSON-encoded resumption tokens over UIDs that have to be resolved on the server side. I spent weeks (or months?) making museum-digital’s frontend stateless. By following the Städel’s example, it can remain so even while offering an OAI-PMH API.

This content is licensed under a Creative Commons Attribution 4.0 International license.

On the other hand, it’s a very clearly defined situation, that is repeated again and again in a very similar pattern. Which is to say, it is a perfect subject for automation.

Now, some years ago, a working group came together in the context of CIDOC to do exactly that. Since not all museums use the same collection management system, the necessary first step towards automating the exchange of object information was obviously to develop an open standard for the same, that could be implemented in the different collection management applications.

The standard thus developed – EODEM, short for Exhibition Object Digital Exchange Model – is now mature enough to have gone into a public beta release phase. We’re very, very late to the party, but it was time to try our hands on implementing an import and export for EODEM. The import had already been done by November, but by the time of the larger update and re-design of January, we also managed to release an EODEM export.

This blog post will continue explaining how to import and export EODEM data before closing on a general perspective on the usefulness of EODEM at this point in time and in the foreseeable future.

How to Use EODEM in musdb?

As with loans in general, there are two situations in which one might want to use EODEM. The receiving museum can save a lot of time by importing EODEM data, while the sending museum can help out colleagues by exporting it.

Importing EODEM data

In the (German language) handbook, one can read a lengthy explanation of how to import object data into musdb (a less extensive description can be found in the blog as well, this time in English). In short: One first needs to generate a password to access the import folder using a WebDAV client. After mounting the folder, one can proceed to upload the data. Finally, a configuration file needs to be written to enable the import tool to identify the format of the import data, a mail address which is to be notified when the import is done etc. As the server checks all museums’ import directories for available import data every four hours, the object information will then be automatically imported after some time.

EODEM was developed as an application profile for LIDO 1.1, meaning that it builds upon and extends the LIDO standard for exchanging object information. As such, we could simply integrate EODEM into our regular LIDO import. Hence, users who want to import EODEM data need to set the “parser” setting to “Lido” in the configuration file.

Depending on the way a museum forms its inventory numbers, EODEM however may present a problem regular LIDO imports do not. If an inventory number in the sending museum (say, in the import data) equals an inventory number of one of the receiving museum’s own objects, the importer would interpret the duplicate inventory number so as to update the existing record of the inventory number rather than adding a new one. In response to this problem, we added a new setting available in the LIDO parser: prefix_inventory_numbers . By entering a value (e.g. the reference number of the loan) to this setting, the given value will be prefixed to the imported inventory number so as to prevent collisions and allow importing objects safely.

Exporting EODEM data

EODEM data can be exported from musdb the same way any other XML export can be run. It is now listed as an additional export format in the regular export form. The export form is accessible either through the navigation for exporting the whole collection – which obviously makes little sense in the case of EODEM – or on the search results page once at least one search parameter is set. One can thus filter the collection for all objects of a given loan, click the export button in the sidebar, possibly adjust the list of exported fields (e.g. to exclude a field one does not actually want to export, but which is usually covered by the EODEM export). Finally, one will receive a ZIP file containing the EODEM-encoded object information. This ZIP can then be sent to the receiving museum.

To both promote and simplify the exchange of loan object information using EODEM, a shortcut has been added in the sidebar of loan editing pages. Clicking on “Objects (EODEM)” in the export section of the sidebar directly takes one to the EODEM export settings page for all objects of the given loan.

In musdb and Beyond

As stated above, we were very late to the party. EODEM has been in development for some years, but we only started seriously looking into it in late 2022 (all credit for developing EODEM thus goes to others; the working group deserves a lot of thanks!).

On the other hand, it turns out that musdb is the first collection management system to actually feature an EODEM import and export in production. For the time being, exchanging EODEM information thus mainly helps in the case of loans between two museums that both use musdb for collection management.

The working group’s page however provides a list of software vendors who plan to implement EODEM imports and / or exports. Many of the big names are already listed there, meaning that a timely implementation of the standard in many of the major collection management systems is almost given.

On the other hand, there are many, many more collection management systems than those ten listed at the time. While EODEM is based on LIDO and thus very simple to implement if a collection management system already features a LIDO import / export, it remains to be seen how many will eventually adopt the it. Hopefully it is many, so that we can improve interoperability and soon enough all save some time when working with loan objects.

Image credit: “Relief zweier sich die Hände reichenden Männer, Seebronn (?)“, CC BY-SA @ Landesmuseum Württemberg

This content is licensed under a Creative Commons Attribution 4.0 International license.