musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Here, as so often, in a list format:

Frontend

• Bugfix: The detailed description was not reflected in the object API even if set to be so via musdb
• Translation: The frontend is now available in Kannada

musdb

• Bugfix: Fixed error in setting up new two factor authentication via TOTP (app-based 2fa)
• Bugfix: Symbols for image rotation switched (counter-clockwise symbol rotated clockwise and vice-versa)
• Feature: If the generation of a PDF catalogue is triggered via the sidebar of series editing pages, the objects will follow their order within the object group

Importer

• If no explicit name/title for a resource (audio, video, PDF, 3D, externally hosted image) was supplied, there was no fallback. Now, the importer follows the same logic as musdb and will fall back to using the linked object’s object name as a resource title if none is supplied.
• The CSVXML parser can now handle multiple objects per XML file.

Auto uploader

The auto uploader is an entirely new tool. When first started, users will be asked to enter the necessary data for running imports. And they are asked for the path to a folder which will subsequently be monitored for automatic uploads.

Whenever the tool then detects suitable contents for an import in the folder, the data will be uploaded and automatically imported. Intermediate steps like the generation of a settings file for the import are covered by the tool.

The tool can thus be useful for museums that regularly import data formed in a consistent format – say, museums using museum-digital for publishing while mainting a separate collection management system. It should be uninteresting to those who spend much effort on preparing imports (e.g. via CSVXML) or who want to migrate to musdb altogether, as it only becomes useful in the case of regular use.

The code is available here, licensed under GPL v3.
See also the more detailed German-language blog post about the auto uploader.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Dates in TEI transcriptions are parsed, irrespective of whether when="" oder when='' was used
• Notes for markings are now publicly displayed
  This was missing thus far and is now implemented similar to how event notes are displayed. If a note exists, a small “[?]” appears behind the marking title line. Upon hovering over it, a tooltip appears with the relevant information.

musdb

• Names and descriptions of exhibitions and object groups can now be translated
• Version control
• Log of “current locations” of an object can be exported as a CSV file
• Uploaded object images can now be hidden or published in one batch operation
• API extended
  • (New functions)
  • Transfer object dimensions
  • List images and resources for an object
  • Image metadata
  • Publish / hide object images

January 2025

Frontend

• Objects can now be sorted by the aesthetics of the thumbnail (A dedicated blog post on this will follow soon)

musdb

• Batch transfer between between free text fields of object data
• Alignment of the maximum field length for notes on opening hours is now consistent between UI and database
• Bug fixed with switching between institutions during consistency checks fixed (relevant only to users with administrative access to multiple museums)
• Literature can now be searched by editors

Importer

• Core
  • Automatic transformation of life dates for actors
    • Year of death “01.01.2012” now becomes “2012”, instead of 01.01 as before
  • “?” and “(?)” are removed from the beginning and end of imported keywords
  • Various types of brackets in keyword names are converted to regular brackets
• Parser
  • Stricter internal implementation of settings, all imports can now implement the start_at setting
    • This is particularly useful for the repeated execution of imports that abort due to new, previously uncovered elements and other debugging.
  • New parsers:
    • MODS (mainly used in library contexts)
    • Parser for Exports from Faust for the Händel-Haus
    • Parser for XML dumps from MuseumPlus Classic (MsSQL > XML export per table > Import)
    • Bugfixes
      • Field “Verwender” in Primus parser was mapped to production events
      • Material / technology are now imported correctly in the parser for BeeCollect exports for the Industrial Museums of Saxony
• „Frontend“
  • CLI now also has options for switching off the import of individual areas
  • Help text for command line tool

nodac

• Splitting of keywords now also recognizes keywords that should be split into places, times, etc.
  • Bsp.: „Helm; Berlin“ > Schlagwort „Helm“ + Ort „Berlin“
  • Example: “helmet; Berlin” > keyword “helmet” + place “Berlin”
• When searching for keywords with ambiguous names, both keywords and generally ambiguous terms are now taken into account
• Times can now be merged with others directly from the time edit page

This content is licensed under a Creative Commons Attribution 4.0 International license.

Say, a museum’s previous database used actor roles over an event structure to express who created an object. As such, the museum entered that the object has a linked actor X that is linked to the object as a “main creator” and a linked time Y marked as the “creation time”. During the import, these roles (“main creator” and “creation time”) are then translated to museuem-digital’s event types to form an event: The object was created by actor X at the time Y. This works, because the terms “main creator” and “creation time” have been matched to the creation event type.

If a term is not yet matched to a corresponding value of a controlled list in museum-digital, the importer will simply abort the import. On the one hand this is a way to uselessly require resources for an import that cannot be completed anyway. On the other, it is tedious. One recognizes yet unmatched entries only one by one.

A Small New Tool

A small new tool, available at concordance.museum-digital.org, makes the process a bit less tedious. Users can upload all the import data from a given field (e.g. the actor roles) – one a line – and check whether they are already matched using the concordance lists or not.

For entries that are not yet matched, the tool will offer selection boxes to perform the matching using the graphical user interface. Once all entries have been matched, one can then generate the relevant lines of code to enter the missing entries to the concordance list upon the click of a button.

While simply checking and extending the relevant open source lists should be trivial even to most non-technical users, this way is certainly more convenient. Importantly, it also removes the need to run the import multiple times until one does not encounter errors caused by unmatched entries anymore. And, well, it certainly is also more convenient to match to regular human language values than to the internal IDs of the target values.

The concordance checker’s MIT-licensed code can be found here.

This content is licensed under a Creative Commons Attribution 4.0 International license.

On the one hand, using shared controlled vocabularies allows for a centralized editing team. Work that’s been done for one museum is available for all. This allows for a generally higher data quality, improved search options, etc. The only downside to centralized vocabularies is rules: Without everybody following a basic set of rules for the different vocabularies the vocabularies would surely be overcome by chaos quickly.

The two most basic rules are that 1) all entries need to be clearly identifiable and 2) actor names – from actual people to companies and governments to peoples – belong to the controlled vocabularies for actors; place names – cities, countries, streets – belong to the vocabulary for places, and so on. In effect, this means that an actor entry “Gaius Iulius Caesar” (better yet, “Gaius Iulius Caesar (-100–44)”) is a good one. It is clearly identifiable. An actor name “Bosch” should not end up in the controlled vocabularies – there are many people and companies sharing that name. If one means to actually record the German tool maker, one can specify as much using brackets and / or using the full name, e.g. “Robert Bosch GmbH” or “Bosch (tool making company)”.

musdb comes with selection lists enabling a simple reuse of existing vocabulary entries. Where a term or a name has not yet been recorded in museum-digital’s controlled vocabularies, the option to import vocabulary data from Wikipedia / Wikidata and the Gemeinsame Normdatei, and alternatively the need to add a description of at least ten characters enforce some level of identifiable and minimally researched entries.

And then there are imports. During imports, there is no way to force importing users to add additional data. To the contrary, especially during data migrations museums import data from different departments and different decades, leaving one with different spellings for the same person and a very varied quality of the recording overall. Importing thus often means bringing unclean, non-uniform data into the vocabularies. Often enough manual cleanup and enrichment are required to make the data usable – and to stop the “unclean” data from disturbing other users, e.g. by appearing in selection lists and then being falsely linked to another museums entries (a classic example are place entries like “Frankfurt”. Both Frankfurt an der Oder and Frankfurt am Main are significant places – how significant they are depends on where a museum may be located. Leaving the unidentifiable place (name) in the database will easily lead to other museums linking the same entry while actually referring to the other Frankfurt).

Automation can however significantly reduce the work of the colleagues manually editing the vocabularies while also allowing their efforts to have longer-lasting internal benefits to the system. The import tool thus features a number of checks to identify the entries actually referred to. Those checks relevant to the import of references to actors will be introduced over the rest of this blog post.

Rough outline

Actors from (or for) the controlled vocabularies usually enter museum-digital by being linked to objects via events. The same usually happens in the case of imports: The import data states that an object was, e.g., created by a given person. This translates to a new event (what happened to the object? And who did it when and where?) being created. The actor of this event is then set based on the provided name, optionally taking into accounts life dates and links to norm data repositories, in so far as they are made available in the import data.

Preparation

Cleaning the name

To identify the actor name, it is first cleaned and made slightly more uniform on a simple level of spelling.

• Leading and trailing spaces, semi colons, tab stops and newlines are removed
• Duplicate white spaces are removed
• Brackets of the different types (“()”, “{}”, “[]”) are replaced with simple brackets (“()”)
• Unwanted components / specifiers of a name are removed (e.g. “mythological creature” or empty brackets)
• Language-specific name components are replaced with their preferred variants in the vocabularies (German: “d.Ä.” is extended to “(der Ältere)” [the Elder])
• Indicators for uncertainty are stripped from the name, e.g. trailing question marks. They are separately used to identify the certainty of an actor link in a dedicated fields that describes exactly that.

Thus, a name stated in the import data as “; Hans Holbein d.Ä.?” will be imported as “Hans Holbein (der Ältere)”.

Parsing life dates from the input name

If no life dates have been provided explicitly in the import data, they may still be included as part of the actor name. Some museums whose database does not feature dedicated fields for an actor’s date of birth and death will append the years in brackets. Hence, a check is done to see if there are any brackets at the end of the name – and if so, whether the bracket contains a parsable time span. In this case, the time span identified will be used for the actor’s life dates and the brackets’ content will removed from the name (it is not actually a part of the actor’s name after all).

“Hans Holbein (der Ältere) (1465-1524)” with no explicitly provided years of birth and death will thus be transformed to “Hans Holbein (der Ältere)”, born 1465 and dead since 1524.

Identifying the actor

After preparing the inputted actor data, it is checked against the database in various ways. The listed checks below take place in a chronological order. If one returns a positive result and works to identify the actor, no following checks are run.

Checking based on norm data links

If links to external norm data repositories (the Library of Congress authority files, the National Diet Library, the French National Library, etc.) are provided in the import data, the actor vocabulary at museum-digital is checked whether it contains an entry featuring the same external ID. If that is the case, this actor is doubtlessly identified.

Checking global rewrites

For many names, there are different spellings. Similarly, many people use different names over the course of their lives. But the person referred to stays the same.

As not all names are significant enough to be listed as synonyms in their own right, museum-digital features a global (though language-specific) list of “permanent rewrites”. If a museum exported a reference to a “Julius Caesar”, this will be automatically rewritten to “Gaius Iulius Caesar”, thus identifying the person of the same name.

This central list is built and extended as the vocabulary editors do their work. Whenever two actor entries are merged in nodac, editors have the option to mark the name of the actor being merged into the other as to be permanently rewritten.

Checking institution-level rewrites

Whenever a new actor entry is added through musdb or by the way of imports, this is marked down in the database. Which museum used what term, resulting in which entry. Which entry may in the future be updated to refer to another. This is most simply explained using an example from the realm of places – the above-used example of “Frankfurt”:

If a museum newly brings “Frankfurt” to the controlled vocabulary for places, an additional entry is made in the database to note that the museum was the first to add an entry named “Frankfurt”, and that that finally resulted in the creation of a new place with the ID #99999. The object data itself makes it clear that the museum actually referred to “Frankfurt am Main” (ID: #217). If the vocabulary editors thus merge “Frankfurt” (#99999) with “Frankfurt am Main” (#217), this obviously cannot be a permanent rewrite. But the museum-specific log can be updated: The museum entered “Frankfurt”, and it referred to the entry #217 (Frankfurt am Main). If the museum thus enters “Frankfurt” without any further specification in the future, the importer and musdb can automatically identify Frankfurt am Main as the place actually referred to. The same logic applies to the import of actor data.

This check is obviously based on the assumption of some level of uniformity between different workers at the same museum. If a museum in Frankfurt an der Oder has almost all of their employees referring to Frankfurt an der Oder as the canonical “Frankfurt”, except for one, this will obviously backfire. But it is to be hoped that this case is rare enough – and that such a very different colleague would recognize their own non-conformity and choose to use the full names for both places to be able to clearly document (and communicate overall) with their colleagues.

Checking by name and life dates

If all the above checks did not work, the actor’s name and life dates are checked against the controlled vocabulary for actors itself. Does anybody with the same name, who was born and died at the provided years, exist in the actor vocabulary (either going by the language-specific translation of the name or by the “base entry” of an any language).

If that is not the case, the actor likely does not exist in the database yet.

Identification failed. Should a new entry be created?

If the actor does not exist in the database yet, they need to be added. Maybe.

Some names are known to be in clear conflict with the rules of the vocabulary for actors.

Checking the blacklist

museum-digital keeps a centralized blacklist for actor names that are known to be in clear conflict to the rules of the actor vocabulary. This list mainly contains either deliberately unspecific actor names (“unknown”; “unidentified”) or very unspecific ones (“John” [a random given name without a family name], “painter” [the name of a profession, not even of the actor themselves]).

If the entered actor name is contained in this blacklist, the attempt to link any actor to the event is aborted. The free-text note about the event is instead extended to mark that an actor of the blacklisted name was noted to be related to the event.

Checking if the actor is actually a place, time, or tag

A classic problem is vocabulary entries being entered in an unsuitable vocabulary – often for simple reasons likely colleagues slipping to a wrong column when working with a table calculation program. And thus one is asked to import an actor “14th century”.

The “14th century” is obviously a time, not an actor. For this, too, there is a centralized list that can be extended using nodac. Whenever the vocabulary editors move an entry from one vocabulary to another, they have the option to mark the entry’s name as always being of the target type.

Say the 14th century was previously entered as a tag. A vocabulary editor moved the entry to the vocabulary for times and marked the string “14th century” as one always referring to a time. If one then attempts to link a new actor “14th century”, the import tool can automatically recognize that this name actually refers to a time and link a time instead of an actor to the event. The attempt to add an actor is thus redirected to a different category.

All else failed: The actor is being added

If all the above checks did neither result in identifying a pre-existing record, nor in identifying the input name as invalid in one way or another, the actor is added to the controlled vocabularies and linked to the event. The entry now needs to be cleaned and enriched manually by the vocabulary editors.

New Developments

The last month has been an exceptionally slow month in terms of technical development around museum-digital. There are however some newsworthy tidbits.

musdb

Recording external IDs for museums

Museums, like all of us, are present in more and more databases. For linking different databases, it is useful – sometimes necessary – to know the ID of the same entity in both databases. Hungarian law thus mandates collection management systems, which are to be accredited for fully paperless use in museums, to allow storing a museum’s ID with the Hungarian Statistical Office. musdb can now do exactly that: Store external IDs of museums in a given list of external source repositories / databases.

The list of available external databases and the regular expressions to validate IDs in them are available here. For now, the list only contains the Hungarian Statistical Office, but the more options there will be in the future, the better.

More fields covered by “list results” and Excel export

The “list results” page (a.k.a. table view) for object search results offers the option to view the selected objects’ data in a customizable tabular format. Mainly because of that exact tabular format, it is not possible to display everything a full database view of the object offers using the “list results” page. Over the last month, we have nevertheless extended the list of displayable fields in the “list results” table.

Thus, it is now possible to display all translations for the object type, object name, descriptions, etc. of an object. Similarly, it is now possible to list all Weblinks linked to the object as a compiled (comma-separated) field. As the automated report generation and the Excel export tools build upon the same basic code as the “list results” page, these new fields are now also available in those cases.

New field for objects: “Last change of permanent location”

It is now possible to manually record the last date, the permanent location of a given object has been updated. This obviously makes sense for tracking relocations and re-organizations within the museum. Importantly, the field needs to be manually filled out, as the date of the last change of the permanent location may be far in the past or frankly unrelated to the database, even if there is one (say, the museum moves the depot to a different place, but the date is only later updated in the database via a batch editing process).

User interface

After there were some reports of people not seeing the difference between a valid form in musdb and an invalid / incomplete one, we updated the design of submit buttons. Submit buttons in incomplete forms are now blurred out, to add another hint at the incompleteness of the form.

Frontend

Improved performance and decreased bandwidth usage: serving images in webp

Over the last years, there have been a number of new image formats seeking to combine all the features of older and well-established formats like JPG and PNG with an improved compression. The most established format of that newer generation of image formats is webp, which is by now well-supported by all modern browsers and most local image viewers.

To save bandwidth and improve loading speed, we now store and serve full-sized webp versions of newly uploaded object images along with the regular jpg versions. If possible, the webp version is served on object pages. An additional benefit is the aforementioned support for features jpg files do not support, such as transparent image backgrounds.

iFrame embedding now only possible from whitelisted sources

If a museum wants to embed their data from museum-digital into their own website, there are traditionally two ways to do so. The better, but much more complicated option is to use the API to fetch the relevant data and present them in any way one wants. Much easier (and cheaper obviously) is embedding the museum’s data using iframes.

Unfortunately, iframes can also be used for attacks on users’ login data. We have thus now restricted this option to only allow embedding from whitelisted sources. If a museum wants to embed their data this way, this means that they now need to notify their regional administrators beforehand.

Importer

The importer now supports imports for exports from Startext’s HiDa in the configuration of the Saxon State Agency for Museums.

Other News

In other news, the Youtube channel has picked up steam, with some new tutorials both in German and Ukrainian.

On the other hand, it’s a very clearly defined situation, that is repeated again and again in a very similar pattern. Which is to say, it is a perfect subject for automation.

Now, some years ago, a working group came together in the context of CIDOC to do exactly that. Since not all museums use the same collection management system, the necessary first step towards automating the exchange of object information was obviously to develop an open standard for the same, that could be implemented in the different collection management applications.

The standard thus developed – EODEM, short for Exhibition Object Digital Exchange Model – is now mature enough to have gone into a public beta release phase. We’re very, very late to the party, but it was time to try our hands on implementing an import and export for EODEM. The import had already been done by November, but by the time of the larger update and re-design of January, we also managed to release an EODEM export.

This blog post will continue explaining how to import and export EODEM data before closing on a general perspective on the usefulness of EODEM at this point in time and in the foreseeable future.

How to Use EODEM in musdb?

As with loans in general, there are two situations in which one might want to use EODEM. The receiving museum can save a lot of time by importing EODEM data, while the sending museum can help out colleagues by exporting it.

Importing EODEM data

In the (German language) handbook, one can read a lengthy explanation of how to import object data into musdb (a less extensive description can be found in the blog as well, this time in English). In short: One first needs to generate a password to access the import folder using a WebDAV client. After mounting the folder, one can proceed to upload the data. Finally, a configuration file needs to be written to enable the import tool to identify the format of the import data, a mail address which is to be notified when the import is done etc. As the server checks all museums’ import directories for available import data every four hours, the object information will then be automatically imported after some time.

EODEM was developed as an application profile for LIDO 1.1, meaning that it builds upon and extends the LIDO standard for exchanging object information. As such, we could simply integrate EODEM into our regular LIDO import. Hence, users who want to import EODEM data need to set the “parser” setting to “Lido” in the configuration file.

Depending on the way a museum forms its inventory numbers, EODEM however may present a problem regular LIDO imports do not. If an inventory number in the sending museum (say, in the import data) equals an inventory number of one of the receiving museum’s own objects, the importer would interpret the duplicate inventory number so as to update the existing record of the inventory number rather than adding a new one. In response to this problem, we added a new setting available in the LIDO parser: prefix_inventory_numbers . By entering a value (e.g. the reference number of the loan) to this setting, the given value will be prefixed to the imported inventory number so as to prevent collisions and allow importing objects safely.

Exporting EODEM data

EODEM data can be exported from musdb the same way any other XML export can be run. It is now listed as an additional export format in the regular export form. The export form is accessible either through the navigation for exporting the whole collection – which obviously makes little sense in the case of EODEM – or on the search results page once at least one search parameter is set. One can thus filter the collection for all objects of a given loan, click the export button in the sidebar, possibly adjust the list of exported fields (e.g. to exclude a field one does not actually want to export, but which is usually covered by the EODEM export). Finally, one will receive a ZIP file containing the EODEM-encoded object information. This ZIP can then be sent to the receiving museum.

To both promote and simplify the exchange of loan object information using EODEM, a shortcut has been added in the sidebar of loan editing pages. Clicking on “Objects (EODEM)” in the export section of the sidebar directly takes one to the EODEM export settings page for all objects of the given loan.

In musdb and Beyond

As stated above, we were very late to the party. EODEM has been in development for some years, but we only started seriously looking into it in late 2022 (all credit for developing EODEM thus goes to others; the working group deserves a lot of thanks!).

On the other hand, it turns out that musdb is the first collection management system to actually feature an EODEM import and export in production. For the time being, exchanging EODEM information thus mainly helps in the case of loans between two museums that both use musdb for collection management.

The working group’s page however provides a list of software vendors who plan to implement EODEM imports and / or exports. Many of the big names are already listed there, meaning that a timely implementation of the standard in many of the major collection management systems is almost given.

On the other hand, there are many, many more collection management systems than those ten listed at the time. While EODEM is based on LIDO and thus very simple to implement if a collection management system already features a LIDO import / export, it remains to be seen how many will eventually adopt the it. Hopefully it is many, so that we can improve interoperability and soon enough all save some time when working with loan objects.

Image credit: “Relief zweier sich die Hände reichenden Männer, Seebronn (?)“, CC BY-SA @ Landesmuseum Württemberg

This content is licensed under a Creative Commons Attribution 4.0 International license.

musdb

New fields

Other page

Batch updating object information

Instituion-wide settings and adding new objects

Notifications

In other news

• The calendar feature (accessible under the puzzle symbol in the main navigation) can now display tasks or make them subscribable via WebCal (thus implementing a “reminder” as had often been requested)
• PDF of all linked information is now in A4 and uses a two-column layout
• Ukrainian translation
• “Simple A5 PDF” now covers inventorization fields on rear side

md:term

• If two actors have been joined and one has an old links to the page of the actor now deleted, one is now referred onwards to the new, single actor entry. The same works with transferrals between vocabularies (an actor that was transformed into a tag, etc.).
• Ukrainian Translation

Frontend

• Ukrainian Translation
• JSON-based settings for specific institution pages have been removed
• Bulk download of object images
  • An overlay with a message from the museum can be displayed during batch downloads (see above)

CSVXML

• Almost completely rewritten
• The served page now is completely static and all checks and conversions run directly in the browser. This way, no uploads actually happen and the application is completely uncritical to the server’s security. On the other hand, this allows for installing CSVXML as a progressive web app and using it offline.
• We also added some explanatory texts did small updates to the design of the page. A footer now links to the source code and offers to refresh all cached contents of the page (this may be useful when visiting the page after a long time, as the whole application is cached in the browser for offline use).
• A new check also checks for the file encoding. A warning is provided if the data does not appear to be UTF-8-encoded.
• CSVXML is now released open source under the AGPL.

Importer

While the individual parsers for the different export formats are updated very often, the core scripts of the importer are very stable. November 2022 however came with a large update to these core sections, as more categories of data that had not before been covered by the importer (many of them new) can now be imported:

• Contact information (e.g. for object owners; loan partner institution)
• Object’s movement log
• Minimum and maximum temperature, humidity, and lux of an object
• Loans
• Events / Appointments

In terms of the parsers, we extended the LIDO parser to cover the new fields suggested by the upcoming EODEM standard for exchanging loan object information.

The most important precondition for this was, once again, the option to upload generic data to the server. This can now be done using a newly added WebDAV interface. WebDAV is a protocoll that can be used in a roughly analogous way to FTP, but it is built on top of HTTP (the main protocol used for accessing websites). Support for the protocol is hence rather good – even Windows Explorer can mount WebDAV network drives – and it should not be blocked by the firewalls of larger museums.

How to import data using the new WebDAV interface?

First, one needs to generate login data to the WebDAV drive for the respective institution (this is always the institution one’s account was created for, even if one has access to multiple museums’ data. To do so, one needs to login to musdb and then navigate to one’s account settings. Here, there is a new tab available below the table for basic login information: WebDAV. Once that tab is opened, a short explanation becomes visible alongside a button “Generate WebDAV password”. Upon clicking that button, a password is automatically generated the login information is displayed in a dialogue overlay. This login information should then be stored in a safe place (at best, a password manager), as it is not accessible from the server anymore at a later point in time.

Using this data, one can now proceed to connect to the WebDAV drive using one’s WebDAV client of choice. For first attempts to try out the import process, one can simply use a file manager (as stated above, even Windows Explorer can handle WebDAV), but if one repeatedly wants to work with the import tool, it is advisable to get a dedicated WebDAV client like WinSCP. Note that, depending on the client, it may be necessary to change the protocol from davs:// to https://.

Uploading data for imports and configuring imports

When one accesses the WebDAV drive for the first, two folders and two files will be listed:

.
├── import_config.sample.txt
├── IMPORT_IMG
├── IMPORT_XML
└── README.md

The meaning of README.md should be evident: here you can also read about the expected folder structure and the next steps required for running an import.

Uploading import data

The folders IMPORT_IMG and IMPORT_XML are where import data should be uploaded to. Images as well as other media files go to IMPORT_IMG; metadata files – irrespective of their format – can be uploaded to IMPORT_XML. It is important to note, that there should be no further subdirectories created within the directories – the import scripts expect a flat folder structure.

Configuring the import

Finally, the server needs to be told what kind of import data is present here and that there is an import to be done at all. The file import_config.sample.txt can be used for this purpose. Upon opening it in a text editor (e.g. Notepad; not MS Word), one can see all the possible import configuration settings in lines that are to be completed.

Required for running the import are only a mail address and the name of the parser (from the user’s perspective: the expected import format) to use. The mail address is required to enable the server to give feedback about the success of an import or any error appearing when importing (the occurence of any error automatically halts the import process). Note that the mail address needs to be one of a user in musdb who holds the user role “museum director” or “administrator”. A detailed list of the available parsers, complete with descriptions, further links and possible parser-specific settings can be found at the end of the file.

All other settings besides the mail address are optional. One may e.g. set objects to be automatically published upon the completion of the import or one can link all the imported objects to a collection, overwriting possible collection links in the import data.

Once the import configuration has been completely filled out, the file should be saved and renamed to import_config.txt. The existence of a file of this name signals to the server, that an import should take place here.

The import runs

The server has a timed script, that checks all WebDAV drives once every four hours to see, if any import data exists there (signaled by the existence of the file import_config.txt). If that is the case, the import will be run.

If the import completes without any problems, the import data and configuration file will be moved to a new directory IMPORTS_SUCCESS and a notification will be sent to the mail address named in the configration file. Eventually, the import data will be removed from the WebDAV drive.

Any problem on the other hand immediately leads to the import being halted and the import data being moved to a folder IMPORTS_FAILED. Similar to successful imports, a notification will be sent out to the mail address from the configuration file and to the development team. The selected parser or the import data then need to be adjusted and the import data moved back to the main IMPORT_XML and IMPORT_IMG folders before one can try importing again. Potential problems can range from incomplete data to the existence of fields not yet covered by the respective parser in the import script.

Necessary adjustments to the import tool

To enable users to safely import data on their own, the import tool had to be substantially remodelled. In general, the import tool consists of four types of scripts:

• Representation of the basic data types available for imports (e.g. objects, events, images, collections)
• Writers, that handle writing the imported data to the database in bulk
• Parsers, that handle the reading of import data of a given format and translate values from the import data format to the expected data types in museum-digital
• Frontend scripts, that provide e.g. the command line interface

Thus far, only the data types and writers had been consistently written using an object-oriented approach. Enabling users to set parser-specific settings required us to refactor the remaining code to similarly use object-oriented programming. As a positive side effect, this allows us to test the parsers much more easily, so that errors in rarely-used parsers arising from updates to other components or the programming language itself will be noticed and solved much more quickly than it would have been the case before.

On the other hand, this refactoring and the new requirement to at least add integration tests for all parsers led us to drop support for some rarely used parsers, for which we could not find example data anymore.

Who may profit from the new import option

While the present update allows users to upload import data and run imports themselves, it only allows limited options to adjusting the parsers themselves. It is thus museums who often import data of the same format who can profit most from the new format.

Museums who have import data in a format that is not yet covered by the available parsers will still need to go with the old way of importing and simply write a mail.

P.S.: An up-to-date description of the import process can be found in the German-language handbook.

This content is licensed under a Creative Commons Attribution 4.0 International license.