musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

• On source / reference pages, linked objects are now sorted by the position within the source work on which they are referenced or which they do themselves reference
• The target URL of the regular / unspecified search bar for objects now follows the new, prettier URL schema
• Support for an OAI-PMH API for object metadata
  • Standardized endpoint for aggregators seeking to retrieve data in batch
  • Metadata formats thus far supported:
    • LIDO
    • OAI-DC (mandatory)
  • See also: Blog
• PDFs are only generated for users with a browser set to a non-default language if load on the server is low
  • The resource use caused by AI bots scraping museum-digital has been growing and growing. Generally, we see bots included in our mission to enable access to cultural heritage. On the other hand, nobody is served if the service is bogged down by bots. One functionality that is commonly used among bots and resource intensive is the generation of PDFs for object pages. The same information can be loaded from the object page itself and printed to a PDF using the browser’s print option. There are thus rather few downsides to limiting access to PDF generation to timmes, when server load is low. So that’s what we did.
• Collection-specific ISIL identifiers are now also used in the LIDO API
• Alternative numbers of an object can now be displayed on object pages
  • This includes tooltips for types of alternative numbers, that can be set by the museum on the institution-wide settings pages of musdb

musdb

• Search for objects
  • Type-ahead search for languages (of the object’s content)
  • Search by object’s revision status (open, read-ony, archived, etc.)
• Batch editing of objects’ revision status
• Parameters of the full text search index were updated to improve the search of word compounds in German

Importer

Core

• The dry-run mode now does not abort an import anymore, if an unmapped value is encountered. Unmapped entries are collected and displayed together afterwards.
  • This means, that unmapped entries can now much more easily be copied to and mapped in concordance.museum-digital.org
• Support for the import of alternative numbers (of objects)
• Support for the import of space hierarchies

Parser

• AdlibXml
  • Support for importing objects’ alternative numbers
• CsvXml
  • Support for importing objects’ alternative numbers
• CsvLocations
  • New parser for csv-based imports of space hierarchies
• ImageByInvno
  • New setting: append_chars (Adds suffixes, that exist in the inventory number, but not in file names)

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Objects that are linked to a source as referencing the source or being referenced in it are now listed on the source’s page
  • Example: “Novalis Schriften. Die Werke Friedrich von Hardenbergs. Historisch-kritische Ausgabe. Erster Band: Das dichterische Werk. 3. Auflage”
• The object page now displays the new transcription fields for notes, status, and type of the transcription
• New types of classifications for the link between an object and a tag
  • Taxon
  • Topic
  • Mentioned subject (similar to the existing “display subject”)
• Dependencies
  • Updated OpenLayers to Version 10.6

musdb

• Renaming vocabulary entries to blacklisted terms is now impossible
  • Someone managed to create an actor “unknown” by adding a different actor and renaming the new actor entry to “unknown” afterwards.
• Added a sidebar to object group overview pages for additional filtering options
• New types of classifications for the link between an object and a tag
  • Taxon
  • Topic
  • Mentioned subject (similar to the existing “display subject”)
• New APIs for listing all vocabulary entries linked to the objects of a museum
• Dependencies
  • Updated OpenLayers to Version 10.6

Importer

• Core
  • Deaccessions are now covered by the import tool
  • Recipients of deaccessions are linked to the address book / contact list
• Parser
  • CSVXML: The CSVXML parser now covers deaccessions as well
  • ImageByInvno: Added an option to ignore any characters before a given starting combination of characters (if there is a consistent start of the inventory number).

CSVXML

• New fields
  • tag_related_identifier_type
  • tag_related_identifier

Dissemination

• Presentation “Von museum-digital zum eigenen Online-Katalog” for the Working Group Digitization of the State Museum Association of Rhineland-Palatine
  • Slides: PDF
  • Slides: ODP

This content is licensed under a Creative Commons Attribution 4.0 International license.

Generally, the pace and type of development seems to have changed this year. Rather than doing many small improvements all over the place, there is less but larger and more labor intensive changes and new features. See for example the versioning in musdb (January), the tool for automating imports based on what others call a “hot folder” and the sort option to sort objects by their images’ aesthetics score (both presented here in March), and the new tool for suggesting formulations for object descriptions using large language models (June).

July

Frontend

• Translation of the software to Hindi and Telugu
• Grouping of tags by their relation to a given object
  If an object is linked to more than 10 tags, the tag list of object pages quickly starts looking unorganized and messy. In such cases, the tags will now be displayed grouped by their relationship to the object (thus far: general tag, material, technique, object type, display subject)

musdb

New Features

• Export option for the specific LIDO as expected by the German Digital Library’s “colonial contexts” portal

Improvements & Changes

• The minimum length of fulltext search terms for object search parameters is now visibly enforced in the extended search user interface
  To not overly burden the fulltext search server, any term in a full text search in musdb needs to be a minimum of two characters long. Thus far, shorter full text search parameters were simply ignored. This certainly was confusing at times. Since June, attempting to perform search queries with shorter search parameters is made impossible by a check in the extended search overlay.
• Reception history of objects: Statements of the relevant position within a source can now be 40 characters long
• Transcriptions
  • May now be up to 4000000 long
  • New fields: Notes on the transcription, status, aims

Bugfixes

• Fixed a bug in the batch editing of specific visibility flags for data fields on the addendum tab

Juni

Frontend

• Performance improvements
  • Object search now runs without a connection to the full text search server if no full text search parameter is set
  • If multiple search parameters for an earliest / latest time have been set, they are parsed and combined before being forwarded to the database
• Improvements in the deletion of temporarily created PDF files (PDF export)
• Navigation has been translated to Tamil

musdb

New Features

• Recipient of deaccessed objects can now be linked from within the address book
• New tool for AI-aided formulation o object descriptions (based on existing other metadata)

Improvements

• Object search now runs without a connection to the full text search server if no full text search parameter is set

Importer

• The CSVXML parser has been extended to cover new event types and markings
• Object groups automatically generated to group all objects of an import can now receive a description as set within the import configuration

nodac

The list of selectable languages for the navigation of nodac has now been restricted to those in which there is actually a complete translation

New Features

• Tabs are available in the configuration when the site is installed as a Progressive Web App (PWA) (See also)

Improvements

• Duplicate search query parameters for objects associated with times before / after are removed.
  Example: If one searches for “Objects after 1900, whose first recorded associated time is also after 2000”, this is a duplicate query. 2000 is always after 1900, so one of the two parameters can be removed.
  Searches by time begin / end are also the foundation for the timeline. As the timeline is both beloved by web crawlers and resource-intensive to generate, this change significantly reduced server load.
• Any links on timeline pages that do not reference single object pages are marked as rel=nofollow
  Which is to say that bots are told to ignore them.

musdb

New Features

• Tabs are available in the configuration when musdb is installed as a Progressive Web App (PWA) (See also)
• New search option for objects: “Can be published”
  Aggregate search for objects that have not yet been published and which comply with the minimum requirements for publication.
• Separated measurements can now be batch-edited
  Available via “assign results”
• Institution-wide setting to enforce the use of the user-defined object editing interface for all users of the institution
• Exhibitions can now be searched via the API (/exhibition/list)
• A user’s user-agent is checked whenever a page is requested. If it changed, the user will be logged out automatically.
  Protection against session hijacking.

Improvements

• Panorama images for tours through exhibitions / institutions are calculated down to 2400 px height rather than 1400 px height
• The APIs for searching for entries in controlled vocabularies are now available via the main API
  See e.g. /actor/search_linked_to_objects/{term} , /actor/search/{term} etc.

Bugfixes

• Fix: “Visiting scientists” couldn’t open the “location” tab (this erroneously required museum-wide editing permissions)
• Fix: User-defined defaults for descriptions of new objects were ignored when new objects were to be added
• Fix: Thumbnails were displayed as duplicate images linked to exhibitions (tag “images”)
• Fix: Prefixing via the batch editing was broken

Importer

New Features

• New parser for CSV exports / imports from Robotron Daphne

Improvements

• CSVXML Parser
  • New literature-related fields (ISSN, editor, etc.) are now covered
  • References to wikidata are now respected as tags are imported
• The maximum length of any single given tag/keyword is centrally set to 95 characters

Bugfixes

• Some more recently added fields from the “admininstration” tag of musdb’s object pages had been read by the importer but their transfer into the database had not been implemented thus far.

nodac

New Features

• Tabs are available in the configuration when nodac is installed as a Progressive Web App (PWA) (See also)

CSVXML

Improvements

• Added new literature fields: type, editor, edition / issue, ISSN

• A Kannada translation of the software is now available
• Museums can now select to display their own citation notes in the menu for such on object pages
  This is especially relevant in case the object itself is to be cited (rather than its record online).
• “Or” search queries can be combined within one search parameter for select attribute search types, e.g. places and tags. The Syntax is as follows: place:61~1
  • For now, this can only be used using the internal query language. As such, there is no corresponding option in the UI for search settings. On the other hand, the search option is thus available using the API.
  • This option is not available when searching for times or full events

musdb

• Institution-wide settings
  • Free text fields that double with similar controlled fields may now be hidden
    • The acquisition of an object may e.g. be directly recorded on the context of a given object or as a separate acquisition process. Recording it as an acquisition process is slightly more labor-intensive, but allows a more fine-grained and accurate documentation. With the new setting, the data fields for recording acquisitions directly in the object context can be hidden from all users of a museum to ensure a uniform use of the preferred functionality.
    • Institution-specific notes on how to cite objects may now be recorded for display on published object pages.
• Search queries for inventory numbers are now character-level searches again (rather than following a fulltext search logic)
• Some event types are incomplete, i.e. they cannot contain a place, or an actor, or a time. This incompleteness was handled differently between musdb, the import tool, and the CSVXML import preparation tool. Now, it is determined by a centralized list and thus similar across all of museum-digital.
• Refactoring of the administrative command line interface
  This mainly concerns auto-correction tools, but also results in that exports for the quick export option are now automatically generated daily.
  
• Separated measurements are now positioned at the very top of the “addendum” tab of object editing pages.
• It is now possible to record web links for object groups

Importer

• The import tool can now be used as a harvester
  • First use case is a harvester for LIDO records delivered via an OAI-PMH interface.
• Externally stored images in formats other than JPG can now be imported
• Significantly extended the LIDO parser to (among others) support the import of multilingual object data

This content is licensed under a Creative Commons Attribution 4.0 International license.

Here, as so often, in a list format:

Frontend

• Bugfix: The detailed description was not reflected in the object API even if set to be so via musdb
• Translation: The frontend is now available in Kannada

musdb

• Bugfix: Fixed error in setting up new two factor authentication via TOTP (app-based 2fa)
• Bugfix: Symbols for image rotation switched (counter-clockwise symbol rotated clockwise and vice-versa)
• Feature: If the generation of a PDF catalogue is triggered via the sidebar of series editing pages, the objects will follow their order within the object group

Importer

• If no explicit name/title for a resource (audio, video, PDF, 3D, externally hosted image) was supplied, there was no fallback. Now, the importer follows the same logic as musdb and will fall back to using the linked object’s object name as a resource title if none is supplied.
• The CSVXML parser can now handle multiple objects per XML file.

Auto uploader

The auto uploader is an entirely new tool. When first started, users will be asked to enter the necessary data for running imports. And they are asked for the path to a folder which will subsequently be monitored for automatic uploads.

Whenever the tool then detects suitable contents for an import in the folder, the data will be uploaded and automatically imported. Intermediate steps like the generation of a settings file for the import are covered by the tool.

The tool can thus be useful for museums that regularly import data formed in a consistent format – say, museums using museum-digital for publishing while mainting a separate collection management system. It should be uninteresting to those who spend much effort on preparing imports (e.g. via CSVXML) or who want to migrate to musdb altogether, as it only becomes useful in the case of regular use.

The code is available here, licensed under GPL v3.
See also the more detailed German-language blog post about the auto uploader.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Dates in TEI transcriptions are parsed, irrespective of whether when="" oder when='' was used
• Notes for markings are now publicly displayed
  This was missing thus far and is now implemented similar to how event notes are displayed. If a note exists, a small “[?]” appears behind the marking title line. Upon hovering over it, a tooltip appears with the relevant information.

musdb

• Names and descriptions of exhibitions and object groups can now be translated
• Version control
• Log of “current locations” of an object can be exported as a CSV file
• Uploaded object images can now be hidden or published in one batch operation
• API extended
  • (New functions)
  • Transfer object dimensions
  • List images and resources for an object
  • Image metadata
  • Publish / hide object images

January 2025

Frontend

• Objects can now be sorted by the aesthetics of the thumbnail (A dedicated blog post on this will follow soon)

musdb

• Batch transfer between between free text fields of object data
• Alignment of the maximum field length for notes on opening hours is now consistent between UI and database
• Bug fixed with switching between institutions during consistency checks fixed (relevant only to users with administrative access to multiple museums)
• Literature can now be searched by editors

Importer

• Core
  • Automatic transformation of life dates for actors
    • Year of death “01.01.2012” now becomes “2012”, instead of 01.01 as before
  • “?” and “(?)” are removed from the beginning and end of imported keywords
  • Various types of brackets in keyword names are converted to regular brackets
• Parser
  • Stricter internal implementation of settings, all imports can now implement the start_at setting
    • This is particularly useful for the repeated execution of imports that abort due to new, previously uncovered elements and other debugging.
  • New parsers:
    • MODS (mainly used in library contexts)
    • Parser for Exports from Faust for the Händel-Haus
    • Parser for XML dumps from MuseumPlus Classic (MsSQL > XML export per table > Import)
    • Bugfixes
      • Field “Verwender” in Primus parser was mapped to production events
      • Material / technology are now imported correctly in the parser for BeeCollect exports for the Industrial Museums of Saxony
• „Frontend“
  • CLI now also has options for switching off the import of individual areas
  • Help text for command line tool

nodac

• Splitting of keywords now also recognizes keywords that should be split into places, times, etc.
  • Bsp.: „Helm; Berlin“ > Schlagwort „Helm“ + Ort „Berlin“
  • Example: “helmet; Berlin” > keyword “helmet” + place “Berlin”
• When searching for keywords with ambiguous names, both keywords and generally ambiguous terms are now taken into account
• Times can now be merged with others directly from the time edit page

This content is licensed under a Creative Commons Attribution 4.0 International license.

Say, a museum’s previous database used actor roles over an event structure to express who created an object. As such, the museum entered that the object has a linked actor X that is linked to the object as a “main creator” and a linked time Y marked as the “creation time”. During the import, these roles (“main creator” and “creation time”) are then translated to museuem-digital’s event types to form an event: The object was created by actor X at the time Y. This works, because the terms “main creator” and “creation time” have been matched to the creation event type.

If a term is not yet matched to a corresponding value of a controlled list in museum-digital, the importer will simply abort the import. On the one hand this is a way to uselessly require resources for an import that cannot be completed anyway. On the other, it is tedious. One recognizes yet unmatched entries only one by one.

A Small New Tool

A small new tool, available at concordance.museum-digital.org, makes the process a bit less tedious. Users can upload all the import data from a given field (e.g. the actor roles) – one a line – and check whether they are already matched using the concordance lists or not.

For entries that are not yet matched, the tool will offer selection boxes to perform the matching using the graphical user interface. Once all entries have been matched, one can then generate the relevant lines of code to enter the missing entries to the concordance list upon the click of a button.

While simply checking and extending the relevant open source lists should be trivial even to most non-technical users, this way is certainly more convenient. Importantly, it also removes the need to run the import multiple times until one does not encounter errors caused by unmatched entries anymore. And, well, it certainly is also more convenient to match to regular human language values than to the internal IDs of the target values.

The concordance checker’s MIT-licensed code can be found here.

This content is licensed under a Creative Commons Attribution 4.0 International license.

• Basic re-implementation of separated measurements for objects (tab: “addendum”)
• The type of measurement (width, length, etc.) is now managed using a controlled list, which can easily be extended. This also allows for measurement types of different levels of specificity (“width” vs. “width of socle”)
• The new implementation allows users to identify whether a measurement is exact and add notes for each measurement
• Measured values now need to be entered as a floating point number – consistent search for objects smaller or larger than a given size is thus made possible (previously only available for entries where the system could deduce a numeric value from whatever had been entered)
• Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
• Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
• The list of spaces that can be linked to an object as its current location is now sorted alphabetically

Frontend

• Re-implemented separated Measurements

Import

• The import tool now also uses the new implementation of separated measurements
• Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
• Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
• Links between two objects are now imported using the dedicated data type for this purpose. They had previously been imported as regular web links.

nodac

• When merging two entries, links between the entries and collections, exhibitions, etc. are now reflected and rewritten
• previously, links to such data types prevented the completion of the merge
• If an entry’s name is marked to always belong to e.g. a tag, the buttons for transferring the entry to the actor, place, or time vocabularies are now hidden.
• If an entry is moved between vocabularies, links between the entry and objects as “display subject” / “displayed place” / “displayed actor” are reflected and translated into new links using the appropriate link type.
• A new context menu on overview pages allows for a quick access to functionalities like the splitting of entries