musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

These are in the form of listings, and this is how it should be here too.

Frontend

Features & Improvements

• Some improvements in background scripts, especially better handling of timeouts when calculating “Similar objects” in very large instances
• Contributors, linked locations and times for an object group are now listed alphabetically by name
• Table headers for event components (who, when, where) are now only displayed in the A4 PDF if there is also content for the row
• New search option for object searches: “Is currently on display”
• Links to the Themator now use the new URL scheme of the Themator
  (https://themator.museum-digital.de/t/690 instead of https://themator.museum-digital.de/ausgabe/showthema.php?m_tid=690&tid=690)

Bugfixes

• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)
• Exactness setting in the “refine search” overlay was not transferred to the actual search query
• Simple embedding of an object (analogous to YouTube videos, for example; accessible via the “Cite” menu of an object page) had various errors / now works again

musdb

Features & Improvements

• In the API documentation of musdb there is now a note that the frontend also has an API
  • Frontend API
    • You do not need to authenticate yourself to use the frontend API
    • The frontend API tends to be faster and easier to use
    • Is read-only
  • musdb API
    • Can do more: Can also see non-public stocks and fields / data types
    • Is much more granular (more queries for the same data, but you likely get exactly the data you are looking for instead of e.g. all data known about a given object)
    • Can be used for writing data
• Suggestion lists when searching for vocabulary terms in the side column of the object search page have been revised
  • Tooltips appear when hovering over
  • Implementation in Vanilla JS, removing jQuery
  • (this means significantly better performance of the search results list in list format, because jQuery no longer needs to be loaded)

• User page / Login
  • Log of logins now also with IP and user agents
  • Login via login persisted in the browser (“Remember me”) is logged and displayed
  • All browsers permanently logged in via cookie are forced to log in again after a password change
  • New option to invalidate all remembered logins on other devices (browser must be logged in again)

• Object
  • More restrictions for the publication of object data records.
  • An object can no longer be published if:
    • … the object name is the same as the object description
    • … the description contains the character string “lorem ipsum”
  • When object entries are unpublished / hidden, the images linked to the image are renamed (thus invalidating links to the images). When publishing the object again, this is reversed so that existing links work again.
  • Spaces in selection lists are now listed alphabetically as the actual location when linking
• Literature
  • Acquisitions can now be linked to literature
    • Previous owners etc. can thus be linked to a literature entry
  • Spaces (actual location) can be linked to literature

Bugfixes

• Overlay for setting searches for objects: Multi-word search terms were converted into multiple searches instead of being searched as a string of words (“red helmet” > “red” AND “helmet” instead of “red helmet”)
• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)

Importer

• Link between literature and spaces (actual location) as well as acquisitions is implemented in the “core” of the import tool
• ImageByInvno parser (assignment of images to objects via inventory numbers contained in the file name) can now be used to import PDF files

files.museum-digital.org

• Added a small script to enhance PDF metadata based on an XML sidecar file. See e.g.: https://files.museum-digital.org/de/Praesentationen/2024-10-18_md-deutschland-eV-stellt-sich-vor_Usertreffen_MA.xml

Main post image generated using illustriousXL_smoothftSPO

This content is licensed under a Creative Commons Attribution 4.0 International license.

General

YouTube channel

There now is a museum-digital YouTube channel. For now, one can find some German-language screencasts on different features in musdb and nodac there.

New sections on project page (www.museum-digital.org)

As already described in the previous post, the project page www.museum-digital.org now features a calendar. Since then, we have also added a small page listing people working on museum-digital – e.g. some of the coordinators and developers – as well as power users. The list will obviously always be more than incomplete, but if you want to be listed, just send a mail.

Development

musdb

Allow batch editing of photographers on image list

Running batch updates on the ownership, license status and photographer of a set of objects can be done either via the institution-wide settings page or via a selection on the image list page. The batch updating functionality of the institution-wide settings page works by replacing the given entries in one of the fields for all objects of the museum – it is thus not suitable if one is to only update the photographer field for a list of objects from a single collection (rather than the whole museum).

Batch updating images’ legal information from the image list (click on an image in the list and drag it to activate selection mode, then update by selection) was only possible in the case of image licenses and rights holders. With a small update, it is now possible to also batch-update the images’ photographers field that way.

The API can now update object event

Thus far, the API could only be used for adding wholly new object events (who did what, when, and where, with the object) or deleting them. With the updates of this month, now possible to directly update events.

This is already actively used in customizing musdb using a Tampermonkey script by some of the norm data editors. They can thus add a custom button to transfer incomplete actor or place information to the object’s description. This button, on the other hand, would not be useful for regular users at all.

UI improvements for the object search interface and new sort option

After a very fruitful discussion and feedback from Hungarian colleagues, we got around to some user interface improvements in the object overview and search interface. A lengthy discussion can be found in a dedicated blog post in German. To sum up: there are now headlines for each subsection of the sidebar and tooltips explain the different buttons and search and sort options upon hovering one’s mouse over them. Additionally, a new sort option was added to sort objects by the number characters within the inventory numbers of the searched objects.

Two bugfixes: Navigating to the previous or next object in a selection

It is a very simple but similarly useful feature: One runs an object search and thus creates a result list of objects. After viewing or editing one of the objects, one can then proceed to the next one in the results list by clicking at the arrows at the top of the sidebar.

Unfortunately, musdb had two bugs preventing this navigation for some of the available sort options. On the one hand, newer sort options required manually adding the search fields to a list specific to the previous/next navigation thus far – and had not been covered yet. Navigating to the next object after sorting by e.g. the length of the searched objects was thus impossible so far. The specific list of sort fields has now been removed in the previous/next navigation. Instead, the main list of sort options from the main object search class is directly used – in effect preventing such issues of “forgotten” values from ever appearing again.

A second issue concerned string-based sort options. For an improved performance, the previous / next navigation works by querying the underlying search index for the next object where e.g. the object ID is lower than the current one (`WHERE id > 20000`). The same search logic is straight-out not possible with string-based sort options, as search queries for values greater than or smaller than a given one cannot be executed with string-based fields.

Fortunately, the issue came up just after we had implemented an option to search and sort objects by the numeric components of their inventory numbers. Thus, the previous / next navigation can fall back to that sort option if a user wants to navigate to the next object after sorting by inventory number. As stated back when we introduced the new search and sort option: It should work for most museums. Unfortunately, there is no such option to mitigate the issues of string-based sorting when sorting objects by their names. The best we could do in this case is displaying a clear error message, stating that the previous / next navigation is not available when sorting by the object’s names.

“Home location” can now be selected as a required field when adding new objects

On the institution-wide settings page, museum directors (or those holding the same user role within musdb) can determine which fields are required when adding new objects. The list of selectable fields automatically covers all free text fields linked to the object. Repeat fields and references to other sections of musdb need to be manually implemented however. As the home or permanent location of an object is a most logical and common field to be required for all object, it was only consequential to prioritize making it available as a required field. We have done so now.

Major performance improvements in “list results” page

The “list results” page provides for a table view of object search results, allows the export of search results to Excel, and is – under the hood – also used for generating custom reports. We managed to achieve a considerable improvement in the page’s performance by bulk loading data and simplifying the code of the HTML page. Both adjustments should not really be noticable when one tries to list or export only some objects. Exporting some thousand objects should work considerably faster now however.

frontend

PDF metadata for main object PDF export

XMP metadata are now written into exported PDF files. Thus, users can more easily identify authorship, titles etc., especially when using specialized software like e-book readers.

Linking from collection descriptions

Collection descriptions are now being parsed for the existence of URLs. If one, starting with https:// (http:// is ignored) exists, the URL is automatically transformed into a clickable link.

nodac

We added some translation variables on the start page.

themator

The “themator” has received a small user interface improvement for mobile devices. The topic-specific navigation or table of contents is now moved below rather than above the main content of a page in the regular “topic” viewing mode on mobile devices.

Importer

Importing GIFs

To simplify the handling of images, museum-digital internally only uses JPG image files (webp versions are generated for a faster loading, but are optional). While they lack some support for features like transparency, JPGs provide for a much better compression with images of three-dimensional things and subjects not featuring solid colors. Newer formats like AVIF and Webp unfortunately still lack the support among locally installed image viewers for us to fully switch over to them.

The importer handles input PNGs by converting them to JPG files. Starting this month, the same can be done with input GIF files.

Improved handling of blacklisted actors, places and times

One of the key issues with imports is what happens after imports. Specifically, museum-digital uses controlled vocabularies for actors, places, times and tags that can be linked to an object directly or via events. When people enter new such entries using musdb, there are a number of safeguards and nudges to ensure productive inputs. Adding a new actor, for example, requires one to enter at least 10 characters of a description for the actor. Input fields for the date of birth and death are also provided. If possible, one can enter a Wikipedia or Wikidata URL (or ID) and directly fetch the relevant information from there.

During imports, there is no space for asking questions to the importing institution. Incomplete or false actor, place, time and tag names (for example, if two actors are linked as one; “John Doe and Jane Doe” is not one single actor!) are given, these are necessarily imported into our controlled vocabularies by default. Depending on the import, this puts a significant burden on our volunteer group of vocabulary editors.

To eventually get to a solution, we have added options for automatically rewriting terms (e.g. “Berlin, Germany” will be autocorrected to “Berlin”) and blacklisting them completely (“Good morning …” is no actor”). Thus far, blacklisted terms were simply ignored by the importer.

“Unknown painter” is not an actor, but the information that it is a painter may still be useful. To allow us to blacklist more and keep the controlled vocabularies “clean” without losing such data, the importer now imports blacklisted terms in event components (“who painted it?”) into the event annotation. If the only known information on the whole event is blacklisted, the event cannot be created – an event always requires at least one linked actor, place or time. The event annotation of “empty events” is hence moved to the object description automatically.

This content is licensed under a Creative Commons Attribution 4.0 International license.

To simplify the management of objects with many images musdb offers the option to upload multiple images at a time. This option is however limited by global server settings like the maximum size of uploads and the maximum number of files that can be uploaded in one go.

To understand the problem, it’s important to know that most tools of museum-digital are written in PHP. Unfortunately, PHP has a tendency to gloss over potential errors and continue without enforcing that the problem to be solved either by the programmer solving a bug or by the user not doing a certain action.

The maximum number of files that can be uploaded simultaneously is a flagrant case of this: Rather than inform the user of their overstepping of the set upload limitations, PHP accepts files until the number of allowed files per upload is reached. Any files after are simply ignored.

Say a user attempts to upload 30 pages of a book in one go. The first 20 will be uploaded correctly, while the last 10 will be skipped. In the worst case, the user doesn’t even notice the missing data. In a better case, they are left confused.

We have thus added a warning that stops the user before submitting the upload form when trying to upload more files than allowed.

This content is licensed under a Creative Commons Attribution 4.0 International license.