musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Generally, the pace and type of development seems to have changed this year. Rather than doing many small improvements all over the place, there is less but larger and more labor intensive changes and new features. See for example the versioning in musdb (January), the tool for automating imports based on what others call a “hot folder” and the sort option to sort objects by their images’ aesthetics score (both presented here in March), and the new tool for suggesting formulations for object descriptions using large language models (June).

July

Frontend

• Translation of the software to Hindi and Telugu
• Grouping of tags by their relation to a given object
  If an object is linked to more than 10 tags, the tag list of object pages quickly starts looking unorganized and messy. In such cases, the tags will now be displayed grouped by their relationship to the object (thus far: general tag, material, technique, object type, display subject)

musdb

New Features

• Export option for the specific LIDO as expected by the German Digital Library’s “colonial contexts” portal

Improvements & Changes

• The minimum length of fulltext search terms for object search parameters is now visibly enforced in the extended search user interface
  To not overly burden the fulltext search server, any term in a full text search in musdb needs to be a minimum of two characters long. Thus far, shorter full text search parameters were simply ignored. This certainly was confusing at times. Since June, attempting to perform search queries with shorter search parameters is made impossible by a check in the extended search overlay.
• Reception history of objects: Statements of the relevant position within a source can now be 40 characters long
• Transcriptions
  • May now be up to 4000000 long
  • New fields: Notes on the transcription, status, aims

Bugfixes

• Fixed a bug in the batch editing of specific visibility flags for data fields on the addendum tab

Juni

Frontend

• Performance improvements
  • Object search now runs without a connection to the full text search server if no full text search parameter is set
  • If multiple search parameters for an earliest / latest time have been set, they are parsed and combined before being forwarded to the database
• Improvements in the deletion of temporarily created PDF files (PDF export)
• Navigation has been translated to Tamil

musdb

New Features

• Recipient of deaccessed objects can now be linked from within the address book
• New tool for AI-aided formulation o object descriptions (based on existing other metadata)

Improvements

• Object search now runs without a connection to the full text search server if no full text search parameter is set

Importer

• The CSVXML parser has been extended to cover new event types and markings
• Object groups automatically generated to group all objects of an import can now receive a description as set within the import configuration

nodac

The list of selectable languages for the navigation of nodac has now been restricted to those in which there is actually a complete translation

New Features

• Tabs are available in the configuration when the site is installed as a Progressive Web App (PWA) (See also)

Improvements

• Duplicate search query parameters for objects associated with times before / after are removed.
  Example: If one searches for “Objects after 1900, whose first recorded associated time is also after 2000”, this is a duplicate query. 2000 is always after 1900, so one of the two parameters can be removed.
  Searches by time begin / end are also the foundation for the timeline. As the timeline is both beloved by web crawlers and resource-intensive to generate, this change significantly reduced server load.
• Any links on timeline pages that do not reference single object pages are marked as rel=nofollow
  Which is to say that bots are told to ignore them.

musdb

New Features

• Tabs are available in the configuration when musdb is installed as a Progressive Web App (PWA) (See also)
• New search option for objects: “Can be published”
  Aggregate search for objects that have not yet been published and which comply with the minimum requirements for publication.
• Separated measurements can now be batch-edited
  Available via “assign results”
• Institution-wide setting to enforce the use of the user-defined object editing interface for all users of the institution
• Exhibitions can now be searched via the API (/exhibition/list)
• A user’s user-agent is checked whenever a page is requested. If it changed, the user will be logged out automatically.
  Protection against session hijacking.

Improvements

• Panorama images for tours through exhibitions / institutions are calculated down to 2400 px height rather than 1400 px height
• The APIs for searching for entries in controlled vocabularies are now available via the main API
  See e.g. /actor/search_linked_to_objects/{term} , /actor/search/{term} etc.

Bugfixes

• Fix: “Visiting scientists” couldn’t open the “location” tab (this erroneously required museum-wide editing permissions)
• Fix: User-defined defaults for descriptions of new objects were ignored when new objects were to be added
• Fix: Thumbnails were displayed as duplicate images linked to exhibitions (tag “images”)
• Fix: Prefixing via the batch editing was broken

Importer

New Features

• New parser for CSV exports / imports from Robotron Daphne

Improvements

• CSVXML Parser
  • New literature-related fields (ISSN, editor, etc.) are now covered
  • References to wikidata are now respected as tags are imported
• The maximum length of any single given tag/keyword is centrally set to 95 characters

Bugfixes

• Some more recently added fields from the “admininstration” tag of musdb’s object pages had been read by the importer but their transfer into the database had not been implemented thus far.

nodac

New Features

• Tabs are available in the configuration when nodac is installed as a Progressive Web App (PWA) (See also)

CSVXML

Improvements

• Added new literature fields: type, editor, edition / issue, ISSN

When musdb and museum-digital’s frontend received their last large-scale update to their respective object search functions around 2021, enabling actually complex search requests across almost all of the fields and data types linkable to objects, this was – among others – made possible by our use of Manticore, a dedicated search server. Traditional relational databases excel at searches by indexes – pre-defined, known search parameters, that one prepares for searchability beforehand – while search servers like Manticore are reasonably good at that and excel at full-text searches.

In moving the search to Manticore, all searches in free text fields were defined as full-text searches. This was mostly the right decision: Full-text searches are the way to go with objects’ titles, descriptions, and the like. But two specific types of fields posed a challenge, because – as indicated above – they usually run on formalized strings that operate very differently from prose: objects’ locations and their inventory numbers. The software does not know about institution-specific and non-standardized rules of formalization, but users do. Hence, the preferred way for those specific types of fields is character-level searching.

For managing locations, we have in the meantime introduced spaces as a dedicated category capable of hierarchization as well as advanced features like the storage of sensor data. Object’s locations can now simply be expressed as a link to a space, which is by far the superior way when compared to the legacy free text field. If one does so, one can search for objects exactly in a given space, those that are located within it or its sub-spaces (e.g. a box in a depot room), etc. A migration tool from the legacy free-text field to the controlled spaces module is available through musdb’s dashboard. “Fixing” the issue of character-driven searches vs. full-text searches in locations is thus a least-priority issue – a better alternative is available anyway.

With inventory numbers on the other hand, there is no alternative to character-driven searching.

Laying the Foundations: From MSQL to Manticore and (Somewhat) Back

The basis for the expansion of search capabilities for objects was the introduction of a dedicated search server running Manticore. As the number of requests increased, this proved to be a blessing and – to some extent – a curse. Manticore offers more and better search options than a classic relational database, but it does not achieve the same level of stability. As long as queries remain index-bound and not concerned with text, the performance is roughly similar on our hardware: (both are about as quick, even with subqueries in MySQL; MySQL uses more resources, but is much more stable). If a query concerns a free-text field on the other hand, there is almost no comparison. Manticore offers a multitude of additional features at a great performance.

As stability had become an issue for a while, we adjusted the search to be able to use Manticore or MySQL as a backend, depending on which was more suitable in a specific context. In practice, this means that each search parameter is translated into a query string for Manticore and – if possible – for MySQL. If all search parameters have a MySQL equivalent, the search will be performed using the MySQL backend. Otherwise, Manticore will be used.

This simple way of negotiating which backend is more suitable works only as long as one of the alternatives (Manticore) supports all search options, while the other (MySQL) is preferrable in a subset of the search contexts. Which is to say, character-driven searches in inventory numbers break the negotiation logic – they work somewhat well in MySQL, but do not work in Manticore.

Breaking the Logic / Mitigating Confusion

Up to this weekend, all search options were compatible with each other:

• If one searches for all objects one has acc ess to, both Manticore and MySQL can handle the query. MySQL will be used.
• If one searches for all “helmets” (tag) from “Europe” (place), both Manticore and MySQL can handle the query. MySQL will be used.
• If one searches for “helmets” (full-text) from “Europe” (place), MySQL can only sufficiently handle the search by place, while Manticore can meet both search requirements. Manticore will be used.

Character-driven searches by inventory numbers break that compatibility. If one were to search for objects for “helmets” (full-text) with inventory numbers starting with 1 (“1*”), the search parameter “helmets” could only be satisfied by Manticore, while the character-driven search by inventory numbers can only be satisfied by MySQL. Which is to say, the combined search cannot be executed.

Due to popular demand, we introduced character-driven searches for inventory numbers back into musdb. As there is no way to sensibly combine all search parameters anymore, given our circumstance, we had to handle reduce the resulting confusion. For this, there are theoretically two ways.

The theoretically cleaner way would have been to disable the extension of search queries by full-text-focused parameters once an inventory number had been searched. As a full-text search by inventory number is theorecally still possible, the opposite direction (setting a full-text search first, then searching by inventory number) might still have been acceptable, as it would not have led to visibly different results. The basic idea of this solution would have been to prevent users from performing combined searches that are not possible in the targetted way. But if users actually managed in some way, the confusion would have been major. Worse yet, it would have been hard to explain – or rather, it would have been hard to find an appropiate spot in the UI for an explanation -, why certain search options are suddenly disabled.

The alternative route we chose is to allow users to do the impossible combinations, perform the search as best as we can (by transforming the character-driven search by inventory number into a full-text search), and aggressively warn about the likelihood of unexpected results when trying to perform such combined searches. This solution looks unpolished, but it is transparent about the imperfections of the software, and it allows users to find their own solutions to actually perform the combined searches they want. The simplest such solution would be to first search by inventory number, move all the objects into a watch list, and then search by the watch list and combine that search with the full-text search.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Version control

Until a few days ago, a central and sorely missed feature in musdb was a detailed version history of the data records. For example to be able to trace and restore data after an attempt at batch processing gone wrong or careless errors when deleting field contents.

Such a view of all previous versions of an object record since the start of recording (May 2024) can now be accessed via the “record history” tab when viewing and editing an object in musdb. A new “Open versioning” button appears right at the top.

Clicking on it opens an overlay in which the various versions are listed as a table. The various aspects of the object data set are divided into different tabs and therefore different tables, e.g. for basic information, administrative information, links to collections, keywords, etc.

Batch transfer

A second frequently requested feature – especially after importing – has been the option to transfer the content of one data field of object records to another. If, for example, the information previously stored in the non-publishable data field “object history” is to be stored in the “detailed description” in the future and published there, the transfer from one field to the other can now be carried out for hundreds of objects with the pressing of a few buttons. Like all other “Global Change” options, the batch transfer between different data fields always refers to the results of an object search. The function is available via the sidebar of the object overview once a search for objects has been executed.

The batch transfer between fields based search results can also be used via musdb’s API. For this, a new API route /object/transfer_by_search/{mode} has been added.

These are in the form of listings, and this is how it should be here too.

Frontend

Features & Improvements

• Some improvements in background scripts, especially better handling of timeouts when calculating “Similar objects” in very large instances
• Contributors, linked locations and times for an object group are now listed alphabetically by name
• Table headers for event components (who, when, where) are now only displayed in the A4 PDF if there is also content for the row
• New search option for object searches: “Is currently on display”
• Links to the Themator now use the new URL scheme of the Themator
  (https://themator.museum-digital.de/t/690 instead of https://themator.museum-digital.de/ausgabe/showthema.php?m_tid=690&tid=690)

Bugfixes

• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)
• Exactness setting in the “refine search” overlay was not transferred to the actual search query
• Simple embedding of an object (analogous to YouTube videos, for example; accessible via the “Cite” menu of an object page) had various errors / now works again

musdb

Features & Improvements

• In the API documentation of musdb there is now a note that the frontend also has an API
  • Frontend API
    • You do not need to authenticate yourself to use the frontend API
    • The frontend API tends to be faster and easier to use
    • Is read-only
  • musdb API
    • Can do more: Can also see non-public stocks and fields / data types
    • Is much more granular (more queries for the same data, but you likely get exactly the data you are looking for instead of e.g. all data known about a given object)
    • Can be used for writing data
• Suggestion lists when searching for vocabulary terms in the side column of the object search page have been revised
  • Tooltips appear when hovering over
  • Implementation in Vanilla JS, removing jQuery
  • (this means significantly better performance of the search results list in list format, because jQuery no longer needs to be loaded)

• User page / Login
  • Log of logins now also with IP and user agents
  • Login via login persisted in the browser (“Remember me”) is logged and displayed
  • All browsers permanently logged in via cookie are forced to log in again after a password change
  • New option to invalidate all remembered logins on other devices (browser must be logged in again)

• Object
  • More restrictions for the publication of object data records.
  • An object can no longer be published if:
    • … the object name is the same as the object description
    • … the description contains the character string “lorem ipsum”
  • When object entries are unpublished / hidden, the images linked to the image are renamed (thus invalidating links to the images). When publishing the object again, this is reversed so that existing links work again.
  • Spaces in selection lists are now listed alphabetically as the actual location when linking
• Literature
  • Acquisitions can now be linked to literature
    • Previous owners etc. can thus be linked to a literature entry
  • Spaces (actual location) can be linked to literature

Bugfixes

• Overlay for setting searches for objects: Multi-word search terms were converted into multiple searches instead of being searched as a string of words (“red helmet” > “red” AND “helmet” instead of “red helmet”)
• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)

Importer

• Link between literature and spaces (actual location) as well as acquisitions is implemented in the “core” of the import tool
• ImageByInvno parser (assignment of images to objects via inventory numbers contained in the file name) can now be used to import PDF files

files.museum-digital.org

• Added a small script to enhance PDF metadata based on an XML sidecar file. See e.g.: https://files.museum-digital.org/de/Praesentationen/2024-10-18_md-deutschland-eV-stellt-sich-vor_Usertreffen_MA.xml

Main post image generated using illustriousXL_smoothftSPO

This content is licensed under a Creative Commons Attribution 4.0 International license.

General

YouTube channel

There now is a museum-digital YouTube channel. For now, one can find some German-language screencasts on different features in musdb and nodac there.

New sections on project page (www.museum-digital.org)

As already described in the previous post, the project page www.museum-digital.org now features a calendar. Since then, we have also added a small page listing people working on museum-digital – e.g. some of the coordinators and developers – as well as power users. The list will obviously always be more than incomplete, but if you want to be listed, just send a mail.

Development

musdb

Allow batch editing of photographers on image list

Running batch updates on the ownership, license status and photographer of a set of objects can be done either via the institution-wide settings page or via a selection on the image list page. The batch updating functionality of the institution-wide settings page works by replacing the given entries in one of the fields for all objects of the museum – it is thus not suitable if one is to only update the photographer field for a list of objects from a single collection (rather than the whole museum).

Batch updating images’ legal information from the image list (click on an image in the list and drag it to activate selection mode, then update by selection) was only possible in the case of image licenses and rights holders. With a small update, it is now possible to also batch-update the images’ photographers field that way.

The API can now update object event

Thus far, the API could only be used for adding wholly new object events (who did what, when, and where, with the object) or deleting them. With the updates of this month, now possible to directly update events.

This is already actively used in customizing musdb using a Tampermonkey script by some of the norm data editors. They can thus add a custom button to transfer incomplete actor or place information to the object’s description. This button, on the other hand, would not be useful for regular users at all.

UI improvements for the object search interface and new sort option

After a very fruitful discussion and feedback from Hungarian colleagues, we got around to some user interface improvements in the object overview and search interface. A lengthy discussion can be found in a dedicated blog post in German. To sum up: there are now headlines for each subsection of the sidebar and tooltips explain the different buttons and search and sort options upon hovering one’s mouse over them. Additionally, a new sort option was added to sort objects by the number characters within the inventory numbers of the searched objects.

Two bugfixes: Navigating to the previous or next object in a selection

It is a very simple but similarly useful feature: One runs an object search and thus creates a result list of objects. After viewing or editing one of the objects, one can then proceed to the next one in the results list by clicking at the arrows at the top of the sidebar.

Unfortunately, musdb had two bugs preventing this navigation for some of the available sort options. On the one hand, newer sort options required manually adding the search fields to a list specific to the previous/next navigation thus far – and had not been covered yet. Navigating to the next object after sorting by e.g. the length of the searched objects was thus impossible so far. The specific list of sort fields has now been removed in the previous/next navigation. Instead, the main list of sort options from the main object search class is directly used – in effect preventing such issues of “forgotten” values from ever appearing again.

A second issue concerned string-based sort options. For an improved performance, the previous / next navigation works by querying the underlying search index for the next object where e.g. the object ID is lower than the current one (`WHERE id > 20000`). The same search logic is straight-out not possible with string-based sort options, as search queries for values greater than or smaller than a given one cannot be executed with string-based fields.

Fortunately, the issue came up just after we had implemented an option to search and sort objects by the numeric components of their inventory numbers. Thus, the previous / next navigation can fall back to that sort option if a user wants to navigate to the next object after sorting by inventory number. As stated back when we introduced the new search and sort option: It should work for most museums. Unfortunately, there is no such option to mitigate the issues of string-based sorting when sorting objects by their names. The best we could do in this case is displaying a clear error message, stating that the previous / next navigation is not available when sorting by the object’s names.

“Home location” can now be selected as a required field when adding new objects

On the institution-wide settings page, museum directors (or those holding the same user role within musdb) can determine which fields are required when adding new objects. The list of selectable fields automatically covers all free text fields linked to the object. Repeat fields and references to other sections of musdb need to be manually implemented however. As the home or permanent location of an object is a most logical and common field to be required for all object, it was only consequential to prioritize making it available as a required field. We have done so now.

Major performance improvements in “list results” page

The “list results” page provides for a table view of object search results, allows the export of search results to Excel, and is – under the hood – also used for generating custom reports. We managed to achieve a considerable improvement in the page’s performance by bulk loading data and simplifying the code of the HTML page. Both adjustments should not really be noticable when one tries to list or export only some objects. Exporting some thousand objects should work considerably faster now however.

frontend

PDF metadata for main object PDF export

XMP metadata are now written into exported PDF files. Thus, users can more easily identify authorship, titles etc., especially when using specialized software like e-book readers.

Linking from collection descriptions

Collection descriptions are now being parsed for the existence of URLs. If one, starting with https:// (http:// is ignored) exists, the URL is automatically transformed into a clickable link.

nodac

We added some translation variables on the start page.

themator

The “themator” has received a small user interface improvement for mobile devices. The topic-specific navigation or table of contents is now moved below rather than above the main content of a page in the regular “topic” viewing mode on mobile devices.

Importer

Importing GIFs

To simplify the handling of images, museum-digital internally only uses JPG image files (webp versions are generated for a faster loading, but are optional). While they lack some support for features like transparency, JPGs provide for a much better compression with images of three-dimensional things and subjects not featuring solid colors. Newer formats like AVIF and Webp unfortunately still lack the support among locally installed image viewers for us to fully switch over to them.

The importer handles input PNGs by converting them to JPG files. Starting this month, the same can be done with input GIF files.

Improved handling of blacklisted actors, places and times

One of the key issues with imports is what happens after imports. Specifically, museum-digital uses controlled vocabularies for actors, places, times and tags that can be linked to an object directly or via events. When people enter new such entries using musdb, there are a number of safeguards and nudges to ensure productive inputs. Adding a new actor, for example, requires one to enter at least 10 characters of a description for the actor. Input fields for the date of birth and death are also provided. If possible, one can enter a Wikipedia or Wikidata URL (or ID) and directly fetch the relevant information from there.

During imports, there is no space for asking questions to the importing institution. Incomplete or false actor, place, time and tag names (for example, if two actors are linked as one; “John Doe and Jane Doe” is not one single actor!) are given, these are necessarily imported into our controlled vocabularies by default. Depending on the import, this puts a significant burden on our volunteer group of vocabulary editors.

To eventually get to a solution, we have added options for automatically rewriting terms (e.g. “Berlin, Germany” will be autocorrected to “Berlin”) and blacklisting them completely (“Good morning …” is no actor”). Thus far, blacklisted terms were simply ignored by the importer.

“Unknown painter” is not an actor, but the information that it is a painter may still be useful. To allow us to blacklist more and keep the controlled vocabularies “clean” without losing such data, the importer now imports blacklisted terms in event components (“who painted it?”) into the event annotation. If the only known information on the whole event is blacklisted, the event cannot be created – an event always requires at least one linked actor, place or time. The event annotation of “empty events” is hence moved to the object description automatically.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Now, what about musdb? For deeper analysis of aggregate collection data – take for example the issue of average lengths above – the best way thus far were exports. The simplest way in this regard was surely the “list results” functionality in the sidebar of object searches, which provides an Excel export of selected fields for all objects filtered through an object search. The data thus exported could then be further investigated and turned into charts using Excel or a similar spreadsheet program.

While this is a rather powerful way for analyzing one’s data in aggregate and does not require the ability to code, it is still cumbersome and involves too many steps to be regularly used (unless one has an immediate occasion like an upcoming report). More immediate ways for analyzing search results for objects (or one’s whole collection) were otherwise limited in scope, like the map view for search results, which displays all events linked to the selected objects.

Starting with a minor update this weekend, musdb now supports a more straight-forward but generally applicable way for evaluating search results or whole collections.

The entry point to this new tool comes in the form of a small button – identified by the light bulb icon – in the search options bar on object search pages.

A click on the icon opens an overlay featuring four tiles. Each tile can be set to provide a chart or a table of the values found in a given field for all objects selected by the search query.

Which field’s contents are evaluated, and the way the data is grouped (grouped by year and year and month in the case of date fields; or totals and averages in the case of numeric fields) can be adjusted by clicking on the two select fields making up each tile’s headline. Changes to this selection are stored in the browser to allow for an easy comparison for the results of different searches.

Below the headline of each tile, one can find a navigation offering two options for displaying the information and a download button:

• Chart: This is the default view. It provides a visualization of the data using line charts for numeric fields or date fields grouped by year and pie charts for text fields. To not clutter the interface and improve performance, pie charts are limited to displaying the 20 most used values.
• Table: This option provides a list of the values encountered in the selected field, sorted by their popularity. Each value is clickable, allowing one to further narrow down the search results (thus, we almost accidentally implemented a faceted search feature). Like the pie charts, the table view is limited to displaying the 50 most used values for performance and a simpler user interface.
• Download: Using the download button, one can download the same data displayed in the table view – but without any limit on the number of results – in a CSV file. This file can then be opened using a spreadsheet application to do a deeper analysis or build other types of charts.

As always, we are very curious about user feedback and suggestions for further evaluation functionalities that can be integrated into the overlay.

Image credits: “Exacta-Continental 6000“. CC BY-NC-SA @ Heinz Nixdorf MuseumsForum & Jan Braun

This content is licensed under a Creative Commons Attribution 4.0 International license.

Realistically there is none. But musdb required one to enter a price for the acquisition of each single object thus far. Hence, users were left with no choice but to make up the price or to not enter one at all.

This problem is now solved, as is the tracking of shipments. Both of these are completely new sections of musdb that can be linked to objects (and in the case of shipments, exhibitions and loans as well).

Acquisitions

To access the acquisitions section in musdb, one can use the theee dots symbol in the main navigation (at the very top of a page) or enable the link to this section using the personal settings (accessible by clicking on one’s name).

Entering, filtering and searching acquisitions works roughly analogous to what one would do when managing loans.

Importantly, acquisitions are linked to a source of previous owner of the acquisition and possibly a source for the funding required to actually acquire the objects. Both of these fields are controlled by the contacts list / address book.

On object editing pages, acquisition processes can be linked to the given object using the object administration tab. Here, “acquisition” is a newly available select fields right next to the older fields for the acquition date, price, and currency. Once an acquisition process has been linked, a preview of the acquisition process and – if available – a preview of the contact information of the previous owner / acquisition source are displayed right below the select field. These new previews are now also available if an owner of the object other than the museum itself has been entered by the way.

Shipments

Shipments can now be added, listed and tracked in a similar way to acquisitions. They, too, build a whole new section of musdb that can toggled into the shortcut navigation by way of the personal settings – or they can be accessed using the three dots symbol in the navigation at the very top right of the window.

While everything else in terms of adding, listing and updating shipments works roughly analogous to the management of acquisitions, shipments can be linked to objects via the object’s destination tab and they can be linked to exhibitions and loans. Hence, there are new tabs on the loans and exhibitions pages for listing the shipments linked to a given loan or exhibition.

Image credits: Kulturstiftung Sachsen-Anhalt – Kunstmuseum Moritzburg Halle (Saale): Frachtschiff im Hafen von Göteborg, CC BY-NC-SA, photographer: Hans Finsler.

To allow administrators and users to get acquainted with the updated design and new features ahead of time, a preview follows below. The update will be distributed on January 11th.

A slight redesign

Sometime in late 2020 or early 2021 – when the reworked dashboard was released – we introduced a new, different design to musdb. While the old design had sidebars with a margin to the window border, the dashboard’s sidebar goes all the way to the windows left end. Where the old design positioned all page contents (minus sidebars and navigation) directly on the background, the dashboard features clearly defined boxes for each section of a page.

Bit by bit, pages that had undergone major updates (e.g. the institution-wide settings page and the image editing page) or were newly added altogether (task management; calendar) also received the new page layout. That way, we could slowly phase in the new design and hopefully managed to warm up users to what musdb would look like in the future.

But keeping two different page layouts side by side also comes at the cost of a harder maintenance and (obviously) a less consistent user experience. With the update, the “new” page layout will hence be extended to all pages.

A few additional improvements beyond even what the dashboard suggested have been made however. Sidebars are now used for displaying additional, directly usable information gained from a given entity’s data much more frequently (e.g. a copy-pastable address block of a contact is now displayed on the contact / address book page). And sidebars of editing pages now (almost) always come with an indicator displaying where the user currently is (“collection”) and the ID of the given entity. This hopefully allows users to better reference the entities down the line – especially when integrating musdb with other applications like Nextcloud (see below).

The clearly distinguishable boxes for each section of a page are now also used on pages for adding or updating entities. Again, to allow users a quicker grasp of where they actually are, these pages also come with much more visible headlines.

In other news …

In November 2022 we introduced maps on which the location of a museum or an object’s closer, unnamed location (e.g. a finding spot of archeological objects) could be determined by a simple click on the map. The colleagues from Baden-Württemberg requested to be able to enter geo-coordinates directly into an input field that interacts with the map, as they already know the coordinates they were to select. The maps thus now come with a button on the top right that allows the user to open a dialogue, in which the location can be entered using pastable geo-coordinates.

User-Defined Reports in musdb

Thus far reports in musdb were exclusively pre-written and provided together with the rest of the software. But in the end, museums are often bound by local regulations or already have forms or reports that may be generally used for a given purpose. Users can now define templates for report formats themselves and generate reports based on a search result, an exhibition (and its objects) or a loan (and its objects) independently. This may e.g. be used for automatically generating loan contracts.

To define a report format, one needs to hold the user role “museum director” and navigate to the institution-wide settings page. At the bottom of the page, one can upload a report template with placeholders marking the spots where object information is to be filled in by the system.

To simplify the implementation and improve security on the server side, only plain-text reports may be uploaded. HTML may be the most useful format for textual information with formatting; CSV for tabular information.

Timed Generation of Reports and Exports

One feature that was often requested, but thus far hard to implement, is to enable museums to generate exports automatically and without user input. This is now possible, both for XML reports and the new custom report formats.

Timed reports and exports are configurable on the institution-wide settings page. Each timed report requires the setting of:

• a start date (when should the first report be sent?)
• an interval (weekly, monthly, annual)
• a selector; usually a search query, written in the query language for searching objects
• mail address of a recipient

Note: As musdb is used by many museums together, we had to set some limitation on this feature. If an export file size exceeds 10 MB (which is also a size that many mail servers simply would deny for attachments), the configuration for the automatic report is automatically removed and a warning mail is sent to the recipient.

Literature entries

Probably the aspect of musdb with the most requests for improvements is the handling of literature entries. Almost any request for making literature entries interoperable with other software (e.g. Zotero for bibliography management or library catalogues) essentially requires a field for defining the type of a given literature entry. The same goes for most common citation standards: Within a citation style, the way of citing for books differs from that for articles, for webpages, or for archival material.

We have now added that rather essential field and – as that is possible using the new field – display a BibTeX representation of the literature entry in the new sidebar of literature pages.

User-specific defaults for adding new Objects

Museums have specializations, and so do people. It’s not rare for people to almost always enter objects of e.g. a given object type, especially if they are working within the context of a project focusing on a given collection (“Digitize all paintings of the museum”). Similarly, all users in the museum likely use the same units for values and measurements of objects.

To speed up the data entry in the case of such fields with unchanging contents, users will now be able set default values for “direct” text fields of an object. Unfortunately, setting defaults for links [e.g. to collections or spaces] and repeatable fields is much harder to implement and not yet covered by this update. Default values for the form for adding new objects can be set in the personal settings.

Take note that defaults can only be set for fields that are displayed on the object addition page. The most basic and generally required fields aside, one can determine which fields are available on the object addition page in the institution-wide settings.

Loans

To be able to better represent the process of a loan in a museum – from the request to the discussion with insurers to the final sending of the objects – we have now added a concise but hopefully reasonably complete checklist of the steps a loan takes within a museum in the sidebar of the loans page. The checklist covers the most common steps within a loan lifecycle and allows simply marking a progress in working on the loan. The last user to update a given entry in the checklist is displayed if a field has been updated at one point to allow following the progress later on.

We were also notified of a very obvious, but thus far overlooked case: Loan requests that are denied. We added the missing field to cover this status of a loan.

Finally, it is now possible to links loans to exhibitions. All loans of an exhibition can be listed together on a tab of the respective exhibition page.

To return to the checklist for a moment: One of the more noteworthy selectable points in the loan checklist is “metadata exchanged”. Adding this point may be opinionated, but hints at the next steps. In November 2022 we added the option to import loan object information following the upcoming EODEM standard. We hope to be able to implement an EODEM export before the update is pushed to the production instances, so that at least museums using musdb can handle loans with minimal duplicate data entry.

Exhibitions

As mentioned above, loans can now be linked to exhibitions. A new tab on the exhibition page allows listing all loans that happened in the context of the given exhibition.

The list of objects of an exhibition is now also redesigned. When linking an object with an exhibition, it is now possible to enter the exhibition room in which the object will be displayed. If this information has been entered for the objects of an exhibition, the list of objects of that exhibition will be grouped by the objects’ locations.

Integration with Nextcloud

Keeping with the theme of allowing for a deeper integration of musdb into the actual everyday work of museums, we have added the option to integrate musdb with a museum’s Nextcloud instance. If the Nextcloud integration is activated, a new widget will be accessible in the sidebar of most editing pages (e.g. for loans).

This widget displays a reference ID of the entity (e.g. LOA-000000005 for the loan with the ID 5). If this ID is present in a folder or filename on Nextcloud (say, there is a folder for everything concerning the loan, which will then be named something like “2022 Loan Brisbane [LOA-000000005]”), musdb can identify the folder or file as belonging to loan and list it in the widget. If the ID is present in a folder name, the folder contents will be listed in musdb.

For this integration to work, musdb connects to Nextcloud using WebDAV (unfortunately we needed to use some properties exclusive to Nextcloud’s and likely OwnCloud’s WebDAV interfaces, which makes our integration incompatible to other storage solutions that also use WebDAV like Google Drive). And to connect via WebDAV, it needs the information to get an authorized access.

To configure the Nextcloud integration, one hence first has to set the base URL to an institution’s Nextcloud instance on the institution-wide settings page. This only has to be done once per institution and simply gives musdb the information necessary to locate the Nextcloud instance.
Once a base URL for the Nextcloud instance has been entered, the username and password for Nextcloud (ideally an app token [can be generated in Settings > Security in Nextcloud]) can be entered on the personal settings page in musdb. Once those are entered, the Nextcloud integration is activated.

Institution and contacts pages

The layout aside, institution and contacts pages have only been minimally changed. A simple but maybe useful small widget has however been added in the sidebars of these pages: An address block to quickly copy-paste the address to e.g. a letter head.

Objects

Finally: Objects. Objects have seen the addition of a lot of new fields, mainly for administrative purposes.

On the “administration” tab of object pages, one can now reserve an object. If an object is currently reserved or will be so in the next week, an indicator will appear in the sidebar.

For logging an object’s history within the museum, we added a number of logs for:

• Damages to an object (Tab: “Restoration”)
• Conservation and restoration treatments for an object (Tab: “Restoration”)
• Scheduled Checks (Tab: Administration). These checks cover e.g. condition checks, but also audits of whether the object information in musdb is complete. This section comes with a notification that can be sent if a check is upcoming.

Deaccessions can similarly now be covered in musdb.

Again reaching for the most practical applications, we have finally implemented linking an object’s actual / permanent location as a “space” rather than simply identifying it using a text field. This way, it is now possible to search for objects that are not currently in the location their expected permanent location.

We also added some simple text fields that are often present in imports and suggested by the Canadian Heritage Information Network’s software requirements checklist. Namely: sex of the object (for biological specimen), the color of the object, and form of the object.

Post image: Courtesy NASA/JPL-Caltech.