museum-digital:blog - Development

Braving the Waves. Surviving in Times of AI.

2026-10-01T00:00:00+00:00

Now that the preview release of musdb’s new UI has finally been released, there’s time to write about the other sides of museum-digital. More usual struggles. Which is to say: New, larger waves of AI scrapers and bots that once again started to impair our services starting July this year.

We maintain that hypocrisy is not for us. I personally have been involved in discussions about the FAIR principles from 2015 onwards. And in 2015 they were already running under that label for a while. As a formalized document, they’ve been around since 2016, but disregarding the label, the principles and surrounding discussions go back, I guess, at least to the 1990s.

In the cultural sector, with many institutions being primarily tax-payer funded, we want to provide our data and services in a Findable, Accessible, Interoperable, and Reusable manner. Relevant here are primarily accessible, interoperable, and reusable.

Realistically, accessibility refers to two things: Immediate, universal barriers like a requirement for authentication should be avoided where possible. On the other hand, user-specific barriers should be reduced as much as possible as well: Human-readable pages should e.g. be designed with sufficient contrast.

If one sees other services and machines as users, which may be reductionist but is logically much more coherent than doing otherwise, then interoperable refers simply to a special form of accessibility: As data should be provided in a way that as many human users can use as possible, it should also be served in formats that machines can read. Ideally in ways that machines can read without further adjustments - following open standards.

Reusable then builds upon these. As the data is now freely accessible by others, setting appropriate licenses allows for the creative reuse of the data.

Ten years after the FAIR principles were formalized, we finally have someone who reuses our data on a massive scale. We should rejoice. Unfortunately all our previous conceptions of how that reuse might take place turned out to be wrong.

Celebrating AI Scrapers?

AI scrapers flooding web services, and especially larger, well-established infrastructures with requests has been an ongoing story for some two years now. Last year museum-digital was met with a first large wave of AI scrapers. See the previous blog posts (1, 2, 3).

On the one hand, using our data for training AI models is undoubtedly reuse. It is even productive reuse: It’s a small contribution to the training of technology that, at its current quality, sounded like science fiction 10 years ago.

On the other hand, this type of reuse takes place without attribution. AI scrapers work based on scale; Machine-readable APIs are uninteresting to scrapers. They suck up as much data as possible, as quickly as possible. Any page-specific adjustments, such as finding and using the API, would reduce the speed of scraping. The requests are so numerous, that they routinely crash servers. Last but not least, those prominently benefitting from the AI hype are far from sympathetic people.

But principles are principles.

It, again, cannot be denied that what AI scraping is eventually aimed at is reuse. Even though they use HTML rather than the APIs we lovingly carved for them, they interoperate with our services in the way most accessible to them.

The problem is, that they are not playing fair. And if the previously unlikely number of requests leads to servers crashing and shutting down, then all principles were for naught. A service that is providing no data at all cannot provide them FAIRly either.

At museum-digital we have seen the new requests as an opportunity to improve our software. If it can withstand the onslaught of thousands of bots a second, it is confirmedly battle-tested and stable. Improving our publishing software on the other hand also meant cutting out overly resource-hungry functionalities. Last year we dropped most publicly available, server-side PDF generation capabilities, we moved the IIIF APIs to an separate configuration that is specced to be able to fall over without impairing the rest of our services. We improved the parsing of search terms and introduced rate-limiting (how many requests a single IP can perform for a given time).

The new wave of AI scrapers since July is even bigger than last year’s. Those improvements were not sufficient to keep our primary server, which hosts the production databases, stable.

From Each According to Their Ability: Load Shedding

We continued on last year’s course however, analyzing the scrapers’ requests, their influence on server stability (which is to say, which requests were especially resource-intensive) and adjusted our setup accordingly. Note, that we did not scale up: museum-digital runs on exactly the same bare-metal servers today as it did two years ago.

In analysis, we identified two types of especially resource-intensive browsing behaviors that are almost entirely limited to scrapers (as well as some very engaged researches, to whom we express our regret for now regularly banning them):

Scrapers follow links on search pages, especially the facet search, and end up combining more and more search parameters. It is unlikely that a human would perform a search for objects that “Are related to Berlin, and related to Germany, and related to Institution X, but not related to Institution Y, and also related to a time between 1990 and 2000.” For an automated scraper it is normal behavior.
If a published object page has been updated, a snapshot of its current state is automatically saved to provide an archived version of the page at a given time. This is thought mainly for researches to be able to cite a given state of the page. In everyday operation, archive pages should be barely used. They are also de-listed from search machines. But saving snapshots again and again leads to the existence of a large number of links and separate pages scrapers can scrape to no benefit to anybody.

Both functionalities are legitimately useful but scale badly.

We hence introduced load shedding: Whenever a search is performed or an archive page is accessed, the current server load is evaluated. If it is above a certain threshold, the server does not perform complicated search queries (depending on how high load is, it is restricted to a minimum of three search parameters) or does not load the archive page. Instead a warning is presented, that the given action is currently unavailable and a custom HTTP error code is sent. If the same IP causes that error code to be sent twice, the IP is blocked for a while.

As of the time of writing writing, we have thus blocked a total of 131,014,530 IPs in two month, with 5,802,771 being currently blocked.

This approach follows our principles: We try to be fair with everybody. But if users are not fair and do not follow advise, we don’t feel obliged to continue playing fair either.
Unfortunately this approach works on the level of blocking whole IPs. As “residential proxies” - back in the days we called them botnets infecting home routers - have become an ever larger problem, it is not unlikely that actually interested, normal users also unknowingly host an AI scraper at home. I fear I’ve already encountered one such case, were a colleague could not access museum-digital for seemingly no good reason from her home network. Otherwise, it is surprisingly effective - the number of requests has barely been reduced, but our services are essentially stable.

The only crash since happened the day before yesterday and was caused by the combination of bots and an error in our controlled vocabularies - the latter of which is entirely our fault.

Numbers Games: Logging Strategy

After the above-mentioned actions had effectively returned stability to our publicly accessible portals, one issue remained for internal services: File uploads were incredibly slow. It turned out that the large number of requests being permanently logged overburdened the SSD. We have hence disabled the general access logging, restricting ourselves to logging requests that either cause errors or trigger slow responses. This reduces the ability to react well-informed to new waves of requests and actual attacks. Thankfully those also commonly trigger actual errors we can still identify.

Fairness, FAIRness

With these actions we have again averted the need to use more drastic strategies that would limit accessibility to users - human and machine alike - like blanket bans on whole regions of the world or the installation of software such as Anubis. And, again, we have not yet needed to improve our hardware (and pay the price for that), even though that might be in order eventually. It might be wise for other reasons as well.

A holistic picture of one’s setup - software, hardware, systems administration - and the capability to think and act upon these together helps a lot.
But, if anything, hosting in a FAIR, principled way still works in these trying times.

info

AI Use

Post image generated using Anima Base v1.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Pre-Release of musdb's New UI

2026-09-30T00:00:00+00:00

For the last months, a new, fully reworked user interface for musdb has been in the works. Two previous blog posts discussed the reasons and the scope of the overhaul (1, 2). The latter of these also came with our target timeline for the release. Which is to say: It is time for the first pre-release.

Starting tonight, an overlay will appear, guiding users of musdb to try out the new UI. Which begs the question: Is the new UI ready for general use. Realistically there can be no general answer to that question. We have tried to prioritize the development of commonly used functionalities, while many less commonly used sections of musdb are still entirely missing in the new UI. Before a (nearly) comprehensive list and categorization of the features that are thus far missing in the new UI is presented, let’s look at a few key new features and benefits of the new UI.

New Architecture, New Possibilities

As has been outlined in the previous blog posts multiple times, the most challenging and yet basic aspect of musdb’s UI rewrite is the move to a different architectural approach. In the old UI, each page is its own script, with all these scripts using the same functions in the background. Opening a page - or e.g. saving a form - thus requires a full page reload, including a reload of all the data that remains unchanged. On the face of it, this sounds inefficient in terms of performance and bandwidth use. It is. It is also noticable in immediately perceivable characteristics like the view jumping back to the top of the page when a form is submitted.

Refactoring musdb’s backend code to a more reusable, testable and logical structure is an endeavour that has been ongoing for over half a decade now. The occassional references to musdb’s growing API in the blog were the immediate benefits of it - functions that were used to provide the regular, human-readable user interface were also made available for machine-to-machine communications.

The new UI is then the result of the final completion of that refactoring: It is a Typescript application built entirely on top of musdb’s API. On the one hand, this means that potentially any action that users can perform using musdb’s graphical UI will also be available for automated use using the API. On the other hand, it simplifies things in development: As we are ourselves using the API everywhere, good maintenance is simplfied a lot (see also).

Given this new architecture, the user interface is entirely processed in the browser. If forms are submitted, only the changed values are transmitted to the server. The required bandwidth is thus reduced greatly, once the application has loaded. Similarly, navigating only reloads those sections of a page that actually change, reducing computing load for server and client alike (and thus increasing performance).

Automation via Webhooks

Only possible due to the new architecture has been the addition of a webhook system in musdb, which offers a simple way to react to updates in musdb in an automatic way. Webhooks can be configured in the institution-level settings pages of the new UI. Note that, as the webhook system requires the use of the new architecture, webhooks configured in the new UI will not take effect when one continues to work in the old one (while both are available). See also: Wikipedia on webhooks.

Design Overhaul

Doing a re-design from the ground up naturally allows for the creation of a much more streamlined user interface. Going by the (albeit limited) feedback we have thus far received on the new UI, we have indeed succeeded in creating one. Similar functionalities that were spread over different sections of a page or different pages altogether have been moved into the same section of a page; redundant or confusing options have been removed. Where things could remain as before - as even the old design received its fair share of praise despite its shortcomings - they remain roughly the same.

The previous blog posts have already provided some in-depth glances into the new UI. Importantly for this post, the design overhaul will include the removal of some redundant, badly thought-out or effectively counter-productive funtcionalities.

List of Features Not Yet Implemented

In the following, a simple list - in some cases with further remarks - of the features the old UI offered, that are not or not yet implemented in the new UI will follow. If some feature is not to be found in the list, it is most likely already implemented.

Features That Will Certainly Be Implemented, But Are Not Yet

Whole Pages

The general search page for object images has not been implemented yet
Visitor counting

Notification system

Misc Tools

There can be accessed in the old UI using the “Tools” subsection of the dashboard or the similarly named “Tools” submenu of the navigation.

QR-code generator
Calendar
Process list
To-Do list
Knowledge management / mini-wiki
Hyperlink checker
URL Shortening
Consistency checks for migrating data from free text fields to controlled ones

Features Available in Multiple Sections of musdb

Notes (for any page)
Nextcloud integration
UI for users with less permissions
The UI should hide submission buttons etc. where a user has no permissions to perform the relevant action. This is by far only implemented in very few cases (while permissions are enforced in the backend).

Object Pages

Overview pages
- Sorting menu
- Watch list
- List view / editing most fields in a tabular view
Sidebar
- Warning if an object is currently reserved
- Object record editing status (e.g. open, locked, archived)
- Export settings for XML exports; quick exports work
- Improvement suggestions
Basic data tab
- Editing translations
- Tag suggestions based on the object image
- Tag suggestions based on the object description
- Ability to add sources / citations for events
Addendum tab
- Data field-specific setting for publication status
Administration tab
- Deaccession (administration tab)
Location tab
- Links to loan management
Most of the restoration / conservation tab
Data history tab
- Versioning UI
Custom / user-defined object editing interace
Comparing objects

Institution pages

Institution editing page
- Tab for self-categorization of institutions / survey
Institution-level settings
- Configuration of scheduled, automatic generation of exports

Collection pages

Hierarchy & sorting
Editing page
Tagging for collections

Series

Series editing pages
- Hierarchy
- Sorting objects of the series
- Sidebar
  - Setting for visibility of the series on the public institution page
  - Option to group related vocabulary entries for revision in nodac

User Settings

Editing pages for other users
- Configuration of user permissions
Own account settings
- Export of own account data
- WebDAV settings (for preparing independently run data imports)
- Notification settings

Contacts / Address Book

Contact editing pages
- vCard export
- List of linked literature entries (to be implemented on the literature search page)

Spaces

Hierarchy
Editing pages
- Sensor data display

Exhibitions

Editing pages
- Tab for publications linked to the exhibition

Vocabularies / Backgrounds

Unreleased catalogue raisonne feature
Statements

Features That Are Implemented But Certainly Need to Get Better

Object editing page
- Linking recently linked entities on base tab

Features That Will Be Removed

General page features
- Bookmarks
  Browsers are much better at bookmarking than musdb. And musdb runs in a browser anyway.
Object page
- Tab: Provenance research
  Feedback from provenance researchers (and common sense and experience) informed us that the current implementation is altogether ill-conceived. A whole separate, general section of musdb for managing research projects might eventually be added to actually do what we aimed to do here.
- Entering free text data from an appropriate prepared QR-code
- “Ask an expert” tool
Article section (never really used)
Podcasting features (never published)
Series page
- Offline HTML catalogue
  This solution has not been properly maintained in a decade. With a generally wider availability of internet connections, API-based solutions are also much more flexible.
- Configuration for display modes of published series pages
Chat
Setting for the start page of a user upon logging in
The primary and most reliable means by which we can communicate updates to musdb (and other museum-digital tools) is the blog, whose feed is embedded into the dashboard. Making sure that everybody occassionally visits the dashboard thus also means that the likelihood that users know about updates is increased. On the other hand understanding users’ bug reports regarding login and startup is simplified if there is a more unified workflow around logging in.

Features to be Discussed

Object page
- Social media buttons setting. With the selection of available social media buttons and current usage trends, it might be wiser to simply scrap the option.

What Does That Mean For the Roadmap?

As one can quite easily see, a lot is left to be done. But the bulk of the work has been done and we are perfectly on schedule. For anybody who relies on the features that are still missing, the old version will remain available until around November 22. For the majority of users, the current state of musdb should actually already support most or all of their requirements better than the old UI.

Until November 22 both versions are available for users to slowly transition over to the new version. We are curious for feedback and continuously push new updates to the new version.

info

AI Use

(Post image generated using Anima Base v1)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

musdb Reimagined - Update & Release Schedule

2026-09-14T00:00:00+00:00

In May 2026 we announced an upcoming large-scale rewrite of musdb’s user interface. Now, it is taking shape and we can announce an actual timeline.

Recap: Scope and Reasons of the Rewrite

musdb as a tool has been around since sometime around 2009 or 2010 - soon after the wider inception of museum-digital. Back then, its scope was to provide an input interface for the recording of publishable museum object data. Since then, the basic architecture to the user interface has remained largely unchanged. While the scope grew to musdb aiming to be a full-featured collection and museum management system, many features that were added were only added “on top” - far enough out of the way to not disturb legacy users or to break existing workflows. In a nutshell: musdb in its current form is quite good, but it’s an application whose user interface has grown without a clear roadmap and a systematic reconsideration and streamlining of the user interface for over a decade.

Since 2009, web development has changed. We do not need to worry about the Internet Explorer anymore, browser compatibility is much better, and web applications can be much more interactive with modern browsers and hardware. To fully make use of these larger advances in the development of musdb, a general change to its architecture is necessary however.

As the architecture changes from a traditional web site with many independent pages to a single page application, the rewrite is indeed that: Instead of generating pages on the server and only displaying them in the browser, musdb’s user interface will be fully generated in the browser, with only the relevant data being queried from the server using dedicated, documented APIs. Even to keep the same functionality, we would need to re-implement its presentation. This means that the rewrite is also a perfect opportunity to streamline the user interface and to reconsider previously unused or overseen functionalities.

On the other hand: With the new user interface changing everything display-related on a technical level, that does not mean that much has to change on a conceptual level. The rewritten musdb will feature the same basic division of pages - lists, editing pages, and pages for adding new entries -, it features the same idea of a tabbed editing pages and - with few exceptions - the same data fields and order thereof.

A Sneak Preview

Where much stays the same, taking a small glimpse into things that will change might be in order. In the following there are a few such previews with small screen casts. Most of these focus on overview / list pages, which are already in a release-ready form.

The screencast shows how menu entries can now be toggled into and out of the quick access navigation. The quick access navigation and the list of other sections of musdb have now been moved closer together.

As had already begun in the old UI of musdb, more and more types of overviews now feature additional fine-grained filter settings.

Display Modes of Overview Pages

All overview pages in the new UI of musdb feature the same three display modes: grid, list, and table.

Batch Editing On All Overview Pages

All overview pages now feature the basic functionality of selecting entries for batch editing. The actual batch editing capabilities will vary depending on the respective type of entities listed. For now, all publishable entities can be published or unpublished in batch this way. Further batch editing functionalities will be simple to implement upon request.

Interactive Overviews

List views of publishable entries in the database now show whether the respective entries are published directly within the overview in a uniform way. Clicking on the publication status toggles it and publishes hidden entries or hides published ones.

Timeline

Now, when will the new UI be done? And when will it be released?

As of now, the development is progressing smoothly and the required building blocks for almost all functionalities are there. Depending on what features one uses, the new UI can already replace the old for most common use cases. To fully do so for common use cases means that at least object editing pages need to be implemented fully. This should be the case as of late September. In the following, seminars and a time period where both versions can be used side-by-side will be had to ensure a somewhat smooth transition from the old to the new UI.

Date	Action
Sept. 14, 2026	Blog post
Sept. 28 - Oct. 4	A dialogue will appear on the dashboard of musdb giving notice about the upcoming re-design. It will include a link to the new version.
Oct. 30	The new version will be presented at the German user meetup (online). The update will be finished at this point.
Nov. 1	The login flow will be updated to link users to the new version first. Here they will see a dialogue similar to the old one, giving notice about the re-design and allowing users to use the old version for the time being.
Nov. 2 (1 PM)	Seminar on the update (German), Hosted by the museum-digital Deutschland e.V.
Nov. 6 (1 PM)	Seminar on the update (English), Hosted by the museum-digital Deutschland e.V.
Nov. 16 (10 AM)	Seminar on the update (German), Hosted by the museum-digital Deutschland e.V.
Nov. 22	The dialogue on the dashboard disappears, the old version is removed.

info

AI Use

(Post image generated using Anima Base v1)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

musdb Reimagined - A Preview

2026-05-05T00:00:00+00:00

musdb is almost as old as museum-digital itself. From its beginning sometime around 2009 or 2010, it has significantly grown in size and scope. Originally, it was created as an input interface for the publication of museum objects. Today, it aims to be a full fledged collection and museum management solution, capable of covering publishable object and exhibition data, internal aspects of collection management such as climate sensor readings and loan management as well as providing general knowledge and project management capabilities.

Over all that time however, the basic architecture of musdb remained basically unchanged, with new functionality fit in or added on top where it seemed suitable. Additionally we have been struggling with the duplicate implementation of several functionalities at different times, where one implementation offers a very simple access to the respective data (e.g. a free text field), while the other offers a representation of the same data in a much more fine-grained, precise, and more interlinked way, whose requirements are however harder to fulfill during imports and that requires a more thorough documentation generally.

We routinely get very positive feedback from colleagues working in museums, that musdb is comparatively easy to use and logical. The age of the architecture as well as the many small inconsistencies in terms of the user interface that have crept in over time however suggest that it is time for a more radical reimagining of what musdb looks and works like.

This post is a first small preview to what musdb should look like starting in the third quarter of 2026. After a more thorough discussion of the reasons for rewriting significant parts of the publications, some first glimpses into the new version are provided at the end of this post.

Reasons

Technical

musdb is primarily a PHP application. As such, the internal logic as well as most of the user interface rest on the server. Almost all of the user interface is rendered into HTML on the server and then sent to the browser to generate what a user can view. Some functionalities that require additional interactivity, are presented in overlays, etc. are implemented in JavaScript and processed directly in the browser, with the JavaScript drawing its data both from the HTML as well as APIs (some internal, some documented for internal and external use).

Speaking in modern web development terms, PHP covers model, view, and controller on the server, with JavaScript extending the view somewhere half-way. Logically, this is problematic. In brutal practicality, it means that musdb’s JavaScript components are a collection of small scripts always dependent of some external (PHP/server-side) preparation and not logically interacting with each other. This prevents lots of the benefits of a more structured programming from being used altogether.

Our aim with the technical architecture of a reimagined musdb is to design musdb as a modern single page application with all data being provided by clearly documented APIs. This will come with several benefits:

The view layer is entirely processed entirely in the client, providing a clear logical division between the different tasks
With a modern build infrastructure, code can be reused much more than thus far, providing a better maintainability in the long run
musdb will come with barely any browser-level page reloads. If an API is queried, the application can clearly communicate what is being loaded and for what purpose.
As musdb’s UI will be entirely built based on the same API as can be used by external developers, that API will be fully feature-complete by the end of the process.

User interface

The general structure of musdb’s user interface has proven itself to be quite good. However, as musdb has grown over the years, a lot of small inconsistencies and opportunities for improvement have crept in.

Take the navigation alone:

To access object groups, one can always hover over the three dots sub-menu of the navigation section on the very top right. If one enables the menu option “object group” in the personal settings (accessible by clicking on one’s name), one can also quickly access it via the quick access navigation (the second line of the navigation). As such, simply accessing a central part of musdb is spread over three different sections of the navigation.
There are two question mark icons in the navigation. One, in a speech bubble, links to communication channels and contact persons that one can use to seek help. The other provides links to the handbook. On the one hand, there is a clear logical differentiation between the two. On the other, both look similar and follow rather similar aims. That the two sub-menus are divided rather than being merged into a single sub-menu is not intuitively apparent.
Clicking on the page title or the colored circle at the right of the navigation opens the context-independent search function. Both bear no resemblance to an icon associated with searching and actually follow entirely different primary aims by themselves.

Many similar issues exist around musdb. In most cases, these somehow make sense and are logical in themselves - but they still contradict a seamless, intuitive use of the application.

Additionally, there are some functionalities that in themselves contradict a more streamlined use of musdb. One such example also accessible via the navigation is the option for users to set their preferred page to be referred to right after they log in. musdb has been built more and more on the premise, that users will start their use of the application on the dashboard, and thus some features - like requiring users to enable two-factor authentication for regional administrators - can be bypassed by selecting any non-dashboard start page. The option to select a non-standard start page will hence almost certainly be scrapped in the new version of musdb.

Preview

For now, only the basic scaffolding, most of the navigation and some of the overview pages have been implemented in the new version. Screenshots of these can be seen below. Importantly and yet missing, each different section of musdb is planned to come with an icon representing it in the navigation (top left).

Note also, that nothing about the new UI is set in stone. These are first drafts. More will follow soon.

The dashboard in the new UI of musdb will offer a reduced, more streamlined and - importantly - subdivided selection of tabs. At the top, the new navigation with the yet missing icons is missing (all icons thus far being a location pin).

The navigation has been reduced into two basic sections: On the left, there is the access to the different main modules of musdb. This section on the one hand comes with links directly visible and accessible from any page. In the same section of the navigation, a hamburger menu offers access to those modules not enabled for quick access. Toggling entries into or out of the quick access navigation is directly accessible in the same place.

The screenshot depicts those menu options not enabled for quick access.

The two help menus in the navigation have been merged into one. As we saw when we first introduced the option to create user profiles in musdb, providing additional information on contact people makes users more likely to actually ask for help if they need it. Hence, some data from the user profiles is directly integrated into the navigation.

The navigation features one’s profile picture, if one has uploaded any. Hovering over it gives access to the account settings. In this, the new UI tracks many popular websites and services.

Overview and search pages roughly follow the current logic. Some exceptions aside, search results can be displayed as a grid, a list, or in a table. This screenshot shows them in grid mode.
It is also the first in this list of screenshots to display the new musdb UI in dark mode.

Overview and search pages roughly follow the current logic. Some exceptions aside, search results can be displayed as a grid, a list, or in a table. This screenshot shows them in list mode.

Overview and search pages roughly follow the current logic. Some exceptions aside, search results can be displayed as a grid, a list, or in a table. This screenshot shows them in table mode.

info

AI Use

(Post image generated using Anima Base v1)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Development, December 2025

2026-01-12T00:00:00+00:00

December 2025 was an interesting month for museum-digital. An update to the PHP version used as well as a flood of requests by what is most likely AI scrapers forced us to make changes for improved stability, reducing and reformulating features rather than adding new ones and working on matters of systems administration over purely matters of code quite often. Add to that the long-promised update of the terms of use for German museums to more structured and lawyer-approved ones, and you get yet more small changes that do not directly concern the work of museums with museum-digital but rather improve the necessary context.

musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

Hovering over an object image in the tile view now also displays the inventory number.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

Two new options allow disabling user accounts and setting expiry dates for user accounts.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

A new tab on the user page lists all user agreements for musdb that the user has agreed to and when they did so.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process - no matter if it is successful or fails - is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

The dashboard in musdb now features a feed of recent news relevant to the development of museum-digital and whatever is going on regionally. The posts are sorted chronologically.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
Re-introduce option to repeat linking to the last used linked object
Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

The reworked default image page.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

The reworked image search settings overlay.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
Object groups with only one object are henceforth not displayed and linked on object pages anymore
Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
Updated Swagger UI to version 5.30.3

nodac

User-provided comments / hints have been removed for times (see above)
Tooltips for linked objects now display which institution an object belongs to
- This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

info

AI Use

(Post image generated using AI)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Trimming.

2025-12-29T00:00:00+00:00

In the last weeks we struggled with server stability. As written before, the critical, resource-heavy and publicly available tasks have for a long time been the generation of timelines (and thus complicated search queries) and on the other hand those involving the processing or generation of large files; namely the IIIF API and PDF generation.

In the last post, I detailed how we severely restricted the availability of the public PDF generation functionalities in museum-digital according to available system resources. That, as it turned out, was not enough to bring reliable stability to our systems. After the server fell over on December 26th once more, we hence moved the IIIF Image API into the same PHP setup used for PDF generation - meaning that any user/IP can only request the API 10 times a minute and that for any instance of museum-digital, only one PHP worker serves it. This allowed us to severely reduce the maximum available resources per worker for the frontend outside of those two use cases (where the IIIF Image API may use up to 80 MB of RAM, no other part of the frontend will go beyond 5). Since then, the system runs as smoothly as if AI scraping had never become an issue.

A Limited Goodbye to IIIF & Server-Side Image Manipulation

Now, what does that mean in practice? On the one hand, we have not fully removed the IIIF image API. All links generated using it remain valid and will be served, even if comparatively slowly.

On the other hand the user experience with viewing the images in a IIIF viewer will be significantly worse, even though this strongly depends on the IIIF viewer. The most popular “full” IIIF viewers being Mirador and Universal Viewer, significant problems (or a complete inability to use an object’s images) are to be expected with Mirador. Mirador in its default configuration loads multiple segments of an image separately to then assemble the displayed image from those - with the creation of the segments happening on the server, thus consuming resources centrally. It also seems to set extremely low limits on accepted response times, which museum-digital’s IIIF Image API now regularly exceeds due to the aggressive rate limiting. Simply looking at the demo installation of Universal Viewer, the software seems to be much more targeted in its API calls and might still work well despite the restrictions.

As far as I know, there are no published numbers on the market share of the different IIIF image viewers. And about whether IIIF viewers external to whoever provides the API are actually regularly used or not. The most jaded - and likely true - assumption would be that the share of users who use IIIF without a viewer hosted next to the API is miniscule and that most users will use one of the abovementioned. Our experience, once again, seems to support that hypothesis: We released our implementation of IIIF 2 in 2020, but essentially nobody noticed before we also started hosting a IIIF viewer.

As we do use Mirador as a viewer, assume the “visible” IIIF image API at museum-digital to be more or less broken. Developers and those making direct use of the API without our installation of Mirador can still benefit from the API. But those are comparatively few.

The radical restriction of resources provided to the IIIF Image API is thus likely indeed a goodbye to IIIF, if a limited one. The basic idea is great - to create a unified way to reference parts of an image (or later a wider media file) and annotate it. In times of significantly increased bot activity, reduced funds, and foreseeably rising hosting costs, our example may be an early sign that the decision to realize that aim by specifying an API to be implemented by the data providers restricts the ability to fully support IIIF to very well resourced institutions. And as funds are shrinking, that is less and less institutions. Let’s hope that the most basic need IIIF wished to fulfill can be achieved in a different way in the future; one that is accessible to anybody. Realistically this means that computing would need to happen on the client PCs, not on a server.

To end the saga on a more positive note: Since we limited the IIIF Image API, our systems run wonderfully smoothly again and we were able to reduce the overall rate limiting on the rest of museum-digital’s portals. We will monitor the situation and increase the limit slowly to allow more simultaneous API requests without risking stability.

Communications

Second, the whole ordeal posed a challenge to our communication channels. If any significant error occurs anywhere on museum-digital, I personally am sent an encrypted error message via mail. Usually. In this case, the primary component falling over was the PHP server, which is also responsible for managing the sending of mails. If a service fell over, the primary way to learn of it was receiving mails about that instead. Reaction times were thus worse than they needed to be. This means that we need to improve our monitoring.

On the other hand there was the issue of explaining what was going on. We had a thread about it in the forum, which few people read. We had the blog posts. Which few people read. We lack (or lacked) a unified source of information about current events that we can assume people to read. The blog could and should be exactly that.

At the top right of the login screen of musdb, the two most recent blog posts from the respective region as well as from the “development” category of the blog have been shown for years. Then we turned on the “remember me” feature by default, which means that people only very rarely see the login page at all anymore.

The first page most users see upon logging in or opening musdb while logged in is the dashboard, the default subsection of which previously offered a summary of the database contents a user has access to, a tile for writing personal notes to oneself, a tile with messages from the respective regional administrators, a tile for the integration of a discourse forum, and links to the museum elsewhere on the web.

The summary of database contents and the links to the museum elsewhere are certainly useful. The other features not so much. Checking their actual use revealed that barely anybody used any of the note-taking features (likely also because musdb itself offers better alternatives elsewhere), while the discourse integration has not been in use for years. The very first features one sees when opening musdb were thus largely unused, wasting space that could be filled with a feed of relevant blog entries.

And so we removed the unused features and replaced them with a more prettily designed feed. This feed now contains the two newest blog posts from the development feed in the user’s language, the regional or national feed (again in the user’s language) as well as - importantly - the English-language development feed. None of the most recent development-related posts were translated to any language other than the original English, mainly because the time was better spent trying to alleviate or fix the issues than describing them in yet another language. Besides, most people know enough English to grasp the posts. And for those who do not: Community contributions to the blog - also translations for those who do not - are always welcome.

The dashboard in musdb now features a feed of recent news relevant to the development of museum-digital and whatever is going on regionally. The posts are sorted chronologically.

info

AI Use

(Post image generated using Z-Image)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Cleaning Out Our Closet

2025-12-22T00:00:00+00:00

Since the last post (i.e. the update to PHP 8.5 amid an onslaught of AI scrapers) and the later introduction of much stricter per-IP rate limiting, the stability issues around md are better - but they are not yet completely resolved.

As such, we have expanded our efforts in rewriting and reformulating key resource-intensive functionalities for increased stability. Different from before, we have also started to fully remove or disable functionalities that are simply not tenable anymore under the current conditions.

PDF Generation

Thus far, there were two basic types of PDFs that were generated (on the server side) in museum-digital’s portals: PDF representations of object pages (“data sheet”) on the one hand and PDFs encapsulating all images of an object in one document for easy printing.

The latter was - simply by nature of its envisioned task - extremely resource-intensive. All image files had to be loaded from disk, embedded into the PDF, compressed and served. The option had thus been available for fewer and fewer objects. Where it was originally available in case of any object with more than three images, it was later limited to objects of less than 40 images. As such, its availability was increasingly hard to communicate clearly, while its usefulness was relatively reduced with the introduction of a new download option for all images of an object. Its natural resource-intensiveness remained a problem however, and as scrapers will click any link they can find, this type of PDF generation continued to be used quite regularly (every few seconds before the recent surge in bot activity). As of last week, the functionality has been entirely removed.

The “data sheet” PDF generation has been further limited as well. As stated in the previous blog post, its usefulness is significantly reduced with the introduction of a print stylesheet (you will get better results simply pressing CTRL + P on an object page and printing the page to PDF). Nevertheless, it remained rather popular and has not been removed entirely. To reduce its impact on server stability, we however further limited its availability: If the server load is any higher than comfortable, the PDF will not be generated and an error message will appear. If the load is high (up from around 70% of comfortable) and the user’s browser language is not the default language of an instance of museum-digital, the same error message will appear.

Failed Search Pages

If a search query for objects fails, users are forwarded to a failed search page, on suggestions for alternative search queries are made. This is essentially the same as Google automatically suggesting corrections when search terms contain typos. Identifying the alternatives and offering previews for each is not free. As it is simply suggestions, the benefit or general accuracy of the suggestions fluctuates from case to case.

Now, looking at the logs, we had a large number of queries for non-existing entities - obviously scrapers who were trying out different IDs after analyzing the URL scheme. Each of those queries was executed and then forwarded to the failed search page, triggering the loading of suggestions and previews and thus further using resources on the server for little benefit (besides getting more links to scrape). We have now introduced a similar logic to the limitations on the data sheet PDF generation. Suggestions and previews are only generated when server load is comparatively low, with non-primary language users being slightly disadvantaged vis-a-vis primary-language users in an instance.

Timelines

Timelines remain popular - and a problem. A very common type of query we would see in our logs would combine timelines with searches by start and end date. This was likely due to another possible loop of endless URL generation for scrapers - specify a timeline until it forwards to search pages for a given timespan, then open the timeline for that timespan. Exactly that behavior has now been made impossible. If a search by a timeline (“start after”, “end before”) has been set, timelines will not be offered in the sidebar anymore. Trying to generate them for such a search using URL manipulation or the API will return an error page.

Search: Cleanup, Image Search & Checking Entity Existence Early

A more messy way of optimizations hit the core of the object search. In around 2021, we introduced a new search logic. Almost all pages relying on the core search logic - search overview pages, maps for objects, timelines, were adjusted to work with the new logic. The only exception from this was the image search. Still, as the new search logic re-used some of the old search logic’s functions, we kept both as separate classes, which grew over time. Simply loading the new search logic took about one ms (without OPCache enabled, measured through PHPBench). This sounds like little, but hints at a lack of modularization of the code and gains relevance with many unpredictable requests with servers automatically spinning up and down.

And indeed, in writing the new search logic, we did not modularize thoroughly HTML generation, query building and database querying. With last weeks updates, there are now separate classes for each of these and functionalities relevant only to the old search functions have been moved to class managing the image search logic. This reduces startup time for only the new / main search logic by about half (ca. 0.6 ms).

Second, we reduced the available search options for image searches. The remaining search parameters are either those actually relevant to the images or those linked to the controlled vocabularies. As a positive side effect, this also solves some issues in communication: Making it legible what the difference between searching images by their own license and by the license of (unrelated) metadata of objects the images are linked to is, is complicated.

Finally, as stated above, the logs revealed a lot of queries for objects linked to e.g. either entirely non-existent places or places that are not linked to any object in the instance of museum-digital altogether. When a place or tag is queried, we hence check whether there exists any public mention of the entity in the current instance of museum-digital during query building. If there is no link at all, it is clear early on that a more detailed (i.e. costly) query combining the search by that entity with other parameters will not return any results.

The Current Situation

All these improvements help, but a look at the current real-world numbers is warranted. On the one hand, the database server now often falls down to half or even less of the expected server load. This is a positive sign for system stability outside of peak times.

On the other hand, there are noticably spikes in the morning (around 10:20 in Germany) and in the afternoon (starting around 5 p.m.). The spike in the morning is likely related to the start of workdays and has led to the server falling over multiple times last week. This can likely be fixed only with a further tuning of the PHP-FPM settings. The spikes in the afternoon and early evening on the other hand remain hard to explain, but are altogether much less critical.

We’re on it.

info

AI Use

(Post image generated using Z-Image)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Updates, AI scrapers, and Resilience

2025-12-09T00:00:00+00:00

Between Thursday last week (November 27th) and yesterday (December 6th), museum-digital has seen its most unstable week in about four years. Now that the dust has settled a bit, there’s finally some time to discuss what happened and how we managed to tackle the multiple issues leading to the (very noticeable) instability.

Background

Scrapers

There were (or are) two factors simultaneously pushing our servers to their limits and requiring changes. On the one hand, scraping of museum-digital has gotten even more aggressive. Where we usually has something around 10-30 requests per second across all of museum-digital a year ago, we had around 300 two weeks ago. Right now it’s often between 500 and 700. This number excludes any access to static files.

As I’ve written elsewhere, the scrapers are mostly noticable by coming from IP ranges in Asia or (to a lesser extent) the US. On the other hand, the IPs change constantly and user-agents etc. resemble regular users. Likely they simply use an actual chrome browser for scraping. Which is to say, attempting to block them is futile. Worse yet, attempts to block scrapers would likely also impact some real users.

Fortunately museum-digital is run on dedicates servers paid by time rather than by compute. The onslaught of scrapers thus has no financial impact on us. But the scrapers still use resources, and as they try to scrape as many different pages as possible, it is much harder to optimize for them than it is to optimize for actual human users (see this article on a similar issue at Wikimedia).

Either way, AI scrapers can result in improvements. Viewed positively, they essentially act as a free stress test on a service and enforce efficiency in all aspects. If most pages are optimized for performance already, scrapers will find the unoptimized ones and bring down a service by overusing those. Which is to say, they help to identify yet unoptimized scripts/pages/classes and enforce that necessary changes are made. At museum-digital, there are three main weak spots that are hard to optimize: timelines, image manipulation (including the IIIF API), and PDF generation.

PHP

On November 20th PHP 8.5 was released. Thus far, museum-digital had been running on PHP 8.3 for web hosting and PHP 8.4 on the command line. When we attempted to update to 8.4 last year, the server fell over. This was mainly caused by the IIIF API (and thus, image manipulation via libvips).

Dependencies at museum-digital are (like pretty much universal with PHP) handled using the package manager composer. Setting up a new instance of museum-digital, composer (managed on version 8.4) required PHP 8.4 or later to run - the new instance was thus unable, being stuck on version 8.3 for hosting.

That leaves two options: Either to set up composer using PHP 8.3 again, or to simply update everything to the current version. While PHP 8.3 will be supported until 2027, it is generally advisable to update when possible. So updating it was.

Importantly, PHP at museum-digital is run via PHP-FPM. Before the update, we had one socket running per subdomain. This means, that if a PHP process serving the frontend stopped working for any reason, users in musdb were impacted as well.

Upgrading PHP to version 8.5

Once we upgraded the PHP version to 8.5 on Thursday, the same problems we faced with PHP 8.4 appeared again. The server would run rather smoothly for some hours, then more and more PHP processes would die and PHP-FPM would fall over for a given subdomain, and users would get a 504 gateway timeout error. Again, the IIIF API and image manipulation were the main causes of PHP-FPM getting stuck. Of course, the number of AI scrappers continuing to use the site did not help.

PHP-FPM settings

A natural first point to consider was the configuration of PHP-FPM. PHP-FPM knows three basic modes for running an application:

ondemand You define a maximum number of processes the application may use. When a new request is made, idle processes get used. If there is no idle process, PHP-FPM starts a new one. After a specified number of requests or a given number of seconds, an old process is closed. This is primarily aimed at being able to scale way down - if there is no requests, there will be no processes (which is to say, less resources used). On the other hand, starting new processes takes time.
static You define a number of processes that should always be running for the application. This means that there should always be processes already started and ready for usage, but it also means that those processes take up resources even when they are little used. Which is to say, this is useful if one has a high and constant stream of users.
dynamic You define a maximum number of processes, as well as how many processes should be always running for immediate use, and a (minimum and maximum) number of spare processes to keep running. PHP-FPM then manages if more processes should be started or if one of the already running ones shall be used. This, in theory, is useful if one wants to reliably and quickly serve users, expects some use all the time, but wants the server to dynamically scale up and down as needed.

With museum-digital spread out over around 80 subdomains, we had thus far used the ondemand mode for most subdomains. Only the largest and most used instances / subdomains of museum-digital were run using dynamic mode. With the update to PHP 8.4 and then 8.5, the behavior of the ondemand mode seems to have changed. If one process dies, the whole subdomain goes seems to go down with it (I have not found a documentation on this, but it’s evident from the last two weeks).

We hence moved critical subdomains impacted by the errors (which is to say, any “regular” instance of museum-digital) to dynamic mode. As dynamic mode enforces stricter limits on how many processes can be run respective to the available hardware (which is to say, dynamic mode requires a better-written configuration), this also meant that we needed to adjust the specified numbers of processes per subdomain according to their use.

To actually grasp real use of a subdomain including bots, we turned to the logs we keep for about a week (and then rotate out). In server logs, usually one line corresponds to a single request. With a small script, we loop all the different subdomains and check how many requests were made. To be really sure that only requests to relevant PHP scripts are processed, we filter them by the presence of the substring “php” before counting. The result for today between 1 a.m. and 4 p.m. looks as follows:

| Requests count in instance                         |      Total |      musdb |        PDF | 
| -----                                              |      ----- |      ----- |      ----- | 
| agrargeschichte.museum-digital.de                  |     341508 |       1245 |        719 | 
| bawue.museum-digital.de                            |     454228 |      12559 |       6819 | 
| bayern.museum-digital.de                           |     176291 |          0 |        158 | 
| berlin.museum-digital.de                           |     223280 |      14917 |       6814 | 
| brandenburg.museum-digital.de                      |      63286 |       6927 |       3873 | 
| bremen.museum-digital.de                           |     221208 |          0 |       2026 | 
| bund.museum-digital.de                             |        261 |        167 |          5 | 
| collectors.museum-digital.de                       |     108398 |        449 |        648 | 
| hamburg.museum-digital.de                          |      35489 |          0 |         11 | 
| hessen.museum-digital.de                           |      50932 |       7962 |       2486 | 
| meckpomm.museum-digital.de                         |      94177 |         11 |        139 | 
| nds.museum-digital.de                              |     137703 |       4105 |       4134 | 
| owl.museum-digital.de                              |     427667 |       1258 |       2412 | 
| rheinland.museum-digital.de                        |      64838 |       1753 |       1276 | 
| rlp.museum-digital.de                              |     207944 |       7405 |       7532 | 
| sachsen.museum-digital.de                          |     120931 |      16117 |       6034 | 
| saarland.museum-digital.de                         |        210 |          0 |          1 | 
| smb.museum-digital.de                              |     228542 |          0 |      11517 | 
| sh.museum-digital.de                               |      21098 |          0 |         48 | 
| st.museum-digital.de                               |     317913 |       6243 |       6217 | 
| thue.museum-digital.de                             |     117893 |          0 |        495 | 
| westfalen.museum-digital.de                        |     101584 |       2033 |       3310 | 
| br.museum-digital.org                              |      43413 |          0 |         16 | 
| jateng.id.museum-digital.org                       |        211 |          0 |          0 | 
| jatim.id.museum-digital.org                        |      23410 |          0 |        159 | 
| lazio.it.museum-digital.org                        |        295 |          0 |          0 | 
| ma.pl.museum-digital.org                           |        385 |          0 |          0 | 
| noe.at.museum-digital.org                          |     906386 |          0 |        369 | 
| tirol.at.museum-digital.org                        |        537 |          0 |          7 | 
| vbg.at.museum-digital.org                          |         96 |          0 |          0 | 
| wien.at.museum-digital.org                         |     472305 |        586 |       3243 | 
| ulster.ie.museum-digital.org                       |      28869 |          0 |          2 | 
| connacht.ie.museum-digital.org                     |        392 |          0 |          0 | 
| va.srb.museum-digital.org                          |       5599 |          0 |         22 | 
| ko.rou.museum-digital.org                          |       9036 |        635 |        567 | 
| mm.rou.museum-digital.org                          |        235 |          0 |          0 | 
| ca.usa.museum-digital.org                          |       3946 |          0 |          0 | 
| ma.usa.museum-digital.org                          |        357 |          0 |          0 | 
| ny.usa.museum-digital.org                          |      19576 |          0 |        294 | 
| syddanmark.dk.museum-digital.org                   |        675 |          0 |          9 | 
| de.pt.museum-digital.org                           |       1241 |          0 |         29 | 
| zh.ch.museum-digital.org                           |     233280 |        512 |        650 | 
| ba.hu.museum-digital.org                           |      99927 |       1901 |         72 | 
| be.hu.museum-digital.org                           |     100830 |        244 |       3005 | 
| bk.hu.museum-digital.org                           |     489446 |         55 |       3985 | 
| bu.hu.museum-digital.org                           |     213616 |       6206 |       5753 | 
| bz.hu.museum-digital.org                           |     598550 |        680 |       1788 | 
| cs.hu.museum-digital.org                           |      88585 |          0 |       1054 | 
| fe.hu.museum-digital.org                           |     199812 |          7 |        215 | 
| gs.hu.museum-digital.org                           |     216680 |       4215 |        912 | 
| hb.hu.museum-digital.org                           |      61250 |          0 |         65 | 
| he.hu.museum-digital.org                           |      26312 |          7 |         26 | 
| jn.hu.museum-digital.org                           |      11970 |          0 |        131 | 
| ke.hu.museum-digital.org                           |     370219 |       2959 |       1680 | 
| no.hu.museum-digital.org                           |     119487 |          0 |       1545 | 
| pe.hu.museum-digital.org                           |     603846 |       2957 |       1446 | 
| so.hu.museum-digital.org                           |     308116 |       6151 |       6698 | 
| sz.hu.museum-digital.org                           |        116 |          0 |          0 | 
| to.hu.museum-digital.org                           |      52406 |          0 |       1229 | 
| va.hu.museum-digital.org                           |     184231 |       2839 |       1666 | 
| ve.hu.museum-digital.org                           |    1015509 |       3672 |        296 | 
| za.hu.museum-digital.org                           |        199 |          0 |          6 | 
| ce.cz.museum-digital.org                           |          3 |          0 |          0 | 
| ccc.cz.museum-digital.org                          |         17 |          0 |          0 | 
| academia.hu.museum-digital.org                     |       9158 |          0 |         13 | 
| cherkasy.ua.museum-digital.org                     |      25567 |          0 |         26 | 
| chernihiv.ua.museum-digital.org                    |       3258 |         99 |        156 | 
| dnipro.ua.museum-digital.org                       |      26725 |          0 |        109 | 
| donetsk.ua.museum-digital.org                      |         17 |          0 |          0 | 
| ivfr.ua.museum-digital.org                         |        722 |          0 |          9 | 
| kharkiv.ua.museum-digital.org                      |      12932 |          0 |         39 | 
| kyiv.ua.museum-digital.org                         |     436482 |       5967 |       1351 | 
| kyivska.ua.museum-digital.org                      |       2159 |          0 |         79 | 
| lviv.ua.museum-digital.org                         |     163358 |        188 |        274 | 
| poltava.ua.museum-digital.org                      |       7657 |        284 |          3 | 
| odesa.ua.museum-digital.org                        |         93 |          0 |          1 | 
| rivne.ua.museum-digital.org                        |      59510 |         65 |        156 | 
| sumy.ua.museum-digital.org                         |      35890 |        303 |          3 | 
| ternopil.ua.museum-digital.org                     |     150700 |         37 |        184 | 
| zhytomyr.ua.museum-digital.org                     |          3 |          0 |          0 | 
| vinnytsia.ua.museum-digital.org                    |      14229 |          0 |          0 | 
| volyn.ua.museum-digital.org                        |      16705 |          0 |        485 | 
| zakarpattia.ua.museum-digital.org                  |       2865 |          0 |         30 | 
| zaporizhzhia.ua.museum-digital.org                 |      24348 |        338 |         56 | 
| scotland.museum-digital.org                        |          0 |          0 |          0 | 
| md.museum-digital.org                              |          0 |          0 |          0 | 
| demo.museum-digital.org                            |         12 |          2 |          0 | 
| goethehaus.museum-digital.de                       |     260072 |          0 |         85 | 
| lmw.museum-digital.de                              |     326724 |          0 |         65 | 
| gedenkstaetten.museum-digital.de                   |       3474 |          0 |          0 | 
| turcica.museum-digital.de                          |      75533 |          0 |          1 | 
| nat.museum-digital.de                              |    1238860 |          0 |       4657 | 
| at.museum-digital.org                              |     631578 |          0 |         89 | 
| cz.museum-digital.org                              |          2 |          0 |          0 | 
| dk.museum-digital.org                              |       5415 |          0 |          4 | 
| hu.museum-digital.org                              |     359619 |          0 |       2827 | 
| id.museum-digital.org                              |       8030 |          0 |          0 | 
| ie.museum-digital.org                              |       2073 |          0 |          0 | 
| it.museum-digital.org                              |         78 |          0 |          0 | 
| rou.museum-digital.org                             |       8277 |          0 |        466 | 
| pl.museum-digital.org                              |        142 |          0 |          0 | 
| pt.museum-digital.org                              |          0 |          0 |          0 | 
| srb.museum-digital.org                             |        565 |          0 |          0 | 
| ua.museum-digital.org                              |     232115 |          0 |        805 | 
| usa.museum-digital.org                             |       3752 |          0 |         34 | 
| ch.museum-digital.org                              |      53417 |          0 |          1 | 
| global.museum-digital.org                          |     727690 |          0 |       2199 |

Note that the number of requests obviously is also impacted by bots changing attention - once a scraper is done with one subdomain, they turn to the next. The elevated number of requests in ve.hu.museum-digital.org is normal, but still starkly exaggerated when compared to other days. The Germany-wide instance is persistently the most frequented one, usually the global one is second at around 80% of requests.

Now equipped with actual numbers, we could scale the PHP-FPM to a much more suitable configuration than before (we had thus far never bothered counting actual requests, instead relying on the number of objects).

A second step in the PHP-FPM configuration was to reduce the impact the problems had. Previously there was one shared configuration and socket per subdomain. On the one hand, this meant that stuck processes in the frontend impacted users in musdb (and vice-versa). On the other hand, some constraints on resource usage cannot be set on a per-directory level but must be set per PHP-FPM socket / server (see the PHP documentation on user.ini and the list of php.ini directives). As the frontend and musdb have different requirements (frontend: low maximum memory use, short timeouts, no file uploads, generally strict settings; musdb: long timeouts for uploads, generally more lenient), being able to configure them independent of each other is useful in general.

We thus separated the configuration for the frontend, musdb, and PDF generation in the frontend; providing dedicated sockets for each. The frontend has a reduced priority on the system overall, strict constraints on how it may be used, etc. The settings are stricter than they were before. musdb has an elevated priority and more lenient settings (file uploads, longer timeouts), in fact more lenient than before. Finally, PDF generation is a special case as it offers no real benefit over the browser’s print tool (see MDN on print CSS), while being resource-intensive. As such, it has a far reduced priority and very strict settings.

With the separated configuration and sockets, we can now better tailor the configuration to each application’s needs and have the added benefit of problems in one application not impacting the other.

Code

As we had already prepared the codebase for PHP 8.4 awaiting an eventual upgrade, the upgrade to PHP 8.5 only required minimal changes. Aside from the deprecation of the functions finfo_close() and curl_close(), references to which were accordingly removed from the code, the update necessitated no further work.

Scaling in Software

Improving the PHP configuration was not enough to fix the issues, especially with the now increased number of requests from bots. To get some breathing room, we adjusted the most resource-intensive pages.

Frontend

In the frontend these are, again, the IIIF API, PDF generation, and timelines. Finally, we made changes to the pages for failed searches to better handle high load situations.

Image pages

The IIIF API was used for the main image pages in the frontend. We used (and use) Mirador as a IIIF viewer. Simply opening an image page thus meant three requests to fetch different regions of an image. Zooming into the image triggered further requests to fetch the relevant parts of the image. Cropping the image to the requested region with IIIF happens on the server (which is no problem if there are few users, but is turning into a problem when you have hundreds of requests per second).

We thus changed the default of image pages: The new default image page is the old, non-IIIF one. As features like zooming into images, that Mirador comes with, are popular and useful and the old image page did not support those, we worked to improve the page. To do so, we rely on OpenLayers, a library we already use for maps. Besides including maps from tile servers, OpenLayers also supports loading simple image files - which we do here. The image is hence loaded once in full size and zooming etc. happen entirely in the browser.

Taking the opportunity, we improved the page overall. An often noticed problem of image pages thus far was, that users who opened image pages coming from external services (think Google Images) had problems identifying that the image was an object image and that there is further object data to be found on object pages. The updated image pages now come with a header stating reflecting the name of the image, the name of the object and the name of the institution. Note that many images do not feature a dedicated title, musdb uses the object name as a default image title in that case, which is why the object title will often appear twice in the header. Maybe this can be used as an encouragement for the colleagues working in musdb to more consistently set expressive image titles in the future.

Also new is a mini map at the bottom left, displaying where in the wider context of the image one has currently zoomed in, as well as the ability to link exactly the region one has currently zoomed into. To enable the latter, the URL updates as one zooms or navigates around the image. Somebody else opening the same URL will then open exactly the same image region the linking person was viewing when copying the URL. Finally, we finally set specific Content Security Policies relevant to the currently opened media. If the displayed media entry is an internally stored image, no external images need to be allowed to load. If the displayed media entry is an audio file stored on archive.org, archive.org needs to be whitelisted as a source for audio files - but only archive.org and no other page. Previously, embedding images from anywhere on the net was allowed, increasing the potential damage a potential attacker may cause.

Making the use of Mirador a secondary, non-default option reduced the need for server-side image manipulation and the corresponding resource use significantly. The IIIF remains largely unchanged, but its use must now be requested explicitly.

PDF generation

As stated above, PDF generation brings little advantages to the browser’s print functionality in combination with object pages. On the contrary, the PDFs generated using the frontend’s templates feature less information. But they come with the file ending “.pdf” and seem to be extremely popular with bots. On the other hand, PDF generation means, among others, loading whatever images are to be embedded into the PDF and manipulating them fit into the PDF. The resulting files are significantly larger than the corresponding HTML files and thus also use more of the available bandwidth.

The update to handle PDF generation respective to resource usage was already introduced in the last months: publicly linked PDFs are now only generated if overall load on the server is low, if a user has set their browser language to any language different from a museum-digital instance’s default language. As most scrapers do not bother to change their browser language (which means they come with either none, English or Chinese), this means they will mostly be unable to trigger the generation of PDFs. They see an error page instead.

Failed Search Pages

If a user tries to execute a search query without any results, they will get suggestions for similar search terms - similar to how Google will ask one searching for “Berrlin”, if they meant “Berlin”. Trying to identify suitable suggestions obviously costs resources and whether the suggestions are actually what a user wanted is by nature hit or miss - it’s suggestions after all. In the case of scrapers, suggesting alternative search queries offers them a never-ending stream of possible search queries to run and keep scraping the subdomain with - to nobody’s benefit (not even the scrapers’, as they likely got the same content with other search queries already).

We thus now use the same function used to identify whether PDFs should be generated for a user to check if search suggestions should be provided. It a user comes with a non-default browser language and resource use is high, no suggestions will be provided.

Timelines

Timeline pages as implemented in museum-digital’s frontend offer another source of endless links and search queries, as they link to further and further specifications of the time searched by. Again, an improvement already introduced months ago, was to better parse queries by time: If a user searches for objects that are linked to times “after 1920” and “after 1930”, the latter already includes the former. “After 1920 and after 1930” means exactly the same as “after 1930”. Which is one join instead of two - half the resource usage.

A minor improvement we noticed on the side was impact of automatic redirects in the timelines. Say, a user searches objects by their link to a given tag and then generates a timeline for said objects. If all objects were created in the 20th century, the timeline will automatically redirect so as to “zoom” into a more appropriate time scale than from the big bang to now. Until the last weekend, script execution was not stopped when that redirect happened - which means that all database queries for time time from the big bang to now were still executed even though the user never got to see them. That is now fixed.

The Anti-Climactical Solution

All of those changes got the frontend more or less stable. Problems with uploading images remained however. Finally, the only thing that helped was uninstalling libvips (which we use for image manipulation) and reinstalling it. That seems to have fixed the issues.

Especially as the number of requests from scrapers continues to increase, the current strategy outlined above seems to be fruitful. By reducing the use (and sometimes the availability altogether) of especially resource-intensive and - depending on the context - little useful functionalities, much stability and can be gained.

The update seems to finally be largely completed (aside from maybe some further fine-tuning of the PHP-FPM configuration) and museum-digital is stable despite the bot problem, while we haven’t had to take more drastic or costly actions yet - such as blocking or adding additional servers.

info

AI Use

(Post image AI-generated using Z-Image)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Development, November 2025

2025-12-03T00:00:00+00:00

Frontend

On source / reference pages, linked objects are now sorted by the position within the source work on which they are referenced or which they do themselves reference
The target URL of the regular / unspecified search bar for objects now follows the new, prettier URL schema
Support for an OAI-PMH API for object metadata
- Standardized endpoint for aggregators seeking to retrieve data in batch
- Metadata formats thus far supported:
  - LIDO
  - OAI-DC (mandatory)
- See also: Blog
PDFs are only generated for users with a browser set to a non-default language if load on the server is low
- The resource use caused by AI bots scraping museum-digital has been growing and growing. Generally, we see bots included in our mission to enable access to cultural heritage. On the other hand, nobody is served if the service is bogged down by bots. One functionality that is commonly used among bots and resource intensive is the generation of PDFs for object pages. The same information can be loaded from the object page itself and printed to a PDF using the browser’s print option. There are thus rather few downsides to limiting access to PDF generation to timmes, when server load is low. So that’s what we did.
Collection-specific ISIL identifiers are now also used in the LIDO API
Alternative numbers of an object can now be displayed on object pages
- This includes tooltips for types of alternative numbers, that can be set by the museum on the institution-wide settings pages of musdb

musdb

Search for objects
- Type-ahead search for languages (of the object’s content)
- Search by object’s revision status (open, read-ony, archived, etc.)
Batch editing of objects’ revision status
Parameters of the full text search index were updated to improve the search of word compounds in German

Importer

Core

The dry-run mode now does not abort an import anymore, if an unmapped value is encountered. Unmapped entries are collected and displayed together afterwards.
- This means, that unmapped entries can now much more easily be copied to and mapped in concordance.museum-digital.org
Support for the import of alternative numbers (of objects)
Support for the import of space hierarchies

Parser

AdlibXml
- Support for importing objects’ alternative numbers
CsvXml
- Support for importing objects’ alternative numbers
CsvLocations
- New parser for csv-based imports of space hierarchies
ImageByInvno
- New setting: append_chars (Adds suffixes, that exist in the inventory number, but not in file names)

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Dev, August 2025

2025-11-25T00:00:00+00:00

The last months have been busy, off and on museum-digital. This is the first of three posts today on recent technical developments around museum-digital to continue the regular state of dev posts.

Development

musdb

Added a tool for the AI-aided detection of displayed subjects in images for tagging
- Has to be explicitly turned on on a per-collection basis, as it makes sense only for certain types of objects (like paintings, drawings, photographs)
- Usable within the tagging overlay on object editing pages
- See also: blog post about the feature (German)
The maximum length of contents in the data field “edition” of literature entries has been extended to 50 characters
Uploaded PDFs may now be up to 40 MB large
New command line option to reset all permissions of a user account to the default provided by their user role
New event type: “Changed”

nodac

Added AI-generated suggestions for tag definitions and translations of tag names in a sidebar
- This is also re-used to identify duplicate tags

The right sidebar of tag editing pages in nodac now features AI-generated tag descriptions as well as possible translations for the title. These are also used to identify possible duplicates of a tag entry (top right, underlined in purple).

Dissemination

Poster presented at CoRDI 2025 (Archived) on August 27, 2025, in Aachen: “To Educate or to Enforce - The Case for Underhanded Methods to Improve Research Data”
- PDF
- Abstract / Zenodo

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Dev, September 2025

2025-11-25T00:00:00+00:00

Development

Frontend

Objects that are linked to a source as referencing the source or being referenced in it are now listed on the source’s page
- Example: “Novalis Schriften. Die Werke Friedrich von Hardenbergs. Historisch-kritische Ausgabe. Erster Band: Das dichterische Werk. 3. Auflage”
The object page now displays the new transcription fields for notes, status, and type of the transcription
New types of classifications for the link between an object and a tag
- Taxon
- Topic
- Mentioned subject (similar to the existing “display subject”)
Dependencies
- Updated OpenLayers to Version 10.6

Source pages in the public frontend / portals of museum-digital now display all objects linked to the source as either referencing the source or being referenced by their source.

musdb

Renaming vocabulary entries to blacklisted terms is now impossible
- Someone managed to create an actor “unknown” by adding a different actor and renaming the new actor entry to “unknown” afterwards.
Added a sidebar to object group overview pages for additional filtering options
New types of classifications for the link between an object and a tag
- Taxon
- Topic
- Mentioned subject (similar to the existing “display subject”)
New APIs for listing all vocabulary entries linked to the objects of a museum
Dependencies
- Updated OpenLayers to Version 10.6

Importer

Core
- Deaccessions are now covered by the import tool
- Recipients of deaccessions are linked to the address book / contact list
Parser
- CSVXML: The CSVXML parser now covers deaccessions as well
- ImageByInvno: Added an option to ignore any characters before a given starting combination of characters (if there is a consistent start of the inventory number).

CSVXML

New fields
- tag_related_identifier_type
- tag_related_identifier

Dissemination

Presentation “Von museum-digital zum eigenen Online-Katalog” for the Working Group Digitization of the State Museum Association of Rhineland-Palatine
- Slides: PDF
- Slides: ODP

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Development, October 2025

2025-11-25T00:00:00+00:00

Development

Frontend

Significantly reworked the display of transcriptions on object pages
- Titles of transcriptions are now displayed
  - If none is set, the type of the transcription (original or translation) is used as a replacement
- Transcriptions are sorted by their titles
- Improved the display of transcriptions in tiles
  - Problems with vertical scrolling are now solved
  - If only one transcription has been recorded, it will be displayed on the full width of the page
  - If there are more than two transcriptions for an object, they are folded in by default and can be unfolded later on
Batch export of object metadata via the API
- Thus far available in JSON & LIDO
- API documentation
- See also
Dots as a separator in floating point numbers for object measurements are replaced with a comma in languages that require that
Collection-specific ISIL IDs are used in the LIDO API

musdb

Added a field for recording titles / names of transcriptions
Added the option to set collection-specific ISIL IDs
Setting object type tags via the improvement suggestions now correctly classifies the thus created link between object and tag
Additional shapes are now available
- E.g.: round, square
Object groups can now be filtered by whether they have a superordinate one or not

Dissemination

2025-10-08: Presentation at the Autumn Conference of the Working Group Documentation of the German Museum Association: “Interoperabilität schaffen - Geschichten aus 1001 Importen”
- PDF
- ODP
2025-10-14: Talk on a workshop of the project CiVers (Citation of Versioned Web Pages by PID)
- PDF
- ODP
2025-10-17: museum-digital Usertagung 2025

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Making Interoperability Easy

2025-11-24T00:00:00+00:00

Interoperability has been one of the focal issues around museum-digital practically since its inception. Offering different, simple ways to bring data into the system was a necessary requirement to even think of what we do. And offering simple ways to get the data out of the system again is just good practice - though all too often neglected.

To that end, there have traditionally been two primary ways for data retrieval. In musdb, one could run batch exports and receive a ZIP with some form of XML files. One per object, with the objects matching the results of any given object search.

On the other hand, there is the public API. Using URL manipulation, one can access the (primary) contents of each page in a machine-readable way. To access the JSON representation of an object’s published metadata, where the object’s ID is 7141 in the Hesse instance of museum-digital (URL: https://hessen.museum-digital.de/object/7141), one simply has to insert json to the path: <a href="https://hessen.museum-digital.de/json/object/7141">https://hessen.museum-digital.de/json/object/7141</a>.

Next to the default JSON output, additional APIs are offered wherever suitable for a given data type. For objects, the primary additional output method is a LIDO API.

Thus far, the main limitation of the public API was that it only allowed one object (or institution, collection, etc.) to be queried at a time.

Querying Object Metadata in Batches

After a significant refactoring of the code to load object data for object pages - primarily to improve caching and allow for parallelized requests to the database - we are now finally able to offer APIs for querying object metadata in batch. Thanks to grouped database queries, performance and resource usage scale nicely. Taking simply the currently most recent objects in the Germany-wide instance of museum-digital: Loading all object data of one object and presenting it in JSON takes 0.0087 seconds and loading and generating the JSON for the 100 most recent objects takes 0.197 seconds (or 0.00197 per object). Note that not all queries for all aspects of an object’s metadata are grouped yet, performance may thus get even better over time. This does also not yet account for the overhead of the many HTTPs requests one would previously need to execute to get each object’s metadata one by one - real performance improvements are thus even greater.

Now, how to access object metadata in batches?

The batch access is linked with the search API and reuses its main query parameter (“s”). Say, if one is searching for objects related to Berlin (a.k.a. the place of the ID 61), the URL of the respective search page would be https://global.museum-digital.org/objects?s=place%3A61. The corresponding API for retrieving all of the objects’ published metadata would then be https://global.museum-digital.org/export/json/place:61?limit=24&offset=0. Like the search page itself and its primary API (/json/objects), the full batch export API is paginated with a maximum of currently 100 objects per page being returned.

Additional to URL manipulation, the batch export API is linked in the menu of object search results pages.

Currently, the batch export of full object metadata is available for JSON and LIDO (XML) representations of the object data. More can rather easily be added later, should a demand arise.

OAI

Implementing a performant way to export full object metadata in bulk was one of the two main missing components for the long-missing implementation of an OAI-PMH API.

OAI is a standard tailored towards data harvesting. Say, an external service like the German Digital Library or Worldcat wants to do something with external data from diverse sources, e.g. to also display objects or implement a search across the different collections / libraries to find which one has which object / book. To be able to do so, they need to be able to access the respective data in some way. Ideally, using a common standard that describes how to query data, helps to identify any data sets that need to be updated or added (or deleted), and finally presents a uniform way to access the data periodically. That, exactly, is OAI-PMH.

In a nutshell: OAI-PMH allows other services to copy all (published) data from another service in a maschine-readable way and can thus significantly improve reuse in aggregation. Of course this only applies to technical questions; legally, potential re-users need to comply with the metadata license applied by the initial data provider regardless of the (technical) means of access.

Since last week, museum-digital now provides a OAI-PMH API at /oai respective to a given subdomain. E.g.: https://hessen.museum-digital.de/oai. As of now, the OAI-PMH API provides access to the objects’ metadata using LIDO (XML) and the mandatory OAI-DC format.

Note that there are some caveats remaining for now: First, the LIDO representation of object metadata is not (and can by definition not be) as complete and fine-grained as the JSON API. It is also not exactly similar to the LIDO as returned by exports from musdb (one is formed natively in PHP, the other using XSLT, leading to divergent development paths). Also, the LIDO output lists different identifiers from the ones used by the OAI-PMH API and the OAI-DC representations otherwise.

Finally, the OAI-PMH API at museum-digital does not implement OAI-PMH data sets to group collections. Instead, it follows the existing search logic (essentially providing a new endpoint per query). Example searches via OAI might thus look as follows:

All objects from the Agrargeschichte instance of museum-digital, represented in OAI-DC: https://agrargeschichte.museum-digital.de/oai?verb=ListRecords&metadataPrefix=oai_dc
All objects linked to Berlin (place #61), in the Berlin instance of museum-digital, represented in LIDO: https://berlin.museum-digital.de/oai/place:61?verb=ListRecords&metadataPrefix=lido
All objects of the Freies Deutsches Hochstift, Frankfurt am Main, represented in LIDO: https://hessen.museum-digital.de/oai/institution:1?verb=ListRecords&metadataPrefix=lido
All objects from Baranya with only their identifiers: https://ba.hu.museum-digital.org/oai?verb=ListIdentifiers&metadataPrefix=lido

Credits

Credits where credit is due: the Städel Museum deserves praise for their implementation of OAI-PMH. For me personally, seeing the Städel’s API was the first time I saw a visibly stateless OAI-PMH implementation, enabled by the ingenious idea of using machine-readable, JSON-encoded resumption tokens over UIDs that have to be resolved on the server side. I spent weeks (or months?) making museum-digital’s frontend stateless. By following the Städel’s example, it can remain so even while offering an OAI-PMH API.

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Dev, June & July 2025

2025-08-23T00:00:00+00:00

June and especially July were at first glance once again rather slow months in terms of development at museum-digital.

Generally, the pace and type of development seems to have changed this year. Rather than doing many small improvements all over the place, there is less but larger and more labor intensive changes and new features. See for example the versioning in musdb (January), the tool for automating imports based on what others call a “hot folder” and the sort option to sort objects by their images’ aesthetics score (both presented here in March), and the new tool for suggesting formulations for object descriptions using large language models (June).

July

Frontend

Translation of the software to Hindi and Telugu
Grouping of tags by their relation to a given object
If an object is linked to more than 10 tags, the tag list of object pages quickly starts looking unorganized and messy. In such cases, the tags will now be displayed grouped by their relationship to the object (thus far: general tag, material, technique, object type, display subject)

musdb

New Features

Export option for the specific LIDO as expected by the German Digital Library’s “colonial contexts” portal

Improvements & Changes

The minimum length of fulltext search terms for object search parameters is now visibly enforced in the extended search user interface
To not overly burden the fulltext search server, any term in a full text search in musdb needs to be a minimum of two characters long. Thus far, shorter full text search parameters were simply ignored. This certainly was confusing at times. Since June, attempting to perform search queries with shorter search parameters is made impossible by a check in the extended search overlay.
Reception history of objects: Statements of the relevant position within a source can now be 40 characters long
Transcriptions
- May now be up to 4000000 long
- New fields: Notes on the transcription, status, aims

Bugfixes

Fixed a bug in the batch editing of specific visibility flags for data fields on the addendum tab

Juni

Frontend

Performance improvements
- Object search now runs without a connection to the full text search server if no full text search parameter is set
- If multiple search parameters for an earliest / latest time have been set, they are parsed and combined before being forwarded to the database
Improvements in the deletion of temporarily created PDF files (PDF export)
Navigation has been translated to Tamil

musdb

New Features

Recipient of deaccessed objects can now be linked from within the address book
New tool for AI-aided formulation o object descriptions (based on existing other metadata)

Improvements

Object search now runs without a connection to the full text search server if no full text search parameter is set

Importer

The CSVXML parser has been extended to cover new event types and markings
Object groups automatically generated to group all objects of an import can now receive a description as set within the import configuration

nodac

The list of selectable languages for the navigation of nodac has now been restricted to those in which there is actually a complete translation

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Hindi

2025-07-13T00:00:00+00:00

After Kannada, Tamil and Telugu, museum-digital can now also be browsed in Hindi. As always, thanks and enjoy!

info

Image credits

“Devanagari_letters.svg” is licensed under Creative Commons Attribution-Share Alike 4.0 International, by बडा काजी, and retrieved via Wikipedia Commons.

info

License

This content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license..

State of Dev, May 2025

2025-07-07T00:00:00+00:00

Frontend

New Features

Tabs are available in the configuration when the site is installed as a Progressive Web App (PWA) (See also)

Improvements

Duplicate search query parameters for objects associated with times before / after are removed.
Example: If one searches for “Objects after 1900, whose first recorded associated time is also after 2000”, this is a duplicate query. 2000 is always after 1900, so one of the two parameters can be removed.
Searches by time begin / end are also the foundation for the timeline. As the timeline is both beloved by web crawlers and resource-intensive to generate, this change significantly reduced server load.
Any links on timeline pages that do not reference single object pages are marked as rel=nofollow
Which is to say that bots are told to ignore them.

musdb

New Features

Tabs are available in the configuration when musdb is installed as a Progressive Web App (PWA) (See also)
New search option for objects: “Can be published”
Aggregate search for objects that have not yet been published and which comply with the minimum requirements for publication.
Separated measurements can now be batch-edited
Available via “assign results”
Institution-wide setting to enforce the use of the user-defined object editing interface for all users of the institution
Exhibitions can now be searched via the API (/exhibition/list)
A user’s user-agent is checked whenever a page is requested. If it changed, the user will be logged out automatically.
Protection against session hijacking.

Improvements

Panorama images for tours through exhibitions / institutions are calculated down to 2400 px height rather than 1400 px height
The APIs for searching for entries in controlled vocabularies are now available via the main API
See e.g. <a href="https://de.handbook.museum-digital.info/musdb/API/index.html#/actor/actorSearchLinkedToObjects">/actor/search_linked_to_objects/{term}</a> , <a href="https://de.handbook.museum-digital.info/musdb/API/index.html#/actor/actorSearch">/actor/search/{term}</a> etc.

Bugfixes

Fix: “Visiting scientists” couldn’t open the “location” tab (this erroneously required museum-wide editing permissions)
Fix: User-defined defaults for descriptions of new objects were ignored when new objects were to be added
Fix: Thumbnails were displayed as duplicate images linked to exhibitions (tag “images”)
Fix: Prefixing via the batch editing was broken

Importer

New Features

New parser for CSV exports / imports from Robotron Daphne

Improvements

CSVXML Parser
- New literature-related fields (ISSN, editor, etc.) are now covered
- References to wikidata are now respected as tags are imported
The maximum length of any single given tag/keyword is centrally set to 95 characters

Bugfixes

Some more recently added fields from the “admininstration” tag of musdb’s object pages had been read by the importer but their transfer into the database had not been implemented thus far.

nodac

New Features

Tabs are available in the configuration when nodac is installed as a Progressive Web App (PWA) (See also)

CSVXML

Improvements

Added new literature fields: type, editor, edition / issue, ISSN

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Browse museum-digital in: Telugu

2025-07-02T00:00:00+00:00

After we were gladly able to release the Tamil translation of museum-digital’s public portals last week, we are equally glad to do so with the all-new Telugu translation.

As always, many thanks to everyone who’s made that possible!

info

Image credits

“Telugu consonants.svg” von Psiĥedelisto, lizensiert unter CC BY-SA 4.0, via Wikimedia Commons

info

License

This content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license..

Browse museum-digital in: Tamil

2025-06-23T00:00:00+00:00

Today, we’re happy to release the Tamil translation of the public frontend of museum-digital. Thanks to everybody who’s made that possible!

As always, users with browsers set to using Tamil as their main language will now automatically see the Tamil translation of the site navigation etc. Others can select to navigate the site in Tamil using the language selector at the top right.

info

Image Credits

“Tamil Community - Worldwide.PNG” von Coppercholride, via Wikimedia Commons, licensed under Creative Commons Attribution-ShareAlike 3.0 Unported.

info

License

This content is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported.

State of Dev, March & April 2025

2025-06-08T00:00:00+00:00

Frontend

A Kannada translation of the software is now available
Museums can now select to display their own citation notes in the menu for such on object pages
This is especially relevant in case the object itself is to be cited (rather than its record online).
“Or” search queries can be combined within one search parameter for select attribute search types, e.g. places and tags. The Syntax is as follows: place:61~1
- For now, this can only be used using the internal query language. As such, there is no corresponding option in the UI for search settings. On the other hand, the search option is thus available using the API.
- This option is not available when searching for times or full events

musdb

Institution-wide settings
- Free text fields that double with similar controlled fields may now be hidden
  - The acquisition of an object may e.g. be directly recorded on the context of a given object or as a separate acquisition process. Recording it as an acquisition process is slightly more labor-intensive, but allows a more fine-grained and accurate documentation. With the new setting, the data fields for recording acquisitions directly in the object context can be hidden from all users of a museum to ensure a uniform use of the preferred functionality.
  - Institution-specific notes on how to cite objects may now be recorded for display on published object pages.
Search queries for inventory numbers are now character-level searches again (rather than following a fulltext search logic)
Some event types are incomplete, i.e. they cannot contain a place, or an actor, or a time. This incompleteness was handled differently between musdb, the import tool, and the CSVXML import preparation tool. Now, it is determined by a centralized list and thus similar across all of museum-digital.
Refactoring of the administrative command line interface
This mainly concerns auto-correction tools, but also results in that exports for the quick export option are now automatically generated daily.
Separated measurements are now positioned at the very top of the “addendum” tab of object editing pages.
It is now possible to record web links for object groups

Importer

The import tool can now be used as a harvester
- First use case is a harvester for LIDO records delivered via an OAI-PMH interface.
Externally stored images in formats other than JPG can now be imported
Significantly extended the LIDO parser to (among others) support the import of multilingual object data

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Bringing back character-driven search for inventory numbers in musdb

2025-03-29T00:00:00+00:00

If you search for “run”, you want to find entries (objects, blog posts, etc.), that mention “ran”. If you search for inventory numbers like “*1”, you want to find “0001”. These are fundamentally different categories of search. In the first case, you want to have a language-aware full-text search. In the latter case, you simply want to work with characters. In technical terms: Inventory numbers are strings (groups of signs or characters), but not common “text”.

When musdb and museum-digital’s frontend received their last large-scale update to their respective object search functions around 2021, enabling actually complex search requests across almost all of the fields and data types linkable to objects, this was - among others - made possible by our use of Manticore, a dedicated search server. Traditional relational databases excel at searches by indexes - pre-defined, known search parameters, that one prepares for searchability beforehand - while search servers like Manticore are reasonably good at that and excel at full-text searches.

In moving the search to Manticore, all searches in free text fields were defined as full-text searches. This was mostly the right decision: Full-text searches are the way to go with objects’ titles, descriptions, and the like. But two specific types of fields posed a challenge, because - as indicated above - they usually run on formalized strings that operate very differently from prose: objects’ locations and their inventory numbers. The software does not know about institution-specific and non-standardized rules of formalization, but users do. Hence, the preferred way for those specific types of fields is character-level searching.

For managing locations, we have in the meantime introduced spaces as a dedicated category capable of hierarchization as well as advanced features like the storage of sensor data. Object’s locations can now simply be expressed as a link to a space, which is by far the superior way when compared to the legacy free text field. If one does so, one can search for objects exactly in a given space, those that are located within it or its sub-spaces (e.g. a box in a depot room), etc. A migration tool from the legacy free-text field to the controlled spaces module is available through musdb’s dashboard. “Fixing” the issue of character-driven searches vs. full-text searches in locations is thus a least-priority issue - a better alternative is available anyway.

With inventory numbers on the other hand, there is no alternative to character-driven searching.

Laying the Foundations: From MSQL to Manticore and (Somewhat) Back

The basis for the expansion of search capabilities for objects was the introduction of a dedicated search server running Manticore. As the number of requests increased, this proved to be a blessing and - to some extent - a curse. Manticore offers more and better search options than a classic relational database, but it does not achieve the same level of stability. As long as queries remain index-bound and not concerned with text, the performance is roughly similar on our hardware: (both are about as quick, even with subqueries in MySQL; MySQL uses more resources, but is much more stable). If a query concerns a free-text field on the other hand, there is almost no comparison. Manticore offers a multitude of additional features at a great performance.

As stability had become an issue for a while, we adjusted the search to be able to use Manticore or MySQL as a backend, depending on which was more suitable in a specific context. In practice, this means that each search parameter is translated into a query string for Manticore and - if possible - for MySQL. If all search parameters have a MySQL equivalent, the search will be performed using the MySQL backend. Otherwise, Manticore will be used.

This simple way of negotiating which backend is more suitable works only as long as one of the alternatives (Manticore) supports all search options, while the other (MySQL) is preferable in a subset of the search contexts. Which is to say, character-driven searches in inventory numbers break the negotiation logic - they work somewhat well in MySQL, but do not work in Manticore.

Breaking the Logic / Mitigating Confusion

Up to this weekend, all search options were compatible with each other:

If one searches for all objects one has acc ess to, both Manticore and MySQL can handle the query. MySQL will be used.
If one searches for all “helmets” (tag) from “Europe” (place), both Manticore and MySQL can handle the query. MySQL will be used.
If one searches for “helmets” (full-text) from “Europe” (place), MySQL can only sufficiently handle the search by place, while Manticore can meet both search requirements. Manticore will be used.

Character-driven searches by inventory numbers break that compatibility. If one were to search for objects for “helmets” (full-text) with inventory numbers starting with 1 (“1*”), the search parameter “helmets” could only be satisfied by Manticore, while the character-driven search by inventory numbers can only be satisfied by MySQL. Which is to say, the combined search cannot be executed.

Due to popular demand, we introduced character-driven searches for inventory numbers back into musdb. As there is no way to sensibly combine all search parameters anymore, given our circumstance, we had to handle reduce the resulting confusion. For this, there are theoretically two ways.

The theoretically cleaner way would have been to disable the extension of search queries by full-text-focused parameters once an inventory number had been searched. As a full-text search by inventory number is theoretically still possible, the opposite direction (setting a full-text search first, then searching by inventory number) might still have been acceptable, as it would not have led to visibly different results. The basic idea of this solution would have been to prevent users from performing combined searches that are not possible in the targeted way. But if users actually managed in some way, the confusion would have been major. Worse yet, it would have been hard to explain - or rather, it would have been hard to find an appropriate spot in the UI for an explanation -, why certain search options are suddenly disabled.

The alternative route we chose is to allow users to do the impossible combinations, perform the search as best as we can (by transforming the character-driven search by inventory number into a full-text search), and aggressively warn about the likelihood of unexpected results when trying to perform such combined searches. This solution looks unpolished, but it is transparent about the imperfections of the software, and it allows users to find their own solutions to actually perform the combined searches they want. The simplest such solution would be to first search by inventory number, move all the objects into a watch list, and then search by the watch list and combine that search with the full-text search.

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Kannada

2025-03-25T00:00:00+00:00

There’s lots of talk about the FAIR principles - publishing findable and accessible data. Logically, to be findable and accessible, the data should at the very least be described in the language of users, even if it is not itself translated. And that means, that before object information becomes multilingual, platforms should become (more) multilingual and especially cover languages that may not be in the first line of languages covered by your next-door Large Language Model for translation.

Today we can gladly announce another large step for making museum-digital more findable and accessible: The frontend is now available in Kannada.

Users with browsers set to using Kannada as their main language will now automatically see the Kannada translation of the site navigation etc. Others can select to navigate the site in Kannada using the language selector at the top right.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Dev, February 2025

2025-03-25T00:00:00+00:00

In terms of development happening around museum-digital, February 2025 was a rather calm month. While more happened in the “machine room”, immediately visible changes are mostly restricted to bugfixes. And a whole new tool.

Here, as so often, in a list format:

Frontend

Bugfix: The detailed description was not reflected in the object API even if set to be so via musdb
Translation: The frontend is now available in Kannada

musdb

Bugfix: Fixed error in setting up new two factor authentication via TOTP (app-based 2fa)
Bugfix: Symbols for image rotation switched (counter-clockwise symbol rotated clockwise and vice-versa)
Feature: If the generation of a PDF catalogue is triggered via the sidebar of series editing pages, the objects will follow their order within the object group

Importer

If no explicit name/title for a resource (audio, video, PDF, 3D, externally hosted image) was supplied, there was no fallback. Now, the importer follows the same logic as musdb and will fall back to using the linked object’s object name as a resource title if none is supplied.
The CSVXML parser can now handle multiple objects per XML file.

Auto uploader

The auto uploader is an entirely new tool. When first started, users will be asked to enter the necessary data for running imports. And they are asked for the path to a folder which will subsequently be monitored for automatic uploads.

Whenever the tool then detects suitable contents for an import in the folder, the data will be uploaded and automatically imported. Intermediate steps like the generation of a settings file for the import are covered by the tool.

The tool can thus be useful for museums that regularly import data formed in a consistent format - say, museums using museum-digital for publishing while mainting a separate collection management system. It should be uninteresting to those who spend much effort on preparing imports (e.g. via CSVXML) or who want to migrate to musdb altogether, as it only becomes useful in the case of regular use.

The code is available here, licensed under GPL v3.
See also the more detailed German-language blog post about the auto uploader.

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Sort by Beauty

2025-03-06T00:00:00+00:00

Last month a new sort option appeared on museum-digital: “Aesthetics prediction”. Based on the LAION aesthetics predictor, each published object’s main thumbnail aesthetics are scored. The objects can then be sorted according to this score.

AI, Aesthetics and Discrimination

When working with AI it is common to criticize the inherent biases of the models used. Already underrepresented entities (people, viewpoints, etc.) are excluded, because they are not sufficiently represented in the data the model was trained on. In turn, AI repeats what it learned - pre-existing biases in society are replayed and reinforced by AI.

A second, well-founded criticism against AI applications is their impreciseness, their tendency to “hallucinate” entirely wrong results, and the lack of reproducibility of results.

Both criticisms are entirely valid. And both hint at why AI might be a useful tool exactly for generating a sort order by aesthetics. Like AI, aesthetics are imprecise: Ask 10 people to rank some pictures by their aesthetic appeal and you will get 10 different answers. But the general thrust of the answers will likely be more or less the same. There are societal rules - biases - for what pictures are more or less beautiful. But they are fuzzy and interpreted differently by each and every observer. A sense of aesthetics is about learning and reproducing those biases - learned by the inputs we learn in everyday life -, just as AI will be biased based on the data it is trained on.

Sorting then is, in essence, an exercise of discrimination. Sorting entries by ID / age in a database system will favor the newest entries and discriminate against older ones. Sorting entries alphabetically in an ascending order will favor entries whose title starts with “A” while it will discriminate against those whose title starts with “Z”. Sorting based on an AI-generated rank is discriminating. Sorting by beauty or aesthetics is as well. AI allows sorting by beauty.

What is discriminated against?

Now, if everything in sorting is discriminating, it is all the more important to consider what is actually evaluated. In other words, on what basis the discrimination takes place when ranking digitized museum objects by the beauty of their digital reproductions. As stated above, the objects are scored based on the aesthetics score established for their main thumbnails.

Images then have a range of aspects that influence their aesthetics. Image composition, lighting, contrast, the motive, and many more (an art historian could likely list hundreds). For the common critique, it is essentially only the motive that matters: Favoring images of white women over images of Asian men reproduces a range of societal problems that should not be reproduced. Transferred to object photography, the motive may be further differentiated between object type and the actual motive (e.g. of a painting). And such a discrimination is noticeable in the results: Paintings seem to be generally slightly better ranked than pictures of tools. A bias based on the displayed subjects of e.g. different paintings is not something anybody from the team would have noticed, but is to be assumed that one will exists.

On the other hand, the influence of motive-focused biases is many times weaker than the actually useful discrimination based on the technical aspects of the images. Objects images taken by a professional photographer with up-to-date equipment are ranked much better than images taken using a digital camera from the 1990s. Images without a timestamp or a watermark are ranked better than ones that do feature one. Images taken with proper lighting and contrast settings are ranked much better than images taken in a dark room, presenting the objects in gray on black. Similarly, one of the most important facets contributing to the aesthetics score the predictor returns seems to be the composition: If an object is centered in the photo taken, it will score way above images featuring multiple objects at different corners of the image (a classic example would be images that show both the actual object and a photo bar).

To reiterate, these technical aspects are visibly the main contributors to the score. And discriminating based on those images is actually useful: It allows museum-digital to present newer users who are just browsing the published collections with objects recorded with a more consistently high visual quality.

What are the alternatives, or who is the audience?

museum-digital suffers the old problem of a lack of a specified target audience. A common user may be a hobbyist looking for other versions of the model train they just bought. They might be a person interested in what art of the 16th century looked like. Or which museum to visit next. Or they might be a specialist, with a much better idea of what they are actually looking for. It is common for users to leave the page after less than a minute. But there is also an astonishing number of users who stay for hours. Accordingly, it is all the more important to offer (sort) options catering to different needs.

A new user who chanced upon the platform and randomly browses it with no further background on museums will benefit from the new sorting method. Sort orders like sorting by ID or title are linear and follow a consistent logic - but that logic may in essence just reflect its own type of randomness. Sorting newer objects over older ones is essentially a random sort order, if the museum does - as is usual - record objects as they are needed in an exhibition, newly enter the museum or are simply the next object in the shelf - all of which follow a very particular, not publicly comprehensible logic. Similarly, sorting objects by their names or titles is linear. In contrast to the object entry’s age in the database, it is also immediately comprehensible to users. But museums are free to determine the object name freely - and often this is necessary. As per the best practices around publication on museum-digital, an object name should be descriptive and usable to distinguish between different objects. As most objects simply are nameless by themselves, this often means that a colleague at the responsible museum invented a name. Seeing a green vase, there may be a rather short list of names that come to mind for most people - “green vase”, “vase, green”, “vase”, “green-ish vase”, “vase in pale green” -, making the selection of the actual name comprehensible. But “green vase” is in a very different position from “vase, green” when sorting objects alphabetically. One might actually be involuntarily be sorting the objects by curator.

Other than an object’s title/name and institution, the one other facet users see when listing objects on an overview or search page is the object’s thumbnail. And as a sense of aesthetics - fuzzy as it certainly is - is roughly shared among most people (with globalization, even actually most people), sorting objects of diverse origins by the aesthetics of their thumbnails suddenly starts to look like a comparatively relatable and reasonable sort order. Nobody will agree 100%, but the rough order instinctively makes sense. That’s more than can be said about the alternatives, unless one actually looks at the sort settings.

Presenting new users with a more consistently appealing and streamlined set of search results (at the first glance at least) on the other hand might encourage them to stay for longer. And if they stay longer, they might end up actually chancing upon more objects - including ones with less well-taken pictures - eventually.

For those users who are researchers and/or specialists, who need a linear, logical sorting, the aesthetics prediction sort option is obviously of much less merit. But it is safe to assume that is this group of users are overrepresented in the above-mentioned group of people who browse the site for hours. And it is also rather safe to assume, that they are generally more used to online databases and the existence of different sort options. Consequently, it can be assumed that they are able to change the sort settings to their needs - or at least have a higher likelihood of being able to do so.

Being very useful for the general public while less so for specialists who can be assumed to be more skilled anyway, it is sensible to make aesthetics-based sorting the default. There are thus three different default sort orders, depending on what objects one lists:

If a user lists all objects of a given instance, the default sort order remains sorting by the age of the entries
If a fulltext search was performed, the results are sorted by how well the query string matched the entry by default
Otherwise - in case of ID-based search queries like a query for all objects created in Berlin - the search results are sorted using the new aesthetics prediction by default

Operationalizing the Prediction

On a side note, operationalizing the prediction proved a challenge in itself. All of museum-digital’s servers are built for traditional web hosting. Hence, they feature quite powerful CPUs, lots of RAM and no GPU. In other words: They are least suitable for AI applications. And they are all the more unsuitable for scoring over a million thumbnails in bulk while remaining otherwise performant.

If the existing servers cannot be used, there are three reasonable alternatives. First, we could have simply rented another server. As a mostly volunteer-run project, this was not an option. Second, we could have used browser-based AI to calculate the score on users’ machines - we might have e.g. let an uploading user’s machine calculate the score of a thumbnail whenever the user uploads an image. But the users’ PCs vary widely, laptops and tablets (usually again without GPUs) have become more and more popular when compared to workstation, and thus such an approach would have made uploading images unbearably slow. It would have also not helped with calculating scores for the already pre-existing million of objects and their main thumbnails. Again, this was not really an option.

Finally, we could use our private machines (specifically: mine). And that is the approach we chose. Based on a new search API parameter aesthetics_score for querying objects whose thumbnails have not been scored yet (aesthetics_score:10001), my PC downloads the yet unevaluated object thumbnails and evaluates a score for each. These scores are then locally stored in one SQLite database per instance of museum-digital and exported into a CSV file. The CSV file is then uploaded to the server and ingested back into the database to set the aesthetics score for the relevant images. The search index is automatically updated with the new scores following the upload.

This structure unfortunately also means that the scoring does not occur in real-time. Depending on when I turn on or shut down my PC, the most recently published objects may remain uncategorized for a while. To be able to fairly represent such objects, they are assigned an impossibly high score that serves to both sort them above the already-scored objects while also marking them as yet unscored. Specifically, the aesthetics predictor returns a score between 0 and 10. As calculating with full integers is generally much more simple than working with floating point numbers, the score is multiplied by a thousand and then rounded, leaving one with a score between 0 and 10000. An object yet unscored will hence be assigned a default score of 10001.

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Dev, December 2024 & January 2025

2025-02-14T00:00:00+00:00

December 2024

Frontend

Dates in TEI transcriptions are parsed, irrespective of whether when="" oder when='' was used
Notes for markings are now publicly displayed
This was missing thus far and is now implemented similar to how event notes are displayed. If a note exists, a small “[?]” appears behind the marking title line. Upon hovering over it, a tooltip appears with the relevant information.

musdb

Names and descriptions of exhibitions and object groups can now be translated
Version control
Log of “current locations“ of an object can be exported as a CSV file
Uploaded object images can now be hidden or published in one batch operation
API extended
- (New functions)
- Transfer object dimensions
- List images and resources for an object
- Image metadata
- Publish / hide object images

January 2025

Frontend

Objects can now be sorted by the aesthetics of the thumbnail (A dedicated blog post on this will follow soon)

musdb

Batch transfer between between free text fields of object data
Alignment of the maximum field length for notes on opening hours is now consistent between UI and database
Bug fixed with switching between institutions during consistency checks fixed (relevant only to users with administrative access to multiple museums)
Literature can now be searched by editors

Importer

Core
- Automatic transformation of life dates for actors
  - Year of death “01.01.2012” now becomes “2012”, instead of 01.01 as before
- “?” and “(?)” are removed from the beginning and end of imported keywords
- Various types of brackets in keyword names are converted to regular brackets
Parser
- Stricter internal implementation of settings, all imports can now implement the start_at setting
  - This is particularly useful for the repeated execution of imports that abort due to new, previously uncovered elements and other debugging.
- New parsers:
  - MODS (mainly used in library contexts)
  - Parser for Exports from Faust for the Händel-Haus
  - Parser for XML dumps from MuseumPlus Classic (MsSQL > XML export per table > Import)
  - Bugfixes
    - Field “Verwender” in Primus parser was mapped to production events
    - Material / technology are now imported correctly in the parser for BeeCollect exports for the Industrial Museums of Saxony
„Frontend“
- CLI now also has options for switching off the import of individual areas
- Help text for command line tool

nodac

Splitting of keywords now also recognizes keywords that should be split into places, times, etc.
- Bsp.: „Helm; Berlin“ > Schlagwort „Helm“ + Ort „Berlin“
- Example: “helmet; Berlin” > keyword “helmet” + place “Berlin”
When searching for keywords with ambiguous names, both keywords and generally ambiguous terms are now taken into account
Times can now be merged with others directly from the time edit page

info

AI Use

Post image generated using AI.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

A Concordance Checker for Preparing Imports to museum-digital

2025-01-23T00:00:00+00:00

When one runs an import to museum-digital - specifically one focused on internal collection management data - there is a chance to encounter errors of unmatched entries. The import tool identified that one tried to import a yet unknown value to what is a controlled field in musdb. Common issues appear especially with actor roles and entry types.

Say, a museum’s previous database used actor roles over an event structure to express who created an object. As such, the museum entered that the object has a linked actor X that is linked to the object as a “main creator” and a linked time Y marked as the “creation time”. During the import, these roles (“main creator” and “creation time”) are then translated to museuem-digital’s event types to form an event: The object was created by actor X at the time Y. This works, because the terms “main creator” and “creation time” have been matched to the creation event type.

If a term is not yet matched to a corresponding value of a controlled list in museum-digital, the importer will simply abort the import. On the one hand this is a way to uselessly require resources for an import that cannot be completed anyway. On the other, it is tedious. One recognizes yet unmatched entries only one by one.

A Small New Tool

A small new tool, available at concordance.museum-digital.org, makes the process a bit less tedious. Users can upload all the import data from a given field (e.g. the actor roles) - one a line - and check whether they are already matched using the concordance lists or not.

For entries that are not yet matched, the tool will offer selection boxes to perform the matching using the graphical user interface. Once all entries have been matched, one can then generate the relevant lines of code to enter the missing entries to the concordance list upon the click of a button.

While simply checking and extending the relevant open source lists should be trivial even to most non-technical users, this way is certainly more convenient. Importantly, it also removes the need to run the import multiple times until one does not encounter errors caused by unmatched entries anymore. And, well, it certainly is also more convenient to match to regular human language values than to the internal IDs of the target values.

The concordance checker’s MIT-licensed code can be found here.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Development, November 2024: "Real" separated Measurements and a Better Recognition of Tags, Places, etc.

2025-01-13T00:00:00+00:00

musdb

Basic re-implementation of separated measurements for objects (tab: “addendum”)
The type of measurement (width, length, etc.) is now managed using a controlled list, which can easily be extended. This also allows for measurement types of different levels of specificity (“width” vs. “width of socle”)
The new implementation allows users to identify whether a measurement is exact and add notes for each measurement
Measured values now need to be entered as a floating point number - consistent search for objects smaller or larger than a given size is thus made possible (previously only available for entries where the system could deduce a numeric value from whatever had been entered)
Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
The list of spaces that can be linked to an object as its current location is now sorted alphabetically

Frontend

Re-implemented separated Measurements

Import

The import tool now also uses the new implementation of separated measurements
Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
Links between two objects are now imported using the dedicated data type for this purpose. They had previously been imported as regular web links.

nodac

When merging two entries, links between the entries and collections, exhibitions, etc. are now reflected and rewritten
previously, links to such data types prevented the completion of the merge
If an entry’s name is marked to always belong to e.g. a tag, the buttons for transferring the entry to the actor, place, or time vocabularies are now hidden.
If an entry is moved between vocabularies, links between the entry and objects as “display subject” / “displayed place” / “displayed actor” are reflected and translated into new links using the appropriate link type.
A new context menu on overview pages allows for a quick access to functionalities like the splitting of entries

info

AI Use

Post image generated using AI.

tip

tip

This content is licensed under a Creative Commons Attribution 4.0 International license.

Version Control & Batch Transfer Between Data Fields of Object Records

2025-01-13T00:00:00+00:00

The new year 2025 comes with two long-awaited new features in musdb: detailed version control of object data and an option to batch transfer object data from one free text field to another.

Version control

Until a few days ago, a central and sorely missed feature in musdb was a detailed version history of the data records. For example to be able to trace and restore data after an attempt at batch processing gone wrong or careless errors when deleting field contents.

Such a view of all previous versions of an object record since the start of recording (May 2024) can now be accessed via the “record history” tab when viewing and editing an object in musdb. A new “Open versioning” button appears right at the top.

The detailed version history can be accessed via a new button at the top of the “record history” tab when editing an object.

Clicking on it opens an overlay in which the various versions are listed as a table. The various aspects of the object data set are divided into different tabs and therefore different tables, e.g. for basic information, administrative information, links to collections, keywords, etc.

The version history of an object is presented in a table view in the overlay. Cells whose values have changed in one version respective to the previous one are displayed with dashed borders. Empty cells are dashed sideways. In the screenshot, the most recent version can be seen at the top (empty cell in the column “end”). Between lines 2 and 3, the object description was significantly shortened, leading to a reduction in the quality index in the most recent version (top row).

Batch transfer

A second frequently requested feature - especially after importing - has been the option to transfer the content of one data field of object records to another. If, for example, the information previously stored in the non-publishable data field “object history” is to be stored in the “detailed description” in the future and published there, the transfer from one field to the other can now be carried out for hundreds of objects with the pressing of a few buttons. Like all other “Global Change” options, the batch transfer between different data fields always refers to the results of an object search. The function is available via the sidebar of the object overview once a search for objects has been executed.

If a search filter is set for objects, various options for export and batch processing appear in the bottom right of the sidebar. A new option “batch transfer” can be found at the very bottom of the list.

Screenshot of the new option for transferring object data in batches from one free text field to another. In addition to the free text fields for the object, the two “special sources” “separate dimensions” and “separate information: material and technology” can be selected, as shown in the screenshot. When transferring between data fields, the content of the target field can be overwritten with the content of the source data field, or the latter can be prepended or appended to the target field’s content.

The batch transfer between fields based search results can also be used via musdb’s API. For this, a new API route /object/transfer_by_search/{mode} has been added.

tip

tip

This content is licensed under a Creative Commons Attribution 4.0 International license.

State of Development, October 2024: Searching Objects Currently On Exhibition, Linking Location and Acquisition of Literature

2024-11-06T00:00:00+00:00

After the blog has been very quiet this year with regard to the technical development of museum-digital, we are now trying to publish the summaries of new developments - enriched with screenshots - that are prepared for the monthly “regional administrators” rounds in Germany anyway.

These are in the form of listings, and this is how it should be here too.

Frontend

Features & Improvements

Some improvements in background scripts, especially better handling of timeouts when calculating “Similar objects” in very large instances
Contributors, linked locations and times for an object group are now listed alphabetically by name
Table headers for event components (who, when, where) are now only displayed in the A4 PDF if there is also content for the row
New search option for object searches: “Is currently on display”
Links to the Themator now use the new URL scheme of the Themator
(https://themator.museum-digital.de/t/690 instead of https://themator.museum-digital.de/ausgabe/showthema.php?m_tid=690&tid=690)

The new filter option “Currently on display” in the overlay for the advanced search for objects in the frontend of museum -digital.

The contributors to an object group are now sorted alphabetically by sorted by name .

Bugfixes

Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)
Exactness setting in the “refine search” overlay was not transferred to the actual search query
Simple embedding of an object (analogous to YouTube videos, for example; accessible via the “Cite” menu of an object page) had various errors / now works again

musdb

Features & Improvements

In the API documentation of musdb there is now a note that the frontend also has an API
- Frontend API
  - You do not need to authenticate yourself to use the frontend API
  - The frontend API tends to be faster and easier to use
  - Is read-only
- musdb API
  - Can do more: Can also see non-public stocks and fields / data types
  - Is much more granular (more queries for the same data, but you likely get exactly the data you are looking for instead of e.g. all data known about a given object)
  - Can be used for writing data
Suggestion lists when searching for vocabulary terms in the side column of the object search page have been revised
- Tooltips appear when hovering over
- Implementation in Vanilla JS, removing jQuery
- (this means significantly better performance of the search results list in list format, because jQuery no longer needs to be loaded)

The suggestion lists for places, times, persons and keywords in the quick search function of the object search mask have been re-implemented. The main visible benefit is that explanations now appear directly when hovering over the terms in the list.

User page / Login
- Log of logins now also with IP and user agents
- Login via login persisted in the browser (“Remember me”) is logged and displayed
- All browsers permanently logged in via cookie are forced to log in again after a password change
- New option to invalidate all remembered logins on other devices (browser must be logged in again)

The “Login log” in the account settings can be used to track when and in what context one’s own user account was accessed in musdb. This allows for the identification of account takeovers by third parties. Newly logged and/or displayed are: IP address used to log in, the user agent (identification of the browser) and whether the browser was automatically logged in via a permanent login cookie (“Remember me”).

A new button in the toolbar of the account settings in musdb allows you to log out all permanently logged in browsers / devices from your own account.

Object
- More restrictions for the publication of object data records.
- An object can no longer be published if:
  - … the object name is the same as the object description
  - … the description contains the character string “lorem ipsum”
- When object entries are unpublished / hidden, the images linked to the image are renamed (thus invalidating links to the images). When publishing the object again, this is reversed so that existing links work again.
- Spaces in selection lists are now listed alphabetically as the actual location when linking
Literature
- Acquisitions can now be linked to literature
  - Previous owners etc. can thus be linked to a literature entry
- Spaces (actual location) can be linked to literature

Via the new tab “Administration” tab on tab on the literature editing page, the location and access context of the literature entry can be linked. This can be useful if the literature module is also used to manage the museum library. is also used to manage the museum library .

Bugfixes

Overlay for setting searches for objects: Multi-word search terms were converted into multiple searches instead of being searched as a string of words (“red helmet” > “red” AND “helmet” instead of “red helmet”)
Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)

Importer

Link between literature and spaces (actual location) as well as acquisitions is implemented in the “core” of the import tool
ImageByInvno parser (assignment of images to objects via inventory numbers contained in the file name) can now be used to import PDF files

files.museum-digital.org

Added a small script to enhance PDF metadata based on an XML sidecar file. See e.g.: https://files.museum-digital.org/de/Praesentationen/2024-10-18_md-deutschland-eV-stellt-sich-vor_Usertreffen_MA.xml

info

AI Use

Post image generated using illustriousXL_smoothftSPO.

tip

tip

This content is licensed under a Creative Commons Attribution 4.0 International license.

Reconciliation APIs arrive to museum-digital

2024-07-03T00:00:00+00:00

Imagine you have a spreadsheet with potentially unclean data or data that is not confirmed to be interoperable. A museum may want to migrate their data to a different system or share it with an aggregator or a researcher may want to analyze data from different museums where each has their own thesaurus. To make the data legible to an external service or to understand that where one museum says “shoe” and another says “shoes” the exact same thing is meant, it makes sense to check the terms in question against larger (norm data) databases and add their ID for the entity in question. Thus, “shoe” and “shoes” become understandable as the same.

This process is called reconciliation. One of the most used tools to actually connect to external services and reconcile data this way is OpenRefine. But to do so, OpenRefine needs to be able to communicate with such services. The open standard by which they communicate is the Reconciliation API specification. Depending on how completely the API has been implemented, the consuming software may offer additional features such as offering alternatives, suggesting entries or displaying previews of entries made available through the API.

Since this weekend, museum-digital offers such APIs for different types of data.

In md:term, reconciliation APIs were added for each of the available controlled vocabularies. Besides the controlled vocabularies of museum-digital (tags, actors, places, times) this means that users of OpenRefine can now reconcile their data with select other vocabularies such as the Oberbegriffsdatei or the hessische Systematik. Besides implementing exact and partial matching of requested terms, this implementation features previews.

Two additional points are noteworthy in terms of md:term’s implementation of the reconciliation API specification.

1) Similar to museum-digital’s import tool, md:term’s reconciliation service attempts to clean up entered terms before reconciling. As such times are automatically rewritten (“1912 bis 1914” [1912 through 1914] becomes “1912-1914”), question marks and other markers of uncertainty are removed, and - whereever known - autocorrections stored in the city (Frankfurt a.M. > Frankfurt am Main) are applied to the terms.
2) Multilinguality: Whereas the current version of the regular reconciliation API does not consider multilinguality, the above-mentioned auto-corrections and rewriting are language-dependent. And so are previews and names of entities in md:term for museum-digital’s vocabularies. Hence, an additional parameter for the requested language can be supplied.

Reconciliation APIs for md:term may be found on the download page linked in the sidebar. Going this way, the language parameter is automatically set to the user’s current language.

Reconciliation by Inventory Number

To identify objects available published through museum-digital, a reconciliation API now exists in the frontend as well. It can be found at /reconcile/invno/ on each instance of museum-digital (e.g. here for the Hessian instance.
It reconciles entries by their inventory number. An additional number may be added to the URL to search only within a single institution. Where https://hessen.museum-digital.de/reconcile/invno/ allows matching objects by inventory numbers across all institutions publishing their objects on the Hessian instance of museum-digital, the https://hessen.museum-digital.de/reconcile/invno/1 will thus only search within the collections of the Freies Deutsches Hochstift / Frankfurter Goethe-Museum (the institution of the ID 1).

Reconciling GLAM data using the MD:term reconciliation API: an example

As they have much more experience with OpenRefine, I asked the colleagues from digiS whether they could provide a description of how to actually use the reconciliation API from a user perspective. This section is theirs:

Using the reconciliation API in OpenRefine is straightforward (cf. the OpenRefine documentation for a general introduction). Let’s walk through a typical workflow using GLAM data and the MD:term API!

Our sample dataset (adapted from actual data provided by the Sportmuseum Berlin) features, among others, columns on material/technique, object type and places.

These elements are just strings and consequently not linked to any authority file. Strings are not guaranteed to be unequivocal, nor are they particularly suitable for retrieval, programmatic reuse or multilingual contexts. In order for the data to be as useful as possible, let’s reconcile them with an authority file, in our case the MD:term vocabularies now readily accessible also via a reconciliation API.

If you have never used the MD:term reconciliation service you first have to add it to OpenRefine. Click on the small triangle in the header of the column you want to reconcile. Select Reconcile → Start reconciling.

In the following window, add the API endpoint you want to use (cf. https://term.museum-digital.de/downloads for a list of MD:term endpoints).

Now you can select the service for reconciliation:

In the following window, the only really important decision you have to take is whether or not to automatically match “candidates with high confidence”. Auto-matching speeds up the whole process, but you have less control over what happens with your data.

Hit “Start reconciling” to trigger the reconciliation process. You can now see that the strings have become links. In fact, they have been linked to possible entries in MD:term. If you choose the auto-match option, the most probable candidates have been selected automatically. If not, you have to select the candidate you want to match manually by clicking the arrow to the left. One arrow matches the value in the selected row only, two arrow match all identical values in the entire columns automatically.

Once you’re done with the reconciliation you can, for example, write the IDs of the reconciled values into a new column. Click once again on the small triangle on the left side of the top row of the column you’re working on. Go to Reconcile and Add entity identifiers column.

Name the column that is to be added and you will have a new column that contains the MD:term IDs.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Who is actually using musdb? And what for?

2024-06-16T00:00:00+00:00

In its most recently published survey of museums in Germany the Institute for Museum Research (Berlin) asked how many museums use controlled vocabularies and norm data. 416 of the 3059 museums who answered the additional question sheet with this particular question answered that they do indeed use norm data. The survey concerns German museums as of 2021.

On January 1st, 2021, 866 museums and similar institutions were registered with museum-digital in Germany. 688 of these were publicly listed, which means that they had recorded and published at least one object entry. The recording of object metadata in musdb is barely possible without using controlled vocabularies (in turn linked to the large norm data catalogues like the Gemeinsame Normdatei, the Library of Congress Subject Headings or Geonames), unless one really only fills out the most basic description fields. Which is to say that 688 museums in Germany were almost certainly using norm data in 2021. An important assumption in the design of musdb is that norm data adoption is most successful if users benefit from it - ideally - without even knowing that they use it.

Maybe it is exactly for that reason, but something clearly is wrong about our numbers.

Aside from questions on a mainly statistically and politically interesting global / national level, questions about the participating institutions pop up regularly in more practically relevant situations. Museums considering musdb as a collection management solution ask how many other museums use musdb for collection management. As it is both a collection management system and the gateway to the publication of museum data via museum-digital, we cannot yet answer those questions safely beyond the better known case studies.

At least for its background, museum-digital in Germany is known to be primarily used by smaller museums. Knowing the German museum landscape, this is pretty much true in terms of collection management, and less so where it comes to the publication of museum collection data. But there is no data to give a more detailed assessment. Given the approach of a centralized vocabulary management with centrally determined names for actors, places, times and tags, it would be likely to assume that art history museums are less likely to use museum-digital than other types of museums. As of now, we cannot confirm or deny that.

Getting better data would be very valuable for linking like-minded museums and gaining a better understanding of who is actually using museum-digital and/or musdb.

At museum-digital, we know details about the participating museums, but there is a lack of an overview. And details can obscure an overarching perspective (if a museum has only published its numismatic collection via museum-digital but is actually collecting all types of objects, it would clearly be wrong to deduce that is a numismatic collection based on its published objects).

Overarching statistical information about institutions can hence not be safely deduced from the published collections. It is information that the colleagues at these institutions know best themselves.

A Small Survey

Given this background, it is time that we’d try to learn some more about the institutions using musdb. And that clearly will work best by asking people themselves.

Starting today, users logging in to musdb will hence be asked to fill out a very quick survey of six questions:

Please select the type of your institution. (museum, archive, …)
Is your institution a privately-run or a public institution?
What does your institution use museum-digital for? (Collection management, publication, both)
How many objects comprise the museum’s collections (estimate)?
How large is your exhibition space (sqm)?
Please select the main topical foci of your collection.

The dialogue asking users to fill out this survey will only appear upon logins until the survey has been filled out for the museum.

What will the data be used for?

For the start, the data will only be used for answering questions such as the ones stated above. Knowing the intended use of musdb and the main topical foci of a museum however also opens up new possibilities in making musdb easier to use, especially for new users. This information may e.g. be used to set more defaults for new users that are tailored towards the intended use case for musdb instead of providing only a single set of defaults for a given functionality.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

A Re-Designed Frontend - Version 2024

2024-01-08T00:00:00+00:00

Over the holidays we worked on a re-design of the frontend of museum-digital. The reasons were manyfold. While the old design of the frontend still looked well, discussing it with different people - especially those who were not regular users - revealed some shortcomings. Others had naturally developed after five years of use or been inherent in the basic concept of the old page design.

This blog post will first deal with the issues of the old design. A summary of the aims of the new design follows as they were often formulated as a direct response to the shortcomings of the then present. Finally, a tour of screenshots through the new design concludes the post.

The new page design will be published on the night from Tuesday to Wednesday.

For those seeking only a quick overview there also is a small playlist on Youtube.

Problems of the Old Design

Inconsistency

A natural problem of the old design was its growing inconsitency. Quite a number of features and even some sections of the frontend were developed after the last re-design (early to mid 2017). As new features were added, they often did not reuse existing styles and instead used slightly different re-implementations of the same.

Hidden Features or “To View or to Act?”

The likely most consequential issue of the old design lies at its very foundation. In 2017 we had done the redesign aiming to create a design that was very much focused on presenting museum objects in a clear and attractive way.

To underscore the issue of clarity, as well as for their then increasing popularity and the possibility to implement one simply even for users of the Internet Explorer (see below), we chose to go with a stream design. Some sidebars existed for presenting additional information, but the aim was to always clearly focus only one thing. While the design gave quite some space to gaps between different sections to improve focus, it ironically reduced our freedom to label buttons (as can be most evidently seen in the settings bar for searching objects).

In summary: Many features were - at times intentionally - hidden. And thus users did not find them or even learn that they existed.

Internet Explorer

This is not necessarily an issue by itself, but a self-limitation that has now become unnecessary. Was our aim to keep the page at least somewhat usable using Internet Explorer 10. Microsoft’s Internet Explorer did, until its demise, not support many of the newer web standards and trying to keep the page simple to maintain while usable in the browser meant that we could only use older styling options (e.g. CSS grids were not supported).

Lines

Especially during the re-design we noticed that the old design had overused lines. Lines to differentiate different sections, lines to form tiles, lines around buttons. While it had not necessarily occured to anybody I personally have talked to (or they did not say so), a comparison with the new design reveals that the number dividers and separating lines effectively lead to cluttered user interface.

Basic Principles of the New Design

Users Want to See, Read - and Then Do Something About It!

As stated above, the frontend has had quite a number of hard to find features - especially interactive ones like a the option to compare different objects or a watch list (the latter even being hidden behind an option in the now obsolete settings page) and export options. Often, they were hidden by being styled as normal entries in a list. This list in turn could either be found at the very bottom of the page (objects) or in a sidebar. This stems from the idea that users of the site primarily want to see and read things and should not be burdened with anything else unless they are explicitly looking for it.

The most radical change of the new design is our leaving that assumption behind. The new design is instead based on the assumption that people want to see and read things - and then do something about them. Be it to save them, to compare and further investigate them, or to share them.

Buttons (styled as such) offering options like the export of a data sheet, subscribing to a feed, comparison, and the watch list are hence now available immediately below the headline of any page offering such options. Whenever the screen size permits, these buttons also feature text explaining their functionality instead of being made up entirely of icons.

Do What Feels Natural

The old design featured some idiosyncrasies. To take the most blatant example: Right next to the page title there was a search bar. The position of the search bar at the top of every page suggests it being a general search bar for all sections of the page. In reality, it was a search bar only for objects. The new design now features a general search bar in its place.

In a similar vein we tried to identify unintuitively positioned or superfluous links. Some features (e.g. the maps for displaying a museum’s location or the locations of all museums represented in a given instance of museum-digital) were integrated into the pages linking to them (the institution’s page or the institution page respectively. They are now accessible as tabs, allowing users to quickly navigate back after viewing the map.

Don’t Repeat Yourself

The old design featured quite a number repeated content. Where possible we tried to identify such duplicates and offer only one, clearer way to access a certain information or visualization.

Hovering over a tag e.g. made links to the timeline and map visualizing objects linked to that tag accessible to users immediately from a given object’s page. These links were removed in favor of letting users click on the tag, see the objects linked to the tag, and then navigate to the timeline or map from there. On the one hand this more clearly communicates what is actually visualized (the objects linked to the tag rather than the tag itself), and on the other it allowed us to remove a flickering and likely unintuitive interaction.

Shapes and Positions Over Lines

This speaks for itself: While the old design used straight lines as dividers a lot, we tried to communicate the separation of different elements primarily by their positioning, size and shape in the new design.

A Tour Through the Re-designed Frontend

Start page

The new start page opens with a rundown of the current number of entries (and their development) and a slogan whose words double as links leading to essential functionalities.

A new feature on the start page is a section that is offered only to return visitors of the page, Here one can see one’s last search queries (for objects) and return to them.

The selected objects on the start page.

Object Overview: Display options

The grid view of object search results. This view is the default on larger screens.

The list view of object search results. This view is the default on mobile devices.

Extended search

Previously there were two extended search options for objects. One, linked from the navigation, listed the search options in a grid. The other - a full screen overlay - featured additional options such as negative searches, but was hidden behind an unlabelled button. The new design only features a single overlay for configuring extended searches.

Object overview: Facetted search

The facetted search has now moved into the sidebar of the object overview page and is thus much more obviously available.

Object Overview: List Results by Museum

The list of objects grouped by museum has now been integrated into the object overview. Previously it was a separate page.

General search

The general search function searches across the different types of entries in a given instance of md. Up to five entries are directly displayed as one types.

Collection overview

The collection overview has been almost entirely reimplemented. It now loads much (!) faster and supports displaying all of an institution’s subcollections, regardless of their “indentation” in the hierarchy of subcollections and sub-subcollections.

Museum overview

The museum overview has similarly been reimplemented to a significant extend. When searching, an institution’s place is now considered besides its name.

The museum overview now features a map of the museums. This map corresponds to possible search queries. Previously, a similar map had been a separate page.

Institution pages

The collection list on institution pages has - like the other sections of the page - now been moved into a tab. It can now be filtered and collapsed.

The object section on institution pages now features some summary information on the objects rather than consisting only of links and numbers.

Object Overview: Interactions Visualization

A new visualization is built upon LIDO events: What happened to the selected objects (say, search results) and when did it happen?

Object Overview: Tag Clouds

Tag clouds now also include actors, places, and times. They can be generated for any object search query.

Object Page

An object page. Significantly, options and interactions such as the generation of a PDF data sheet are now made available immediately below the page title.

Second half of the object page.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Automatically enforcing consistent naming of places

2023-11-27T00:00:00+00:00

Last week I wrote about how new actors find their way into museum-digital’s controlled vocabulary for actors during imports. One of the first steps detailed in the post is the automatic cleanup of the actor’s name and the application of some rules to ensure a consistent naming of actors.

For time names a much more extensive cleaning is done, both in the case of imports and when working directly in musdb. Time names in the sense of the controlled vocabulary of museum-digital describe clearly defined times. This may be timespans, years, or full dates. But the number of possible input values is thus already significantly limited. In fact, it is small enough, and timespans, years, etc. are uniform enough to allow the automatic parsing of time names. Where that is possible, they are automatically rewritten to their default form in the controlled vocabulary and automatically translated to some 30 languages.

In the case of place names a similar cleaning and consolidating of place names was limited to the simple stripping of leading and trailing spaces and commas and the removal of indicators for uncertainty (e.g. trailing question marks). Since the weekend, place names are more extensively rewritten to ease the work of the vocabulary editors and ensure a more immediately consistent data entry both in musdb and when importing.

General rewriting

1. Simple spelling issues

The very first step in rewriting entered place names remains the removal of superfluous characters. As such, duplicate white spaces, trailing commas, etc. are removed.

Example: , Berlin , > Berlin

2. Removal of indicators for uncertainty

Next, known indicators for uncertainty are used to update the dedicated flag describing the certainty of the link to the place and then stripped. As they indicate the relation between object and place, they are not actually part of the place name and can thus be removed safely.

Example 1: Berlin ? > Berlin
Example 2: Maybe Budapest > Budapest

3. Removal of duplicates in enumerations

Commas are used in input place names in two ways. On the one hand, they may be used to further specify a place name (Beijing, PRC), on the other, they are often used in import data to designate multiple space names at a time (Beijing, Tokyo, Nairobi; this contradicts the logic of a database altogether and needs to be cleaned up manually be the vocabulary editors).

In both cases, duplicate names in the enumeration are superfluous and can be removed.

Example 1: Berlin, Germany, Germany > Berlin, Germany

4. Language-dependent rewriting

The next steps in rewriting entries depend on the language of the entry. In musdb, the language the user set to use musdb is used to guess the language they are entering data in. In the case of imports, the default language of the given instance of museum-digital that a museum imports to is used.

4.1. Extension of common abbreviations

Where there are abbreviations, there are also unabbreviated names. And surely, both will be used. Will inevitably leads to duplicate entries. In the case of some common abbreviations, they are thus automatically rewritten to a canonical form.

Example (German): Adalberthstr. 12 (Berlin) > Adalbertstraße 12 (Berlin)
Example (Hungarian): Vaci u. 12 (Budapest) > Vaci utca 12 (Budapest)

4.2. Reordering names in commas based on indicators for more specific place names

As stated above (3.), commas may either indicate a specification of a single place, or they may indicate that the entry actually refers to more than one place. Some components of a name can be used to almost certainly determine that the former is the case - and which place of the given list is the specific one and which one is a superordinate named mainly for clarification. Common such names are “street”, “plaza”, “pier”.

If there is exactly one comma and such a name component is encountered, the entered place name can be rewritten to contain the less specific name only in brackets.

Example (German): Berlin, Adalberthstraße 12 > Adalbertstraße 12 (Berlin)
Example (Hungarian): Vaci utca 12, Budapest > Vaci utca 12 (Budapest)

If both names contain such an indicator, no rewriting is applied. Vaci utca 12, Vaci utca 13’ is thus not rewritten.

4.3. Budapest special: Extending the names of districts

Street names in Budapest are usually referred to including the naming of the district. These districts are referred to in a number of ways. If they are referred to using roman numerals, this is automatically extended to the canonical form.
This rewrite is only applied if the language is set to Hungarian.

Example: Petőfi Sándor utca 3. Budapest, IV. > Petőfi Sándor utca 3. (Budapest, 4. kerület)

4.4. Reordering names in commas based on country names

Similar to the rewrites described in 4.2., country names can be used to indicate a hierarchical relationship between two places in a comma-separated list. If one given name is a country name and the other is not, it is likely that the non-country name is part of the given country. The comma can be replaced with brackets while the name can be reordered into the preferred specific (unspecific) form.
This check is also applied to names separated by hyphens.

Example (German): Budapest, Ungarn > Budapest (Ungarn)
Example (Hungarian): Berlin-Németország > Berlin (Németország)

There are however some common cases in which this logic does not apply - significantly cardinal directions. If one such term is found to be the non-country part, the rewrite is not applied. As such West-Deutschland remains West-Deutschland without being rewritten.

The list of names of countries and historical countries used stems from Wikidata (thanks!).

Where do these rewrites apply?

The rewrites listed above are now implemented in musdb, the import tool, and nodac.

They have also been used to consolidate existing place names, allowing us to identify some 500 duplicate place entries over the weekend (that amounts to almost 0.7 percent of the whole vocabulary). Clearly, identifying similar cases of regularly appearing, varied ways to express the same thing, and determining a canonical way of naming places in such cases holds a lot of potential for reducing the editors workload and improving data quality for everybody.

Quality Assessments Like in musdb: Now For Everybody

2023-10-12T00:00:00+00:00

At yesterday’s Autumn Conference of the Working Group Documentation of the German Museum Association (Herbsttagung der Fachgruppe Dokumentation des Deutschen Museumsbunds) a new web service in the broader realm of museum-digital was released: museum-digital:qa.

museum-digital:qa reuses the importer’s relevant functionalities to accept museum object data in a variety of input formats - both open standards and software-specific export formats - and allows running different tools concerning data quality on the uploaded data.This way we can make the quality checking functionalities that have thus far only been available as part of musdb accessible to museums almost regardless of the collection management system the museum uses.

At the same time, the tool can be extended with further checks that are not (yet?) built into musdb itself. As a first such check, museum-digital:qa can be used to check the object records’ conformity to the field set suggested for a minimum viable object record for publication of the working group of the same name (German: AG Minimaldatensatz).

While aspects of the import tool are reused extensively, no actual import takes place. On the other hand, this reuse means that museum-digital:qa will automatically support yet more input formats as more data of different sources can be imported to museum-digital, while actual maintenance work on museum-digital:qa should remain minimal.

P.S.: A long awaited check - checking the plausibility of the stated license status of images and media files linked to an object based on its creator / creation - has now been integrated into musdb as well as md:qa.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

EODEM Version 1.0 released

2023-09-04T00:00:00+00:00

Since September 1, 2023, the first stable version of EODEM has been released. EODEM is implemented as a LIDO profile and aims to enable museums to share their object data - especially in the contexts of loans and exhibitions - with other museums at the click of a button. Congratulations!

museum-digital:musdb has supported EODEM since February. But the usefulness of a standard is determined by how widely it is adopted. Thankfully, there is now a second collection management system to implement EODEM: Zetcom’s museumPlus.

You can learn more about EODEM on the project’s website. See also the very insightful notes from the project’s co-coordinator Rupert Shepherd.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

API documentation for frontend and md:term

2023-08-16T00:00:00+00:00

md:term was designed to provide an API first and foremost. The frontend supported a full API providing to all access that the HTML version does since about 2016. We never got around to fully and systematically document the APIs however. Starting today, an OpenAPI documentation is available for both md:term and the frontend of museum-digital.

Along with this, we can gladly annouce a new, updated and much more performant search API for md:term (see the API documentation for now).

You can access the documentation here:

md:term
frontend (via the global portal)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Summary of the monthly user meetup (April 2023) / New features and improvements

2023-05-07T00:00:00+00:00

We continued the series of monthly user meetups and again discussed the new features and improvements. A summary can be found below.

New Developments

The last month has been an exceptionally slow month in terms of technical development around museum-digital. There are however some newsworthy tidbits.

musdb

Recording external IDs for museums

Museums, like all of us, are present in more and more databases. For linking different databases, it is useful - sometimes necessary - to know the ID of the same entity in both databases. Hungarian law thus mandates collection management systems, which are to be accredited for fully paperless use in museums, to allow storing a museum’s ID with the Hungarian Statistical Office. musdb can now do exactly that: Store external IDs of museums in a given list of external source repositories / databases.

The list of available external databases and the regular expressions to validate IDs in them are available here. For now, the list only contains the Hungarian Statistical Office, but the more options there will be in the future, the better.

More fields covered by “list results” and Excel export

The “list results” page (a.k.a. table view) for object search results offers the option to view the selected objects’ data in a customizable tabular format. Mainly because of that exact tabular format, it is not possible to display everything a full database view of the object offers using the “list results” page. Over the last month, we have nevertheless extended the list of displayable fields in the “list results” table.

Thus, it is now possible to display all translations for the object type, object name, descriptions, etc. of an object. Similarly, it is now possible to list all Weblinks linked to the object as a compiled (comma-separated) field. As the automated report generation and the Excel export tools build upon the same basic code as the “list results” page, these new fields are now also available in those cases.

New field for objects: “Last change of permanent location”

It is now possible to manually record the last date, the permanent location of a given object has been updated. This obviously makes sense for tracking relocations and re-organizations within the museum. Importantly, the field needs to be manually filled out, as the date of the last change of the permanent location may be far in the past or frankly unrelated to the database, even if there is one (say, the museum moves the depot to a different place, but the date is only later updated in the database via a batch editing process).

User interface

After there were some reports of people not seeing the difference between a valid form in musdb and an invalid / incomplete one, we updated the design of submit buttons. Submit buttons in incomplete forms are now blurred out, to add another hint at the incompleteness of the form.

Frontend

Improved performance and decreased bandwidth usage: serving images in webp

Over the last years, there have been a number of new image formats seeking to combine all the features of older and well-established formats like JPG and PNG with an improved compression. The most established format of that newer generation of image formats is webp, which is by now well-supported by all modern browsers and most local image viewers.

To save bandwidth and improve loading speed, we now store and serve full-sized webp versions of newly uploaded object images along with the regular jpg versions. If possible, the webp version is served on object pages. An additional benefit is the aforementioned support for features jpg files do not support, such as transparent image backgrounds.

iFrame embedding now only possible from whitelisted sources

If a museum wants to embed their data from museum-digital into their own website, there are traditionally two ways to do so. The better, but much more complicated option is to use the API to fetch the relevant data and present them in any way one wants. Much easier (and cheaper obviously) is embedding the museum’s data using iframes.

Unfortunately, iframes can also be used for attacks on users’ login data. We have thus now restricted this option to only allow embedding from whitelisted sources. If a museum wants to embed their data this way, this means that they now need to notify their regional administrators beforehand.

Importer

The importer now supports imports for exports from Startext’s HiDa in the configuration of the Saxon State Agency for Museums.

Other News

In other news, the Youtube channel has picked up steam, with some new tutorials both in German and Ukrainian.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Summary of the monthly user meetup (March 2023)

2023-04-05T00:00:00+00:00

Yesterday, we held our regular user meetup as scheduled. As promised, below you can find an overview of the new features and updates below some more general points.

General

YouTube channel

There now is a museum-digital YouTube channel. For now, one can find some German-language screencasts on different features in musdb and nodac there.

New sections on project page (www.museum-digital.org)

As already described in the previous post, the project page www.museum-digital.org now features a calendar. Since then, we have also added a small page listing people working on museum-digital - e.g. some of the coordinators and developers - as well as power users. The list will obviously always be more than incomplete, but if you want to be listed, just send a mail.

Development

musdb

Allow batch editing of photographers on image list

Running batch updates on the ownership, license status and photographer of a set of objects can be done either via the institution-wide settings page or via a selection on the image list page. The batch updating functionality of the institution-wide settings page works by replacing the given entries in one of the fields for all objects of the museum - it is thus not suitable if one is to only update the photographer field for a list of objects from a single collection (rather than the whole museum).

Batch updating images’ legal information from the image list (click on an image in the list and drag it to activate selection mode, then update by selection) was only possible in the case of image licenses and rights holders. With a small update, it is now possible to also batch-update the images’ photographers field that way.

The API can now update object event

Thus far, the API could only be used for adding wholly new object events (who did what, when, and where, with the object) or deleting them. With the updates of this month, now possible to directly update events.

This is already actively used in customizing musdb using a Tampermonkey script by some of the norm data editors. They can thus add a custom button to transfer incomplete actor or place information to the object’s description. This button, on the other hand, would not be useful for regular users at all.

UI improvements for the object search interface and new sort option

After a very fruitful discussion and feedback from Hungarian colleagues, we got around to some user interface improvements in the object overview and search interface. A lengthy discussion can be found in a dedicated blog post in German. To sum up: there are now headlines for each subsection of the sidebar and tooltips explain the different buttons and search and sort options upon hovering one’s mouse over them. Additionally, a new sort option was added to sort objects by the number characters within the inventory numbers of the searched objects.

Two bugfixes: Navigating to the previous or next object in a selection

It is a very simple but similarly useful feature: One runs an object search and thus creates a result list of objects. After viewing or editing one of the objects, one can then proceed to the next one in the results list by clicking at the arrows at the top of the sidebar.

Unfortunately, musdb had two bugs preventing this navigation for some of the available sort options. On the one hand, newer sort options required manually adding the search fields to a list specific to the previous/next navigation thus far - and had not been covered yet. Navigating to the next object after sorting by e.g. the length of the searched objects was thus impossible so far. The specific list of sort fields has now been removed in the previous/next navigation. Instead, the main list of sort options from the main object search class is directly used - in effect preventing such issues of “forgotten” values from ever appearing again.

A second issue concerned string-based sort options. For an improved performance, the previous / next navigation works by querying the underlying search index for the next object where e.g. the object ID is lower than the current one (`WHERE id > 20000`). The same search logic is straight-out not possible with string-based sort options, as search queries for values greater than or smaller than a given one cannot be executed with string-based fields.

Fortunately, the issue came up just after we had implemented an option to search and sort objects by the numeric components of their inventory numbers. Thus, the previous / next navigation can fall back to that sort option if a user wants to navigate to the next object after sorting by inventory number. As stated back when we introduced the new search and sort option: It should work for most museums. Unfortunately, there is no such option to mitigate the issues of string-based sorting when sorting objects by their names. The best we could do in this case is displaying a clear error message, stating that the previous / next navigation is not available when sorting by the object’s names.

“Home location” can now be selected as a required field when adding new objects

On the institution-wide settings page, museum directors (or those holding the same user role within musdb) can determine which fields are required when adding new objects. The list of selectable fields automatically covers all free text fields linked to the object. Repeat fields and references to other sections of musdb need to be manually implemented however. As the home or permanent location of an object is a most logical and common field to be required for all object, it was only consequential to prioritize making it available as a required field. We have done so now.

Major performance improvements in “list results” page

The “list results” page provides for a table view of object search results, allows the export of search results to Excel, and is - under the hood - also used for generating custom reports. We managed to achieve a considerable improvement in the page’s performance by bulk loading data and simplifying the code of the HTML page. Both adjustments should not really be noticable when one tries to list or export only some objects. Exporting some thousand objects should work considerably faster now however.

frontend

PDF metadata for main object PDF export

XMP metadata are now written into exported PDF files. Thus, users can more easily identify authorship, titles etc., especially when using specialized software like e-book readers.

Linking from collection descriptions

Collection descriptions are now being parsed for the existence of URLs. If one, starting with https:// (http:// is ignored) exists, the URL is automatically transformed into a clickable link.

nodac

We added some translation variables on the start page.

themator

The “themator” has received a small user interface improvement for mobile devices. The topic-specific navigation or table of contents is now moved below rather than above the main content of a page in the regular “topic” viewing mode on mobile devices.

Importer

Importing GIFs

To simplify the handling of images, museum-digital internally only uses JPG image files (webp versions are generated for a faster loading, but are optional). While they lack some support for features like transparency, JPGs provide for a much better compression with images of three-dimensional things and subjects not featuring solid colors. Newer formats like AVIF and Webp unfortunately still lack the support among locally installed image viewers for us to fully switch over to them.

The importer handles input PNGs by converting them to JPG files. Starting this month, the same can be done with input GIF files.

Improved handling of blacklisted actors, places and times

One of the key issues with imports is what happens after imports. Specifically, museum-digital uses controlled vocabularies for actors, places, times and tags that can be linked to an object directly or via events. When people enter new such entries using musdb, there are a number of safeguards and nudges to ensure productive inputs. Adding a new actor, for example, requires one to enter at least 10 characters of a description for the actor. Input fields for the date of birth and death are also provided. If possible, one can enter a Wikipedia or Wikidata URL (or ID) and directly fetch the relevant information from there.

During imports, there is no space for asking questions to the importing institution. Incomplete or false actor, place, time and tag names (for example, if two actors are linked as one; “John Doe and Jane Doe” is not one single actor!) are given, these are necessarily imported into our controlled vocabularies by default. Depending on the import, this puts a significant burden on our volunteer group of vocabulary editors.

To eventually get to a solution, we have added options for automatically rewriting terms (e.g. “Berlin, Germany” will be autocorrected to “Berlin”) and blacklisting them completely (“Good morning …” is no actor“). Thus far, blacklisted terms were simply ignored by the importer.

“Unknown painter” is not an actor, but the information that it is a painter may still be useful. To allow us to blacklist more and keep the controlled vocabularies “clean” without losing such data, the importer now imports blacklisted terms in event components (“who painted it?”) into the event annotation. If the only known information on the whole event is blacklisted, the event cannot be created - an event always requires at least one linked actor, place or time. The event annotation of “empty events” is hence moved to the object description automatically.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Use Pixels, not em in media queries

2023-03-08T00:00:00+00:00

In CSS, one can use different units to determine sizes. The most relevant are:

A pixel (px) is - in theory - fixed to the single dot of a screen. It thus depends on the resolution of the screen primarily, at least in grey theory.
A point (pt) is a single point on a printed sheet of paper. Its roots lie in the DPI setting of the print job.
em are relative to the font size. If nested elements come with font sizes determined in em, the scaling is appropriate nested (a headline in a generally enlarged section will be further enlarged than a headline in a regular section).
rem is relative to the primary font size of the document.
vh and vw are relative to the total screen height and width respectively. 10vw are, e.g., 10 percent of the total width of the screen.
Some others, like cm or mm exist, but are less used - especially when working outside of print contexts.

Being relative to the font size and nest-able, em values are a great fit for setting sizes of page elements and we thus use almost exclusively use em and rem values for borders, font size, etc.

To make a website responsive, CSS offers a functionality called media queries, with which one can check for characteristics of the browser window. This, obviously, concerns primarily the size of the window, but can also extend to issues such as the activation of a dark mode (if you see this page with white text on a black background, that is the consequence of a media query).

Now, we had also been using em values for probing screen properties using media queries, and the results were far from ideal. Especially phones from different manufacturers provide very different em values even at roughly similarly sized screens. Thus, it becomes hard to determine, if a user really is using a phone, a tablet, or a full sized PC.

Ironically, it is exactly pixel values, that work much more reliably. In theory, a media query asking if a screen is less than 768 pixel wide should definitely be answered negatively in the case of a current flagship phone with roughly 1400 pixel width. In reality, browsers scale down the page and will answer positively, accepting that phones should be presented to CSS as less than 768px wide.

We have now switched to using pixel values for running the media queries to determine the target page layout. And the results are much better, than when we used em values.

Analyzing and Evaluating Collection Data in musdb

2023-02-26T00:00:00+00:00

A general benefit of using computer systems is - or should be - that data once recorded can be retrieved and reused in different contexts than originally intended and retrieval and reuse are incredibly easy to do when compared to the physical realm. And thus comes the want for ever more pervasive and broad evaluation - it has always been useful, but using computers it is possible as well. Where finding out the average length of all objects in one’s collection would have been a work of months before and was very error-prone - checking the index card for each object in the collection, noting down the length in a new list, and summing them up - it is now done with a single database query or formula in a spreadsheet program.

Now, what about musdb? For deeper analysis of aggregate collection data - take for example the issue of average lengths above - the best way thus far were exports. The simplest way in this regard was surely the “list results” functionality in the sidebar of object searches, which provides an Excel export of selected fields for all objects filtered through an object search. The data thus exported could then be further investigated and turned into charts using Excel or a similar spreadsheet program.

While this is a rather powerful way for analyzing one’s data in aggregate and does not require the ability to code, it is still cumbersome and involves too many steps to be regularly used (unless one has an immediate occasion like an upcoming report). More immediate ways for analyzing search results for objects (or one’s whole collection) were otherwise limited in scope, like the map view for search results, which displays all events linked to the selected objects.

Starting with a minor update this weekend, musdb now supports a more straight-forward but generally applicable way for evaluating search results or whole collections.

The entry point to this new tool comes in the form of a small button - identified by the light bulb icon - in the search options bar on object search pages.

Clicking on the new light bulb icon in the search bar opens an overlay for evaluating the search results.

A click on the icon opens an overlay featuring four tiles. Each tile can be set to provide a chart or a table of the values found in a given field for all objects selected by the search query.

Which field’s contents are evaluated, and the way the data is grouped (grouped by year and year and month in the case of date fields; or totals and averages in the case of numeric fields) can be adjusted by clicking on the two select fields making up each tile’s headline. Changes to this selection are stored in the browser to allow for an easy comparison for the results of different searches.

Below the headline of each tile, one can find a navigation offering two options for displaying the information and a download button:

Chart: This is the default view. It provides a visualization of the data using line charts for numeric fields or date fields grouped by year and pie charts for text fields. To not clutter the interface and improve performance, pie charts are limited to displaying the 20 most used values.
Table: This option provides a list of the values encountered in the selected field, sorted by their popularity. Each value is clickable, allowing one to further narrow down the search results (thus, we almost accidentally implemented a faceted search feature). Like the pie charts, the table view is limited to displaying the 50 most used values for performance and a simpler user interface.
Download: Using the download button, one can download the same data displayed in the table view - but without any limit on the number of results - in a CSV file. This file can then be opened using a spreadsheet application to do a deeper analysis or build other types of charts.

The “analyze” overlay provides four tiles for visualizing and listing values in a given field of all selected objects.

As always, we are very curious about user feedback and suggestions for further evaluation functionalities that can be integrated into the overlay.

info

Image Credits

“Exacta-Continental 6000”. CC BY-NC-SA @ Heinz Nixdorf MuseumsForum & Jan Braun

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

A Timeline for an Object's History Within the Museum

2023-02-19T00:00:00+00:00

In musdb, there’s PuQi, indicating aspects of an object that may be better or more extensively recorded for publication. There’s “Plausi”, indicating implausibilities in an object’s recorded events (e.g. if the object was supposedly created by somebody who was clearly not alive anymore at the time of creation entered in the object record). There are the improvement suggestions, which - true to their name - suggest obvious improvements to the object record that can be entered almost automatically: If a painting is a self-portrait, and the painter is known, it is most likely also known, who is displayed in the painting.

Similarly located in the sidebar of the object editing page, musdb now features a timeline for all time-based information of an object. Primarily focused on providing an overview over said time-based information and hinting at less prominent data points like conservation / restoration measures and loans in the history, this new feature has proven to be located just right. Provenance researchers now use it to find blank spots in the object history that are to be further identified.

Timeline preview in the sidebar of an object editing page in musdb.

How to Use the Timeline?

A preview of the timeline is always visible in the sidebar of the object editing page in musdb, displayed as a simple line with colored indicators for any event, shipment, etc. that has been recorded as happening to the object.

By clicking on the timeline’s preview, the timeline opens in a full screen overlay, which shows the timeline in its full form - with markers for the year of the earliest and latest entry on the timeline (the end of the timeline is always the current year, unless a date in the future has been set for the object [e.g. the copyright expiration date]). Above and below each indicator for something taking place, there’s now also a textual representation of the type of event taking place.

Below the timeline, there is a table presenting the same information in a more orderly fashion. Hovering over an entry in the table highlights the same entry’s representation in the timeline and vice versa. A left click on the entry opens the object page at the respective position, so that one can easily access detailed information about the data point represented.

By clicking on the timeline preview in the sidebar of the object editing page in musdb, one can open an overlay displaying the timeline in full. While taking the screenshot, the mouse pointer was hovered over the event entry “Painted in 1456-1458”.

Finally, the overlay features two date pickers at the very top right. These represent the earliest and latest displayed date in the timeline and table. By updating the dates in these respective fields, one can “zoom” into the selected timespan.
This “zoom” may especially be useful if one works with very old objects like roman coins, whose history is largely unknown between their well-recorded minting and the entry to the museum. In such cases, the timeline may have many entries at its very start and its very end with a large blank spot of 2000 years in between. By appropriately scaling the timeline however, all events that have taken place since the object entered the museum would be so close to each other that they would practically be indistinguishable in the timeline without the zoom feature.

What Can Be Seen on the Timeline?

Now, let’s recapitulate. What can the timeline display exactly? All time-based data points means that the following fields and concepts are displayed:

Events
Acquisition
Reservations
Checks (for completeness or condition)
Damages
Restoration and conservation measures
Loans
Shipments
Exhibitions
“Periods”: Date any copyright on the object expires, freeze period, scheduled end of deposit, etc.

Additionally, there are some fields that may or may not provide legible information to display. These fields are historically free text fields that are however meant for recording date informations and that can be converted into date fields by enabling the respective strict modes. musdb attempts to parse the information provided in these fields as date information as best as possible, and will display the data points if the extraction of date information was successful. A full date like 2022-01-02 clearly is no proble. A year like 2022 would be parsed to a time span from 2022-01-01 until 2022-12-31. But a statement like “At the start of 2022” is practically illegible and will be ignored in the timeline.
The fields thus displayable only if they contain legible information are:

Estimation of value
Determination of insurance value
Time of first registration in the museum
Time of entry to the museum (this same information can however be parsed clearly from shipments and acquisition if the respective dedicated modules are used)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

EODEM - Efficiently Exchange Object Information During Loans

2023-02-15T00:00:00+00:00

It is a classic situation. One museum loans objects to another. Along with the objects, the receiving museum gets an Word or Excel file containing information relevant to the object’s conservation (e.g. minimum and maximum viable temperature). The museum then proceeds copying and pasting the data into their collection management system. This obviously takes a lot of time and provides for a lot of opportunities to make mistakes.

On the other hand, it’s a very clearly defined situation, that is repeated again and again in a very similar pattern. Which is to say, it is a perfect subject for automation.

Now, some years ago, a working group came together in the context of CIDOC to do exactly that. Since not all museums use the same collection management system, the necessary first step towards automating the exchange of object information was obviously to develop an open standard for the same, that could be implemented in the different collection management applications.

The standard thus developed - EODEM, short for Exhibition Object Digital Exchange Model - is now mature enough to have gone into a public beta release phase. We’re very, very late to the party, but it was time to try our hands on implementing an import and export for EODEM. The import had already been done by November, but by the time of the larger update and re-design of January, we also managed to release an EODEM export.

This blog post will continue explaining how to import and export EODEM data before closing on a general perspective on the usefulness of EODEM at this point in time and in the foreseeable future.

How to Use EODEM in musdb?

As with loans in general, there are two situations in which one might want to use EODEM. The receiving museum can save a lot of time by importing EODEM data, while the sending museum can help out colleagues by exporting it.

Importing EODEM data

In the (German language) handbook, one can read a lengthy explanation of how to import object data into musdb (a less extensive description can be found in the blog as well, this time in English). In short: One first needs to generate a password to access the import folder using a WebDAV client. After mounting the folder, one can proceed to upload the data. Finally, a configuration file needs to be written to enable the import tool to identify the format of the import data, a mail address which is to be notified when the import is done etc. As the server checks all museums’ import directories for available import data every four hours, the object information will then be automatically imported after some time.

EODEM was developed as an application profile for LIDO 1.1, meaning that it builds upon and extends the LIDO standard for exchanging object information. As such, we could simply integrate EODEM into our regular LIDO import. Hence, users who want to import EODEM data need to set the “parser” setting to “Lido” in the configuration file.

Depending on the way a museum forms its inventory numbers, EODEM however may present a problem regular LIDO imports do not. If an inventory number in the sending museum (say, in the import data) equals an inventory number of one of the receiving museum’s own objects, the importer would interpret the duplicate inventory number so as to update the existing record of the inventory number rather than adding a new one. In response to this problem, we added a new setting available in the LIDO parser: prefix_inventory_numbers . By entering a value (e.g. the reference number of the loan) to this setting, the given value will be prefixed to the imported inventory number so as to prevent collisions and allow importing objects safely.

Exporting EODEM data

EODEM data can be exported from musdb the same way any other XML export can be run. It is now listed as an additional export format in the regular export form. The export form is accessible either through the navigation for exporting the whole collection - which obviously makes little sense in the case of EODEM - or on the search results page once at least one search parameter is set. One can thus filter the collection for all objects of a given loan, click the export button in the sidebar, possibly adjust the list of exported fields (e.g. to exclude a field one does not actually want to export, but which is usually covered by the EODEM export). Finally, one will receive a ZIP file containing the EODEM-encoded object information. This ZIP can then be sent to the receiving museum.

To both promote and simplify the exchange of loan object information using EODEM, a shortcut has been added in the sidebar of loan editing pages. Clicking on “Objects (EODEM)” in the export section of the sidebar directly takes one to the EODEM export settings page for all objects of the given loan.

The most simple way to export a loan’s object information in EODEM is by navigating to the loan’s page and clicking the EODEM export button in the sidebar.

In musdb and Beyond

As stated above, we were very late to the party. EODEM has been in development for some years, but we only started seriously looking into it in late 2022 (all credit for developing EODEM thus goes to others; the working group deserves a lot of thanks!).

On the other hand, it turns out that musdb is the first collection management system to actually feature an EODEM import and export in production. For the time being, exchanging EODEM information thus mainly helps in the case of loans between two museums that both use musdb for collection management.

The working group’s page however provides a list of software vendors who plan to implement EODEM imports and / or exports. Many of the big names are already listed there, meaning that a timely implementation of the standard in many of the major collection management systems is almost given.

On the other hand, there are many, many more collection management systems than those ten listed at the time. While EODEM is based on LIDO and thus very simple to implement if a collection management system already features a LIDO import / export, it remains to be seen how many will eventually adopt the it. Hopefully it is many, so that we can improve interoperability and soon enough all save some time when working with loan objects.

info

Image Credits

“Relief zweier sich die Hände reichenden Männer, Seebronn (?)”, CC BY-SA @ Landesmuseum Württemberg

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Upcoming Major Update to musdb

2023-01-04T00:00:00+00:00

Usually the development of musdb (and the other parts of museum-digital software) follows a rolling release paradigm. A new feature is developed, tested, and then distributed. Updates are - usually - not held back. Over the last month, we made an exception, as there will be a lot of new features and a slight redesign to musdb overall.

To allow administrators and users to get acquainted with the updated design and new features ahead of time, a preview follows below. The update will be distributed on January 11th.

A slight redesign

Sometime in late 2020 or early 2021 - when the reworked dashboard was released - we introduced a new, different design to musdb. While the old design had sidebars with a margin to the window border, the dashboard’s sidebar goes all the way to the windows left end. Where the old design positioned all page contents (minus sidebars and navigation) directly on the background, the dashboard features clearly defined boxes for each section of a page.

Bit by bit, pages that had undergone major updates (e.g. the institution-wide settings page and the image editing page) or were newly added altogether (task management; calendar) also received the new page layout. That way, we could slowly phase in the new design and hopefully managed to warm up users to what musdb would look like in the future.

But keeping two different page layouts side by side also comes at the cost of a harder maintenance and (obviously) a less consistent user experience. With the update, the “new” page layout will hence be extended to all pages.

A few additional improvements beyond even what the dashboard suggested have been made however. Sidebars are now used for displaying additional, directly usable information gained from a given entity’s data much more frequently (e.g. a copy-pastable address block of a contact is now displayed on the contact / address book page). And sidebars of editing pages now (almost) always come with an indicator displaying where the user currently is (“collection”) and the ID of the given entity. This hopefully allows users to better reference the entities down the line - especially when integrating musdb with other applications like Nextcloud (see below).

The clearly distinguishable boxes for each section of a page are now also used on pages for adding or updating entities. Again, to allow users a quicker grasp of where they actually are, these pages also come with much more visible headlines.

Editing pages now feature boxes around the main sections of the page. In the sidebar there is a indicator (colored in the color of the current section of musdb) showing that this is an object page.

In other news …

In November 2022 we introduced maps on which the location of a museum or an object’s closer, unnamed location (e.g. a finding spot of archeological objects) could be determined by a simple click on the map. The colleagues from Baden-Württemberg requested to be able to enter geo-coordinates directly into an input field that interacts with the map, as they already know the coordinates they were to select. The maps thus now come with a button on the top right that allows the user to open a dialogue, in which the location can be entered using pastable geo-coordinates.

User-Defined Reports in musdb

Thus far reports in musdb were exclusively pre-written and provided together with the rest of the software. But in the end, museums are often bound by local regulations or already have forms or reports that may be generally used for a given purpose. Users can now define templates for report formats themselves and generate reports based on a search result, an exhibition (and its objects) or a loan (and its objects) independently. This may e.g. be used for automatically generating loan contracts.

To define a report format, one needs to hold the user role “museum director” and navigate to the institution-wide settings page. At the bottom of the page, one can upload a report template with placeholders marking the spots where object information is to be filled in by the system.

To simplify the implementation and improve security on the server side, only plain-text reports may be uploaded. HTML may be the most useful format for textual information with formatting; CSV for tabular information.

Custom report templates and scheduling of timed reports on instituion-specific settings page

Instituion-specific, custom reports accessible in the sidebar of object list

Timed Generation of Reports and Exports

One feature that was often requested, but thus far hard to implement, is to enable museums to generate exports automatically and without user input. This is now possible, both for XML reports and the new custom report formats.

Timed reports and exports are configurable on the institution-wide settings page. Each timed report requires the setting of:

a start date (when should the first report be sent?)
an interval (weekly, monthly, annual)
a selector; usually a search query, written in the query language for searching objects
mail address of a recipient

Note: As musdb is used by many museums together, we had to set some limitation on this feature. If an export file size exceeds 10 MB (which is also a size that many mail servers simply would deny for attachments), the configuration for the automatic report is automatically removed and a warning mail is sent to the recipient.

Literature entries

Probably the aspect of musdb with the most requests for improvements is the handling of literature entries. Almost any request for making literature entries interoperable with other software (e.g. Zotero for bibliography management or library catalogues) essentially requires a field for defining the type of a given literature entry. The same goes for most common citation standards: Within a citation style, the way of citing for books differs from that for articles, for webpages, or for archival material.

We have now added that rather essential field and - as that is possible using the new field - display a BibTeX representation of the literature entry in the new sidebar of literature pages.

New features on literature pages: A field “type” has been added and a BibTeX representation of the literature entry is displayed in the sidebar.

User-specific defaults for adding new Objects

Museums have specializations, and so do people. It’s not rare for people to almost always enter objects of e.g. a given object type, especially if they are working within the context of a project focusing on a given collection (“Digitize all paintings of the museum”). Similarly, all users in the museum likely use the same units for values and measurements of objects.

To speed up the data entry in the case of such fields with unchanging contents, users will now be able set default values for “direct” text fields of an object. Unfortunately, setting defaults for links [e.g. to collections or spaces] and repeatable fields is much harder to implement and not yet covered by this update. Default values for the form for adding new objects can be set in the personal settings.

Take note that defaults can only be set for fields that are displayed on the object addition page. The most basic and generally required fields aside, one can determine which fields are available on the object addition page in the institution-wide settings.

Loans

To be able to better represent the process of a loan in a museum - from the request to the discussion with insurers to the final sending of the objects - we have now added a concise but hopefully reasonably complete checklist of the steps a loan takes within a museum in the sidebar of the loans page. The checklist covers the most common steps within a loan lifecycle and allows simply marking a progress in working on the loan. The last user to update a given entry in the checklist is displayed if a field has been updated at one point to allow following the progress later on.

We were also notified of a very obvious, but thus far overlooked case: Loan requests that are denied. We added the missing field to cover this status of a loan.

Finally, it is now possible to links loans to exhibitions. All loans of an exhibition can be listed together on a tab of the respective exhibition page.

To return to the checklist for a moment: One of the more noteworthy selectable points in the loan checklist is “metadata exchanged”. Adding this point may be opinionated, but hints at the next steps. In November 2022 we added the option to import loan object information following the upcoming EODEM standard. We hope to be able to implement an EODEM export before the update is pushed to the production instances, so that at least museums using musdb can handle loans with minimal duplicate data entry.

Loan pages now allow tracking the status of the loan using a checklist and the new “Loan denied” field.

Exhibitions

As mentioned above, loans can now be linked to exhibitions. A new tab on the exhibition page allows listing all loans that happened in the context of the given exhibition.

The list of objects of an exhibition is now also redesigned. When linking an object with an exhibition, it is now possible to enter the exhibition room in which the object will be displayed. If this information has been entered for the objects of an exhibition, the list of objects of that exhibition will be grouped by the objects’ locations.

Integration with Nextcloud

Keeping with the theme of allowing for a deeper integration of musdb into the actual everyday work of museums, we have added the option to integrate musdb with a museum’s Nextcloud instance. If the Nextcloud integration is activated, a new widget will be accessible in the sidebar of most editing pages (e.g. for loans).

This widget displays a reference ID of the entity (e.g. LOA-000000005 for the loan with the ID 5). If this ID is present in a folder or filename on Nextcloud (say, there is a folder for everything concerning the loan, which will then be named something like “2022 Loan Brisbane [LOA-000000005]”), musdb can identify the folder or file as belonging to loan and list it in the widget. If the ID is present in a folder name, the folder contents will be listed in musdb.

For this integration to work, musdb connects to Nextcloud using WebDAV (unfortunately we needed to use some properties exclusive to Nextcloud’s and likely OwnCloud’s WebDAV interfaces, which makes our integration incompatible to other storage solutions that also use WebDAV like Google Drive). And to connect via WebDAV, it needs the information to get an authorized access.

To configure the Nextcloud integration, one hence first has to set the base URL to an institution’s Nextcloud instance on the institution-wide settings page. This only has to be done once per institution and simply gives musdb the information necessary to locate the Nextcloud instance.
Once a base URL for the Nextcloud instance has been entered, the username and password for Nextcloud (ideally an app token [can be generated in Settings > Security in Nextcloud]) can be entered on the personal settings page in musdb. Once those are entered, the Nextcloud integration is activated.

The Nextcloud integration widget can be found at the bottom left of the page in the sidebar.

Institution and contacts pages

The layout aside, institution and contacts pages have only been minimally changed. A simple but maybe useful small widget has however been added in the sidebars of these pages: An address block to quickly copy-paste the address to e.g. a letter head.

Objects

Finally: Objects. Objects have seen the addition of a lot of new fields, mainly for administrative purposes.

On the “administration” tab of object pages, one can now reserve an object. If an object is currently reserved or will be so in the next week, an indicator will appear in the sidebar.

This object is currently reserved. Hence, a notification is displayed at the top of the sidebar.

For logging an object’s history within the museum, we added a number of logs for:

Damages to an object (Tab: “Restoration”)
Conservation and restoration treatments for an object (Tab: “Restoration”)
Scheduled Checks (Tab: Administration). These checks cover e.g. condition checks, but also audits of whether the object information in musdb is complete. This section comes with a notification that can be sent if a check is upcoming.

Deaccessions can similarly now be covered in musdb.

Again reaching for the most practical applications, we have finally implemented linking an object’s actual / permanent location as a “space” rather than simply identifying it using a text field. This way, it is now possible to search for objects that are not currently in the location their expected permanent location.

We also added some simple text fields that are often present in imports and suggested by the Canadian Heritage Information Network’s software requirements checklist. Namely: sex of the object (for biological specimen), the color of the object, and form of the object.

Damages and restoration / conservation log on object page (Tab: Restoration)

info

Image Credits

Post image: Courtesy NASA/JPL-Caltech.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

New Features at museum-digital (November 2022)

2023-01-02T00:00:00+00:00

After trying a monthly change log once some month ago, we have unfortunately been rather lenient with notifying everyone of new features and updates in the last months. To approach betterment, here there is a list of the updates of November 2022 the form of screenshots. As a very large update is upcoming in the next days, a separate post on the updates of December 2022 will follow tonight.

musdb

New fields

New section for time limits on administration tab of object pages. Special mention should go to the fields “Freeze period” and “Publish object at”. Filling out these fields enables some automation: A “frozen” object cannot be published before the entered date has been reached. This may e.g. be useful with archival material that cannot be published before a given date. The “Publish object at” field offers a counterpart to this. If a date has been entered into this field and the date is reached, the object will be published automatically by the system.

A non-public closer location for an object (which may e.g. be necessary with archeological findings, whose finding spots have no name and are not to be published to not give information to grave robbers) can now be set using a map on the addendum tab.

A number of new fields for noting conditions on how the object should best be displayed in exhibitions, among others, have been added on the “remarks” tab of object pages in musdb.

On the “restoration” tab of object pages, generic fields can be entered with the name of the described feature and the value. Because of the flexible subject of these fields however, they make searching in the fields much harder. Hence, new fields that are applicable to almost all museum objects have been added as easily searchable, “static” properties of an object: Minimum and maximum viable temperature, minimum and maximum viable humidity, and the maximum lux an object may be exposed to.

Other page

It is quite common for users of musdb to only use the same some event types all the time, while not needing many of the other available event types. People working at archeological museums will likely need the “found” event type all the time, while barely ever using the event type “copied by hand”. To directly access those often used types, one can now click the “star” symbols at the end of a line for an event type when accessing the page for selecting the event type of a new event. Favorited event types will then be listed in a bottom sheet on the page.

Batch updating object information

The batch update menu for objects’ visibility can now also be used to set the visibility of publishable fields that are publishable on a field level.

The “batch assignment” menu can now be used to assign spaces, owners, linked loans, and full events (e.g. the creation of objects by a given artist at a given time) to all objects of a search results list.

A “smaller” way of batch updating objects can be used in the object overview by selecting an object by clicking and dragging an object. Now, objects can be selected and updated in bulk. The menu for doing these updates (visible here at the top of the screenshot) now comes with an additional option: “Open in new tab”. By clicking on this menu option, all selected objects are opened in new tabs. As browsers often prevent the opening of multiple tabs in bulk, one may have to allow the opening of pop-ups for musdb in the browser to use this functionality.

Instituion-wide settings and adding new objects

The download button for images in the frontend has been repurposed to enable bulk downloading of all images of an object. While the images are downloaded, the users see an overlay where the museum may display a message (e.g. on how to use the images, or for asking the users to notify the museum about the images being reused in print). The message can be set in the institution-wide settings (available for users of the role “museum director” by hovering over the academy symbol in the navigation in musdb and then selecting the menu option “settings”).

The instituion-wide settings page now also comes with two other new features. On the one hand, users can now be required to select a tag for the object type when adding new objects. On the other, the inventory number suggestion when adding new objects has been improved. It is now possible to generate inventory numbers with variable length numerical components (e.g. ABC-9; followed by ABC-10).

Notifications

The notification framework in musdb has been fully rewritten. Along with that comes the option to specifically subscribe to email notifications only for some types of notifications. To do so, one can navigate to one’s personal settings. A new tab “notifications” on this page allows setting the primary route of notification and a fallback. If the primary route is set to “email” for upcoming ends to loans, the user will immediately receive a mail once the system recognizes an upcoming end to a loan. If the primary route here is set to “Internal” and “Email” is set to be the fallback route, the user will only see a notification on the upcoming loan in the notification overlay within musdb for a week. If the notification has not been marked as read after a week, a mail will be sent.

In other news

The calendar feature (accessible under the puzzle symbol in the main navigation) can now display tasks or make them subscribable via WebCal (thus implementing a “reminder” as had often been requested)
PDF of all linked information is now in A4 and uses a two-column layout
Ukrainian translation
“Simple A5 PDF” now covers inventorization fields on rear side

md:term

If two actors have been joined and one has an old links to the page of the actor now deleted, one is now referred onwards to the new, single actor entry. The same works with transferrals between vocabularies (an actor that was transformed into a tag, etc.).
Ukrainian Translation

md:term is now available in Ukrainian!

Frontend

Ukrainian Translation
JSON-based settings for specific institution pages have been removed
Bulk download of object images
- An overlay with a message from the museum can be displayed during batch downloads (see above)

CSVXML

Almost completely rewritten
The served page now is completely static and all checks and conversions run directly in the browser. This way, no uploads actually happen and the application is completely uncritical to the server’s security. On the other hand, this allows for installing CSVXML as a progressive web app and using it offline.
We also added some explanatory texts did small updates to the design of the page. A footer now links to the source code and offers to refresh all cached contents of the page (this may be useful when visiting the page after a long time, as the whole application is cached in the browser for offline use).
A new check also checks for the file encoding. A warning is provided if the data does not appear to be UTF-8-encoded.
CSVXML is now released open source under the AGPL.

CSVXML has been (almost) completely rewritten).

Importer

While the individual parsers for the different export formats are updated very often, the core scripts of the importer are very stable. November 2022 however came with a large update to these core sections, as more categories of data that had not before been covered by the importer (many of them new) can now be imported:

Contact information (e.g. for object owners; loan partner institution)
Object’s movement log
Minimum and maximum temperature, humidity, and lux of an object
Loans
Events / Appointments

In terms of the parsers, we extended the LIDO parser to cover the new fields suggested by the upcoming EODEM standard for exchanging loan object information.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Monthly museum-digital user meetup (September 2022) / New Features

2022-09-23T00:00:00+00:00

On September 6th 2022, we continued our monthly user meetups. As should best become the norm, we discussed recent new features of the preceeding month and plan a next meetup on the first Tuesday of October (October 4th, 2022, 5 p.m. at https://meet.jit.si/museum-digital-meetup-202210).

A summary of the new features and updates can be found in the following.

Recent Updates and New Features

musdb

Support for Imperial Measurement Units for Separated Measurements

Thus far, musdb only supported metric units for length, width, diameter, etc. It is now also possible to enter measurements in inches and feet. For searching objects smaller or larger than a given number, the number is still expected to be provided in millimeters.

Reworked Institution-Wide Settings Page

Certainly the main focus of our work on improving musdb has been a rather extensive overhaul of the institution-wide settings page. This page is available for users of the user role “museum director” and can be reached through the academy-icon in the navigation.

With the batch of updates, the page gets a new design with an always visible table of contents and rather extensive descriptions of each of the available settings. We removed some outdated and confusing options while adding new ones for the newly added strict modes and inventory number suggestions.

Suggestions for Inventory Numbers for New Objects

On the institution-wide settings page, it is now possible to set a scheme for how the museum forms its inventory numbers. Variable parts are expressed using placeholders:

{no} for a number counting up. For a counter padded to a given number of digits, a number may be added after the place holder (e.g. {no}6 for a 6 digit counter in the inventory number.
{c_sig} for a signature of the collection
{year} for the ongoing calendar year).

If a regular inventory number in the house is, say, “2022-000025”, the scheme would thus be {year}-{no}6.

If an inventory number scheme has been set, an additional suggestion button will appear in front of the entry field for inventory numbers on the page for adding new objects.

Searching for Objects by Their Entry Dates

Similar to how secondary search indexes for measurements greater or smaller than a given value have existed for some time, entry dates, estimation dates, etc. are now made properly searchable through the object search in musdb. For this, dates need to be set in a coherent, maschine-readable form that is greatly aided by the new strict modes.

Strict Modes

See the dedicated blog post on this topic.

If one wanted to delete an object group, one thus far needed to find the object group in the object group overview and delete it from there. There was no option to delete it right from the page of the object group itself.

For a more straight-forward control, we now added deletion buttons on the editing pages of object groups, exhibitions, etc. These deletion buttons can be found in the toolbar on the right of the screen.

PNG Supported When Uploading Object Images

Since object images are almost always photos or scans that feature a multitude of colors and few solid blocks of color, JPG is a much better format than PNG for storing (compressed) object images. musdb hence only accepted JPG images for uploads thus far. Since some museums however have internal rules for keeping their object images’ working copies in PNG, we now allow PNG uploads and then convert the PNG images to JPG during the upload process.

Frontend

License Descriptions Consistently Displayed on Object Pages

Until last month, a description of the metadata license of an object page (to be found at the bottom of the page) was only available for the license CC BY-NC-SA. We now removed the exception and made descriptions and links to the license text available for all available licenses.

Importer

Improved documentation
Further modularize parsers
Added new parsers for EasyDB (work in progress)
Allow importing images by their URLs to locally hosted ones

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Strict Modes

2022-08-25T00:00:00+00:00

In my collection management system, I want to be able to see which objects have newly been acquired by the museum on a given day. Or on any day of a given month. Obviously.

In musdb, the former was possible if imperfect thus far. The latter was not possible at all. This is because data that should be in rather controlled fields with suitable data types in the underlying database - dates, value information, sizes - are stored as strings in the database. On the one hand, this allows us to import also inconsistent or unsuitable data into the fields (especially sizes are often not available is in numeric form). Say, keeping these fields as free text fields is a way to prevent data loss for people switching systems.

On the other hand, keeping them as free text fields prevents us from sufficiently making use of them, especially in providing accurate search options for making larger / smaller searches. Second, it allows different workers at a museum to enter the same information in different, incompatible ways.

A Conservative Solution

In a perfect world, we would simply convert the date field for entry dates into an actual date field in the database and mark the field as one in the HTML of musdb (thus providing a date picker in modern browsers). Users would not expect to be able to enter incompatible data (“ca. 2010”) and imports would already contain well-formatted data.

The world is not perfect.

museum-digital hence needs to be able to do both: Ensure consistency and allow the necessary search functionalities on the one hand, and allowing for inconsistency at the additional expense of search functionalities on the other. And that we can do using two new features.

First, we added additional search indexes for the relevant fields that are of the correct numeric or timestamp data types. Only appropriately formatted information can be entered into these fields, and they using these we can provide the search functionalities we sought.

Since only correct and consistently formatted data can be entered into this index, we try to make sense of the available information. Databases generally expect “0.01”, whereas German users would spell the same “0,01”. This we can automatically correct easily. Dates are a more complicated case. 2022-08-25 may be spelled that way. But even in a given language there are often multiple ways to express a date. 2022.08.25. is Hungarian, 25.08.2022 is German, but so is “25 August 2022”. Gladly we already have a rather capable auto-correction function from auto-correcting new time entries in our controlled vocabularies.

Still, there are some values that cannot be made sense of. If you expect a date, and someone gives you “ca. 2020”, there’s nothing really to work with. The search indexes thus only contain any values for an object, if we actually were able to understand whatever had been entered into the respective free text fields.

For new museums that do not regularly use imports, we added a second new feature: strict modes. In the institution-wide settings, there are now options to enable strict modes for values statements, dates, and sizes. If the respective strict mode is enabled, the entry fields will be changed into the correct types. The browser’s date picker thus becomes available for date fields, and the browser will prevent users from entering non-dates. Similarly, values statements and sizes will be marked as numeric, making the browser prevent non-numeric inputs.

If the strict mode is enabled, one can be sure that extended search functionalities like smaller / larger or before / after searches can be used on all objects that have had their respective fields filled after the strict mode had been enabled. On the downside, it will now be impossible to see older, incorrect values in those fields. Upon submitting the forms, the data will be lost.

Enabling the strict mode thus only makes sense for museums starting to use musdb for collection management. We however plan to provide checking and migration scripts eventually to allow museums to keep their existing data entered before enabling the strict mode without risk of data loss when enabling the strict mode later.

Update woes

We pushed the update yesterday night and unfortunately ran into a problem filling the updated search indexes in case of certain malformed dates: Our auto-correction script for getting sensible time information from strings thus far did not contain a safeguard against time entries like 2022-14-14 - that is, if value for a month was accepted as a two-digit numeric value, but larger than 12 (in the example it would be the 14th month, which obviously does not exist).

Since the update required a full regeneration of the search indexes, the attempt to enter such malformed dates fully halted the update (this is on purpose with internal scripts to force us to immediately fix such problems). Some instances of musdb (mainly Saxony-Anhalt, the Rhineland, Berlin, Budapest and Baden-Württemberg) thus ran with incomplete search indexes for the object search in musdb from about 4 a.m. until noon. The error is now fixed.

tip

tip

This content is licensed under a Creative Commons Attribution 4.0 International license.

Referencable Maps in the Frontend and in musdb

2022-07-22T00:00:00+00:00

Development on museum-digital will be concentrated on more internal functionalities in the next months, but some small improvements continue to be made in the public frontend. Like today: We have now implemented the option to directly link to a position on a map view for object search results both in the frontend and in musdb.

To allow this, navigating or zooming to a given position within the map now automatically updates the URL to include the new selected positioning on the map. If one thus copies the URL from the browser and shares it to a colleague, the colleague will be able to open the map at the same position.

Additionally, we have been able to fulfill an often-voiced request for the map view: a legend of the many available event type icons on “objects on map” pages in the frontend is now accessible by clicking on the new “Question Mark” button at the top right of the map.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Managing object information using the musdb API

2022-06-20T00:00:00+00:00

The public API of the frontend of museum-digital has long been in use - for example for embedding objects directly from museum-digital in a given museum’s website. The API is stable and well-established.

In musdb, our inventorization and museum management tool, however, the situation is more complicated. On the one hand, musdb is simply much larger in terms of the offered functionality. Many features have no public equivalent, while the whole question of updating the database is not present in the frontend. The tab-based user interface and the unfortunately often still rather tight coupling between user interface and loading of data from the database has been hindering the development of an API for musdb. A simple API design like the one in the frontend, where you can just add &output=json (or ?output=json) to a page’s URL to retrieve the page’s information in a machine-readable format is made largely impossible by that very architecture. Just adding &output=json to a URL in musdb provides that machine-readable information in an astonishingly high number of cases, but often it is incomplete or otherwise of limited use.

Finally, it is especially musdb where a well-designed API may become interesting to have in the future. It would be great if museums could use their inventory app of choice besides musdb - and any update to public objects would automatically be mirrored in musdb and the frontend of museum-digital in the background. Currently, the same objective can almost be reached using imports, but imports are limited to updates of the same type and do not happen in real-time.

Off to a Good Start: Updating Object Information Using musdb’s API

A first step towards a dedicated API for musdb has now been taken. For the start, all fields directly relevant to the management of objects have been made available using a new public (though obviously authenticated) API for musdb. Further functionalities will be added when the need arises.

Obviously, starting over and writing a whole new API offers a chance to implement things cleanly. Adding the new API required us to refactor the code considerably - and that in turn provided a good opportunity to write unit tests for all functions relevant to the API. The API is thus ensured to stay stable for the foreseeable future (extensions notwithstanding).

In designing and documenting the new API, we took the opportunity to write the documentation using the OpenAPI standard. A desciption of the API in machine-readable JSON can be accessed at /musdb/api in each instance, e.g. at https://hessen.museum-digital.de/musdb/api.

Using OpenAPI also opens the way to all those tools that build on OpenAPI in general. Hence, a human-readable interface for reading about and testing the API could be generated using Swagger UI (thanks!) and is linked in the German-language handbook. Since each musdb API is linked to its respective regional instance, the “try out” buttons do not work in the handbook, but working Swagger UI instances describing the musdb API can be found linked in the “tools” menu of the navigation of musdb.

Next Steps

Even though object information can now be read and updated using the musdb API, there are a lot of features not yet covered. Even a simple task like listing the collections available to the API user is still missing. As this hinders the finding of IDs for linking with objects, it is a natural next step for implementation.

Working with OpenAPI on the other hand has been a pleasure. We are looking forward to eventually (hopefully sooner than later) being able to provide an OpenAPI description of the frontend API as well.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

files.museum-digital.org: Archive for Documents About museum-digital

2022-06-07T00:00:00+00:00

Over the years, many presentations have been held about museum-digital. Articles have been written, and so have tutorials and other documents. To not let them fade into obscurity on the hard drives of their authors, we have now set up a document archive for such files: files.museum-digital.org.

For the start we have compiled primarily presentations there along with those logos currently in use at museum-digital. Going over the archives also presented a welcome chance to explicitly license most of the files under a Creative Commons Attribution 4.0 license.

Imports can now be triggered by users

2022-06-04T00:00:00+00:00

Until May 2022, the import of new data to museum-digital was always limited by the need to involve a member of the development team. Not least, because users had no option to upload potential import data in bulk. After many years, we have now finally opened up a way for users with access to musdb to import data on their own.

The most important precondition for this was, once again, the option to upload generic data to the server. This can now be done using a newly added WebDAV interface. WebDAV is a protocoll that can be used in a roughly analogous way to FTP, but it is built on top of HTTP (the main protocol used for accessing websites). Support for the protocol is hence rather good - even Windows Explorer can mount WebDAV network drives - and it should not be blocked by the firewalls of larger museums.

How to import data using the new WebDAV interface?

First, one needs to generate login data to the WebDAV drive for the respective institution (this is always the institution one’s account was created for, even if one has access to multiple museums’ data. To do so, one needs to login to musdb and then navigate to one’s account settings. Here, there is a new tab available below the table for basic login information: WebDAV. Once that tab is opened, a short explanation becomes visible alongside a button “Generate WebDAV password”. Upon clicking that button, a password is automatically generated the login information is displayed in a dialogue overlay. This login information should then be stored in a safe place (at best, a password manager), as it is not accessible from the server anymore at a later point in time.

Once the button “Generate WebDAV password” has been clicked, a dialogue with login information for the WebDAV interface opens

Using this data, one can now proceed to connect to the WebDAV drive using one’s WebDAV client of choice. For first attempts to try out the import process, one can simply use a file manager (as stated above, even Windows Explorer can handle WebDAV), but if one repeatedly wants to work with the import tool, it is advisable to get a dedicated WebDAV client like WinSCP. Note that, depending on the client, it may be necessary to change the protocol from davs:// to https://.

Uploading data for imports and configuring imports

When one accesses the WebDAV drive for the first, two folders and two files will be listed:

.
├── import_config.sample.txt
├── IMPORT_IMG
├── IMPORT_XML
└── README.md

The meaning of **README.md **should be evident: here you can also read about the expected folder structure and the next steps required for running an import.

Uploading import data

The folders IMPORT_IMG and IMPORT_XML are where import data should be uploaded to. Images as well as other media files go to IMPORT_IMG; metadata files - irrespective of their format - can be uploaded to IMPORT_XML. It is important to note, that there should be no further subdirectories created within the directories - the import scripts expect a flat folder structure.

Configuring the import

Finally, the server needs to be told what kind of import data is present here and that there is an import to be done at all. The file import_config.sample.txt can be used for this purpose. Upon opening it in a text editor (e.g. Notepad; not MS Word), one can see all the possible import configuration settings in lines that are to be completed.

Required for running the import are only a mail address and the name of the parser (from the user’s perspective: the expected import format) to use. The mail address is required to enable the server to give feedback about the success of an import or any error appearing when importing (the occurence of any error automatically halts the import process). Note that the mail address needs to be one of a user in musdb who holds the user role “museum director” or “administrator”. A detailed list of the available parsers, complete with descriptions, further links and possible parser-specific settings can be found at the end of the file.

All other settings besides the mail address are optional. One may e.g. set objects to be automatically published upon the completion of the import or one can link all the imported objects to a collection, overwriting possible collection links in the import data.

Once the import configuration has been completely filled out, the file should be saved and renamed to import_config.txt. The existence of a file of this name signals to the server, that an import should take place here.

The import runs

The server has a timed script, that checks all WebDAV drives once every four hours to see, if any import data exists there (signaled by the existence of the file import_config.txt). If that is the case, the import will be run.

If the import completes without any problems, the import data and configuration file will be moved to a new directory IMPORTS_SUCCESS and a notification will be sent to the mail address named in the configration file. Eventually, the import data will be removed from the WebDAV drive.

Any problem on the other hand immediately leads to the import being halted and the import data being moved to a folder IMPORTS_FAILED. Similar to successful imports, a notification will be sent out to the mail address from the configuration file and to the development team. The selected parser or the import data then need to be adjusted and the import data moved back to the main IMPORT_XML and IMPORT_IMG folders before one can try importing again. Potential problems can range from incomplete data to the existence of fields not yet covered by the respective parser in the import script.

Necessary adjustments to the import tool

To enable users to safely import data on their own, the import tool had to be substantially remodelled. In general, the import tool consists of four types of scripts:

Representation of the basic data types available for imports (e.g. objects, events, images, collections)
Writers, that handle writing the imported data to the database in bulk
Parsers, that handle the reading of import data of a given format and translate values from the import data format to the expected data types in museum-digital
Frontend scripts, that provide e.g. the command line interface

Thus far, only the data types and writers had been consistently written using an object-oriented approach. Enabling users to set parser-specific settings required us to refactor the remaining code to similarly use object-oriented programming. As a positive side effect, this allows us to test the parsers much more easily, so that errors in rarely-used parsers arising from updates to other components or the programming language itself will be noticed and solved much more quickly than it would have been the case before.

On the other hand, this refactoring and the new requirement to at least add integration tests for all parsers led us to drop support for some rarely used parsers, for which we could not find example data anymore.

Who may profit from the new import option

While the present update allows users to upload import data and run imports themselves, it only allows limited options to adjusting the parsers themselves. It is thus museums who often import data of the same format who can profit most from the new format.

Museums who have import data in a format that is not yet covered by the available parsers will still need to go with the old way of importing and simply write a mail.

P.S.: An up-to-date description of the import process can be found in the German-language handbook.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

IIIF and museum-digital

2022-04-03T00:00:00+00:00

Many have seen it and the feedback has been consistently positive thus far: For many objects, museum-digital can now offer a revamped image viewer with improved features, such as a much better zoom function. This is made possible by the image viewer Mirador and our IIIF API.

The collection of paintings of the Stiftung Preußische Schlösser und Gärten as a IIIF Collection in Mirador’s demo instance

The fact that museum-digital offers an IIIF API at all, and that a IIIF viewer ist now used at museum-digital, is noteworthy, but quick to report. Especially, since both IIIF and Mirador are very well-documented on their own websites (IIIF und Miradorr).

But if one looks into it deeply, it becomes obvious, that the IIIF-based image view is not available for all objects.

Limitations of museum-digital’s implementation of IIIF

IIIF offers a standardized API format for retrieving image metadata through an appropriately formulated URL, but it also enables the retrieval of image files that are cropped or rotated accordingly on the server side. This server-side image editing is useful for making IIIF-based applications run on low-powered devices and it provides easily referencable URLs for specific sections of an image. But it also means, that all images made available through the IIIF API should be available on the server itself in a high resolution.

At museum-digital, museums can upload their images - but since some museums have their own image servers, museums can also link images from external sources. In this latter case, museum-digital only stores previews up to a width of 500 pixels. These previews are however not usable for an adequate server-side image editing as required for implementing IIIF. Thus we offer IIIF manifests only for objects that exclusively have locally stored images.

We currently also implement IIIF only in version 2.1, which does not yet support other media types such as PDFs, videos, audio files or 3D representations of objects. Hence, objects that have any linked representations of these types are also excluded from the IIIF API.

IIIF Collections

One of the nicer features of IIIF are IIIF collections, with which many objects / manifests can be grouped into a collection. For implementing the minimal requirements for an IIIF API two years ago, we had thus far already set up the relevant files for IIIF collections - but we didn’t fill them with contents.

Now, IIIF collections at museum-digital represent collections and object groups. The limitations described above persist however: IIIF collections at museum-digital only list objects with exclusively internally stored images.

IIIF collections can be accessed - as always - by their respective URL. E.g.:

https://brandenburg.museum-digital.de/apis/iiif-presentation/collection/c217

This URL represents the collections of paintings of the Stiftung Preußische Schlösser und Gärten. The noteworthy section is the last: c217. Since the IIIF collection API at museum-digital represents three different types of groupings at once - institutions, collections, and series / object groups - the first character of this URL section represents the respective type of grouping. The ID of the entry follows.

An “i” in front of the ID thus represents institutions, an “s” represents series. One example of a IIIF collection for an object group may thus be:

https://hessen.museum-digital.de/apis/iiif-presentation/collection/s20

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

"Smarter" Entry of Links to Vocabularies in musdb

2020-09-21T00:00:00+00:00

Many imports of data confront us with Places like “Berlin ?” and times like “ca. 1328” konfrontiert. The import tool of museum-digital has been able to handle such entries for quite a time: “Berlin ?” is recognized to mean that the place is actually “Berlin”, but that the entry is not made with complete certainty.

In a similar fashion, the vocabulary control tool of museum-digital, nodac, has been able to parse and normalize time names for some times. While the canonical formulation of e.g. a single day is the German date format DD.MM.YYYY, similar times like “15. Januar 1920” (German) or “1920. január 15” (Hungarian) are entered often. These would have been transformed into 15.01.1920 upon the push of a button and then translated to many languages upon the push of another.

In line with some larger imports, we have now improved these and moved them into a dedicated module, that are now also used in the general input tool, musdb.

Recognizing Uncertainty and Normalizing Times in musdb

musdb can thus recognize uncertainty based on a given list of indicators of uncertainty. In the case of times, these are e.g. “um “ (“about” in German), “wohl um “ (“likely about”), “circa “, “ca. “ oder “ca “ in the beginning of an entered time term and “(?)” oder “?” at its end. If one of these indicators is present, the time name is freed of the indicator and the entry is saved as uncertain. The same works with links between events and actors and places.

Entering a place with an indicator of uncertainty …

… the indicator is stripped off the entry and translated into uncertainty of the reference.

In a similar fashion, many entered times can be automatically normalized and cleaned. Thus far, entries in German and Hungarian for single days and months (15. January 1920 and January 1920) can thus be automatically parsed and cleaned.

Entering a normalizable and uncertain time ….

… results in a normalized entry. The reference to the event is stored as uncertain.

Together with normalizing the time term, the entries can in many cases be automatically translated. This is especially important for times before 1000 CE, where an indication of whether they are BCE or CE is often necessary for a quick understanding. This is thus far implemented for time spans before 1000 CE: “312 - 315” is automatically translated to “312-315 n. Chr” in German, “312-315 CE” in English and “西暦312年から315年” in Japanese.

Batch Publishing or Hiding by Object Selection

A small improvement on a sidenote, we have now added batch publishing and hiding of objects to the options available through the object selection tool in musdb. To use it, the objects to be manipulated can be selected by first dragging one with the mouse and then clicking on the others. Finally, the respective menu option at the top of the page (red in the screenshot can be pressed).

Menu options for batch hiding or publishing can be found at the top right (in red borders).

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

More Stability and Some New Features

2020-09-15T00:00:00+00:00

Over the last months, we have not been inactive in further developing museum-digital. We mainly worked to improve the legibility of our code under the hood - say, most changes will hopefully not be noticeable, but an improved speed of especially searches for objects may be a positive and visible effect.

Either way, we have also worked on some new features: A dedicated area for markings (stamps, watermarks, etc.), one for references and reception (e.g. the publication history of an object), and entering uniform information for the field “object type” has gotten easier. Most importantly, we have added some new event types to better meet the requirements of provenance research.

Markings

Especially for art historians, a precise recording of information on different types of markings, that have been added to an object for indicating authorship and ownership are essential. Be it stamps, watermarks, signatures. Thus far, information on these could only be added using the unspecific field “Inscription” in museum-digital. The new “markings” area (to be found at the bottom of the tab “Addendum” in musdb) aims to tackle this problem. The term “markings” is deliberately chosen to be wide in meaning to also emcompass logos and other types of markings that are more relevant for e.g. technical collections.

Markings in musdb can be found at the bottom of the “addendum” tab.

If a user of musdb wants to add a new marking, they can scroll down to the bottom of the “Addendum” tab and find an accordion button. A click on this button toggles open the area for adding new markings. First, the tye of the marking and its position on the object need to be selected. A transcription must also be provided to safe. Additional information to add are the creator, a possible mentioned time on the marking, its size and a note.

Creators and times are drawn from the centralized controlled vocabularies for actors and times respectively. Hence, they can be entered by tiping the given term and then selecting the correct value from a drop down menu appearing as one types. If the term does not exist yet, it can be added by clicking on the “+” symbol right before the text field.

Once a marking has been entered, it is public for now, even though we consider adding an option to hide markings even when their object is public.

Markings in the frontend of museum-digital (at the top left of the screenshot).

Reception and References

With “reception”, there is a whole new tab in musdb for linking objects with sources. These sources come from a new controlled vocabulary for sources, which is used together by all institutions and runs on a similar level as actors, times, places, and tags. Reception now describes all instances, in which an object is referenced in a source - e.g. if a painting was printed in a book, or a photo published in a newspaper. Similarly, the discussion of a custom-made car in a car magazine may be recorded here.

The counterpoint to this is the “is reference” area just below reception: This can be used, if the object itself references a publication. E.g., a painting may be drawn after a scene from a book.

Reception and references in musdb

Like markings, reception and references are by default public.

References and reception are shown at the bottom of object pages. A click on a source name opens the source page, where one can find all objects referencing the given source.

Object Types From a List

museum-digital traditonally treats object types as a free text field, while many - if not most - museum databases provide controlled lists of available object types. We use keywords / tags to provide the same functionality, but it surely makes sense to also have some level of uniformity in the field “object type”.

Hence, there is a new aid in adding uniform object types when adding a new object in musdb. As one types in the field, a drop-down list of similarly-phrased keywords appears. Clicking on an entry from that list enters its value into the “object type” field and (if one is adding a new object) automatically links the object with the selected tag.

JSON-LD Metadata for Image Licenses

Another minor improvement is that single image pages now feature JSON-LD markup describing the respective image license. This enables especially Google, to provide users with a more detailed view on how to attain the license to use a given image file.

Google Images shows a small “Licennsable” marker at the left of an image. Clicking on it shows more information on the image license at the right.

Importantly, images from museum-digital now also appear in Google Images if one opts to only view Creative Commons-licensed works.

Events: New Event Types and References for Events

The probably most important change are the changes made for entering events in museum-digital. One the one hand, important missing event types have been added. These are:

Assembled (to be used for objects, that have been assembled from parts. E.g. a car)
Auctioned
Bought
Owned
Sold
Restorated
Damaged
Destroyed
Lost
Edited (for books and publications, that have an editor)

As can be seen from the list, most of the new event types are related to provenances (for all museums that want to use these event types: Please (!!) ensure that you have cleared the necessary rights to publish this information, as events are public). To make events at museum-digital more closely meet the requirements of provenance researchers, a missing piece were also sources for events, which can now also be added.

To do so, the addition of sources for events must first be activated on a museum-level. To do so, a museum director or user with similar permissions must first navigate to the “institution-wide settings” page in musdb, and activate the event sources there.

Once they are activated, an accordion button appears on event editing pages (accessible by clicking on the event type of an already added event). Here, sources can be linked using a drop-down menu similar to the ones used in the features described above.

Finally, with options to add event annotations and sources, there was an increased danger of cluttering the object page in the frontend of museum-digital. Hence, we redesigned the display of event annotations to now appear in a tooltip - if an event has an annotation or a source, an “i”-symbol appears next to the respective element for the event type. Hovering over that element makes sources and annotations visible.

Beitragsbild: Caravaggio: „Der ungläubige Thomas“ (CC BY-NC-SA @ Stiftung Preußische Schlösser und Gärten Berlin-Brandenburg / Murza, Gerhard)

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

OpenSearch: Search museum-digital directly from the browser's search bar

2020-06-05T00:00:00+00:00

OpenSearch is an open web standard for describing search functions of web services. If a website supports it, the browser will take note and offer the user to install the website as a search machine.

One especially nice aspect of it is, that OpenSearch can be implemented in just a few lines of code. It was thus easy to bring to both the frontend of museum-digital and musdb, to make searching objects right from the browser’s search bar possible, just as you can quickly search Wikipedia or Amazon on many browsers by default.

At the top right, next to the search (loupe) symbol, there appears a green “+” …

… clicking on the search symbol offers installing the instance of museum-digital as a search provider.

Since musdb does not feature a simple, unqualified search function (one always needs to specify, whether one searches for a tag, place, etc.), OpenSearch offers using the full text search on musdb.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Using Textblocks in musdb

2020-02-16T00:00:00+00:00

museum-digital has a lot of “hidden” features; hidden on purpose to not blur the users’ focus. One of these is the text blocks features in musdb, our inventorization and data input tool. Thus far, the text blocks had only been described in a presentation from mid-2018 (in German) - hence this blog post.

Text blocks in musdb are managed on a per-field basis. The first step to using them is thus to select an input field, for which the text blocks are to be used. To do so, a klick into a given field is sufficient.

Klicking on an input field triggers an indicator at the bottom right of the screen to appear. This indicator features basic information about the field and the entered contents - e.g. whether it is a required field or the relation between entered characters and the maximum number of allowed ones. At the bottom of the indicator, there are however also buttons that lead to field-specific functionalities.

The field indicator is opened at the bottom right of the window once an input field has been activated (e.g. after clicking into the field).

A click on the “repeat” symbol (or pressing CTRL+r on the keyboard while an input field is activated) opens a sidebar at the window’s right. This sidebar consists of a list of all available text blocks and - at its bottom - an input field for entering new text blocks.

The textblock sidebar lists all available text blocks. The bottom of the sidebar contains an input field for adding new textblocks to the list.

Clicking on a text block or pressing its corresponding number inserts the text block into the previously selected input field. Inserting new text blocks doubles as a deletion and sorting function: for each input field one can only store up to nine text blocks. If a tenth text block is entered, the oldest one is deleted and the numbering adjusted accordingly.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

"Select all" in Object Selection Mode in musdb

2020-02-12T00:00:00+00:00

While most of the weekend had us preoccupied with doing a long-thought-over plan, we managed to squeeze in a minor improvement of the object selection mode in musdb: A “select all” option similar to how - for example - all files are selected in the Windows Explorer. To use it, press <CTRL+a> while in the object selection mode in musdb.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

"List Results" Tool Improved

2020-02-03T00:00:00+00:00

In response to suggestions of colleagues from Brandenburg, we have removed some restrictions from the “list results” function of musdb, which also provides the export option to spreadsheets. Since we were at it already, we improved the performance as well.

The main visible new addition to the page is a button for exporting all available columns of the page to a spreadsheet. It can be found in the top menu, just to the right of the usual button for exporting all currently selected columns to a spreadsheet.

While implementing the button, we ran into two roadblocks. First, we realized, that thus far it had only been possible to export tables with up to 26 columns (since PHPSpreadsheet, like Excel, names columns as A-Z, and begins at AA after). We fixed this bug and soon ran into a performance problem. While most of musdb had been optimized strongly for an improved performance in 2018, the “list results” function had not. Large selections of objects to list thus came to the limits of resources we allow the script to use. Through optimizations of the run database queries, we fixed this problem as well and made the tool much more performant overall. A nice side effect of these reworked database queries is that the list can now also be properly sorted by actors and places.

Next, we worked on improving the user interface a bit. On the one hand, the selection menus for changing which columns are to be displayed are now shown as table headings in their respective column. On the other, tables with more than 10 columns are now horizontally scrollable.

Finally, we added keywords to the list of object information that can be listed using the “list results” tool.

At the top right, the button for exporting all available columns as a spreadsheet can be found.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

"Not More Than 20": Maximum Number of Simultaneous Uploads

2020-02-03T00:00:00+00:00

A lighter is an object. With one, or maybe two perspectives - images. A book is an object. With potentially hundreds of pages - images.

To simplify the management of objects with many images musdb offers the option to upload multiple images at a time. This option is however limited by global server settings like the maximum size of uploads and the maximum number of files that can be uploaded in one go.

To understand the problem, it’s important to know that most tools of museum-digital are written in PHP. Unfortunately, PHP has a tendency to gloss over potential errors and continue without enforcing that the problem to be solved either by the programmer solving a bug or by the user not doing a certain action.

The maximum number of files that can be uploaded simultaneously is a flagrant case of this: Rather than inform the user of their overstepping of the set upload limitations, PHP accepts files until the number of allowed files per upload is reached. Any files after are simply ignored.

Say a user attempts to upload 30 pages of a book in one go. The first 20 will be uploaded correctly, while the last 10 will be skipped. In the worst case, the user doesn’t even notice the missing data. In a better case, they are left confused.

We have thus added a warning that stops the user before submitting the upload form when trying to upload more files than allowed.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Multilinguality in md:term

2020-02-02T00:00:00+00:00

Since last weekend, the last publicly accessible page of museum-digital has been made fully multilingual: md:term. md:term is the central public frontend for our controlled vocabularies of actors, places, tags, and time terms.

The probably most important part of md:term is its API, which is also - for example - used by the “graph navigation” and is available for external use as well. Since we - like anywhere else - do not require authentication for accessing the md:term API, we don’t have a full list of API consumers which we could contact about API changes. Hence, we hope that this blog post will also alert them to make necessary changes.

How then, is md:term made multilingual? And how does it express itself in md:term?

Months ago, we implemented the multilinguality in our controlled vocabularies by adding an additional translation table, which means that there is the base entry, which is entered by the museums. The “norm data editors” then enrich the data and also attempt to add translations to a separate table by fetching data from Wikidata. If translations in the user’s language are available, they replace the “base entry” when outputting the data.

In the API, the base entries are replaced as well. If they are used, it hence makes sense to explicitly state the wanted language of the entry. To fetch German language information on Berlin easily, one should thus query <a href="https://term.museum-digital.de/md-de/place/61/json?lang=de">https://term.museum-digital.de/md-de/place/61/json?lang=de</a> instead of <a href="https://term.museum-digital.de/md-de/place/61/json">https://term.museum-digital.de/md-de/place/61/json</a> (which, before multilinguality, would have been the appropriate URL).

Alternatively, there is a new section in the JSON API: langs. All available translations can be found there listed by language.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

museum-digital:blog - Development

Braving the Waves. Surviving in Times of AI.

Celebrating AI Scrapers?

From Each According to Their Ability: Load Shedding

Numbers Games: Logging Strategy

Fairness, FAIRness

Pre-Release of musdb's New UI

New Architecture, New Possibilities

Automation via Webhooks

Design Overhaul

List of Features Not Yet Implemented

Features That Will Certainly Be Implemented, But Are Not Yet

Whole Pages

Navigation

Misc Tools

Features Available in Multiple Sections of musdb

Object Pages

Institution pages

Collection pages

Series

User Settings

Contacts / Address Book

Spaces

Exhibitions

Vocabularies / Backgrounds

Features That Are Implemented But Certainly Need to Get Better

Features That Will Be Removed

Features to be Discussed

What Does That Mean For the Roadmap?

musdb Reimagined - Update & Release Schedule

Recap: Scope and Reasons of the Rewrite

A Sneak Preview

Navigation

New Filter Options

Display Modes of Overview Pages

Batch Editing On All Overview Pages

Interactive Overviews

Timeline

musdb Reimagined - A Preview

Reasons

Technical

User interface

Preview

State of Development, December 2025

musdb

Object overview

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

List of Terms of Use

Imports

Limiting Report Mail Size

Dry-run Mode

Dashboard

Annotations for the Vocabulary Editing Team

Smaller Updates and Bugfixes

Frontend

Reworked Default Image Page

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

Image Search

Batch Export of Object Metadata / OAI

Smaller Updates and Bugfixes

nodac

Trimming.

A Limited Goodbye to IIIF & Server-Side Image Manipulation

Communications

Cleaning Out Our Closet

PDF Generation

Failed Search Pages

Timelines

Search: Cleanup, Image Search & Checking Entity Existence Early

The Current Situation

Updates, AI scrapers, and Resilience

Background

Scrapers

PHP

Upgrading PHP to version 8.5

PHP-FPM settings

Code

Scaling in Software

Frontend