We maintain that hypocrisy is not for us. I personally have been involved in discussions about the FAIR principles from 2015 onwards. And in 2015 they were already running under that label for a while. As a formalized document, they’ve been around since 2016, but disregarding the label, the principles and surrounding discussions go back, I guess, at least to the 1990s.

In the cultural sector, with many institutions being primarily tax-payer funded, we want to provide our data and services in a Fair, Accessible, Interoperable, and Reusable manner. Relevant here are primarily accessible, interoperable, and _reusable

Realistically, accessibility refers to two things: Immediate, universal barriers like a requirement for authentication should be avoided where possible. On the other hand, user-specific barriers should be reduced as much as possible as well: Human-readable pages should e.g. be designed with sufficient contrast.

If one sees other services and machines as users, which may be reductionist but is logically much more coherent than doing otherwise, then interoperable refers simply to a special form of accessibility: As data should be provided in a way that as many human users can use as possible, it should also be served in formats that machines can read. Ideally in ways that machines can read without further adjustments – following open standards.

Reusable then builds upon these. As the data is now freely accessible by others, setting appropriate licenses allows for the creative reuse of the data.

Ten years after the FAIR principles were formalized, we finally have someone who reuses our data on a massive scale. We should rejoice. Unfortunately all our previous conceptions of how that reuse might take place turned out to be wrong.

Celebrating AI Scrapers?

AI scrapers flooding web services, and especially larger, well-established infrastructures with requests has been an ongoing story for some two years now. Last year museum-digital was met with a first large wave of AI scrapers. See the previous blog posts (1, 2, 3).

On the one hand, using our data for training AI models is undoubtedly reuse. It is even productive reuse: It’s a small contribution to the training of technology that, at its current quality, sounded like science fiction 10 years ago.

On the other hand, this type of reuse takes place without attribution. AI scrapers work based on scale; Machine-readable APIs are uninteresting to scrapers. They suck up as much data as possible, as quickly as possible. Any page-specific adjustments, such as finding and using the API, would reduce the speed of scraping. The requests are so numerous, that they routinely crash servers. Last but not least, those prominently benefitting from the AI hype are far from sympathetic people.

But principles are principles.

It, again, cannot be denied that what AI scraping is eventually aimed at is reuse. Even though they use HTML rather than the APIs we lovingly carved for them, they interoperate with our services in the way most accessible to them.

The problem is, that they are not playing fair. And if the previously unlikely number of requests leads to servers crashing and shutting down, then all principles were for naught. A service that is providing no data at all cannot provide them _FAIR_ly either.

At museum-digital we have seen the new requests as an opportunity to improve our software. If it can withstand the onslaught of thousands of bots a second, it is confirmedly battle-tested and stable. Improving our publishing software on the other hand also meant cutting out overly resource-hungry functionalities. Last year we dropped most publicly available, server-side PDF generation capabilities, we moved the IIIF APIs to an separate configuration that is specced to be able to fall over without impairing the rest of our services. We improved the parsing of search terms and introduced rate-limiting (how many requests a single IP can perform for a given time).

The new wave of AI scrapers since July is even bigger than last year’s. Those improvements were not sufficient to keep our primary server, which hosts the production databases, stable.

From Each According to Their Ability: Load Shedding

We continued on last year’s course however, analyzing the scrapers’ requests, their influence on server stability (which is to say, which requests were especially resource-intensive) and adjusted our setup accordingly. Note, that we did not scale up: museum-digital runs on exactly the same bare-metal servers today as it did two years ago.

In analysis, we identified two types of especially resource-intensive browsing behaviors that are almost entirely limited to scrapers (as well as some very engaged researches, to whom we express our regret for now regularly banning them):

• Scrapers follow links on search pages, especially the facet search, and end up combining more and more search parameters. It is unlikely that a human would perform a search for objects that “Are related to Berlin, and related to Germany, and related to Institution X, but not related to Institution Y, and also related to a time between 1990 and 2000.” For an automated scraper it is normal behavior.
• If a published object page has been updated, a snapshot of its current state is automatically saved to provide an archived version of the page at a given time. This is thought mainly for researches to be able to cite a given state of the page. In everyday operation, archive pages should be barely used. They are also delisted from search machines. But saving snapshots again and again leads to the existence of a large number of links and separate pages scrapers can scrape to no benefit to anybody.

Both functionalities are legitimately useful but scale badly.

We hence introduced load shedding: Whenever a search is performed or an archive page is accessed, the current server load is evaluated. If it is above a certain threshold, the server does not perform complicated search queries (depending on how high load is, it is restricted to a minimum of three search parameters) or does not load the archive page. Instead a warning is presented, that the given action is currently unavailable and a custom HTTP error code is sent. If the same IP causes that error code to be sent twice, the IP is blocked for a while.

As of the time of writing writing, we have thus blocked a total of 131,014,530 IPs in two month, with 5,802,771 being currently blocked.

This approach follows our principles: We try to be fair with everybody. But if users are not fair and do not follow advise, we don’t feel obliged to continue playing fair either.
Unfortunately this approach works on the level of blocking whole IPs. As “residential proxies” – back in the days we called them botnets infecting home routers – have become an ever larger problem, it is not unlikely that actually interested, normal users also unknowingly host an AI scraper at home. I fear I’ve already encountered one such case, were a colleague could not access museum-digital for seemingly no good reason from her home network. Otherwise, it is surprisingly effective – the number of requests has barely been reduced, but our services are essentially stable.

The only crash since happened the day before yesterday and was caused by the combination of bots and an error in our controlled vocabularies – the latter of which is entirely our fault.

Numbers Games: Logging Strategy

After the above-mentioned actions had effectively returned stability to our publicly accessible portals, one issue remained for internal services: File uploads were incredibly slow. It turned out that the large number of requests being permanently logged overburdened the SSD. We have hence disabled the general access logging, restricting ourselves to logging requests that either cause errors or trigger slow responses. This reduces the ability to react well-informed to new waves of requests and actual attacks. Thankfully those also commonly trigger actual errors we can still identify.

Fairness, FAIRness

With these actions we have again averted the need to use more drastic strategies that would limit accessibility to users – human and machine alike – like blanket bans on whole regions of the world or the installation of software such as Anubis. And, again, we have not yet needed to improve our hardware (and pay the price for that), even though that might be in order eventually. It might be wise for other reasons as well.

A holistic picture of one’s setup – software, hardware, systems administration – and the capability to think and act upon these together helps a lot.
But, if anything, hosting in a FAIR, principled way still works in these trying times.

(Post image generated using Anima Base v1)

This content is licensed under a Creative Commons Attribution 4.0 International license.

Starting tonight, an overlay will appear, guiding users of musdb to try out the new UI. Which begs the question: Is the new UI ready for general use. Realistically there can be no general answer to that question. We have tried to prioritize the development of commonly used functionalities, while many less commonly used sections of musdb are still entirely missing in the new UI. Before a (nearly) comprehensive list and categorization of the features that are thus far missing in the new UI is presented, let’s look at a few key new features and benefits of the new UI.

New Architecture, New Possibilities

As has been outlined in the previous blog posts multiple times, the most challenging and yet basic aspect of musdb’s UI rewrite is the move to a different architectural approach. In the old UI, each page is its own script, with all these scripts using the same functions in the background. Opening a page – or e.g. saving a form – thus requires a full page reload, including a reload of all the data that remains unchanged. On the face of it, this sounds inefficient in terms of performance and bandwidth use. It is. It is also noticable in immediately perceivable characteristics like the view jumping back to the top of the page when a form is submitted.

Refactoring musdb’s backend code to a more reusable, testable and logical structure is an endeavour that has been ongoing for over half a decade now. The occassional references to musdb’s growing API in the blog were the immediate benefits of it – functions that were used to provide the regular, human-readable user interface were also made available for machine-to-machine communications.

The new UI is then the result of the final completion of that refactoring: It is a Typescript application built entirely on top of musdb’s API. On the one hand, this means that potentially any action that users can perform using musdb’s graphical UI will also be available for automated use using the API. On the other hand, it simplifies things in development: As we are ourselves using the API everywhere, good maintenance is simplfied a lot (see also).

Given this new architecture, the user interface is entirely processed in the browser. If forms are submitted, only the changed values are transmitted to the server. The required bandwidth is thus reduced greatly, once the application has loaded. Similarly, navigating only reloads those sections of a page that actually change, reducing computing load for server and client alike (and thus increasing performance).

Automation via Webhooks

Only possible due to the new architecture has been the addition of a webhook system in musdb, which offers a simple way to react to updates in musdb in an automatic way. Webhooks can be configured in the institution-level settings pages of the new UI. Note that, as the webhook system requires the use of the new architecture, webhooks configured in the new UI will not take effect when one continues to work in the old one (while both are available). See also: Wikipedia on webhooks.

Design Overhaul

Doing a re-design from the ground up naturally allows for the creation of a much more streamlined user interface. Going by the (albeit limited) feedback we have thus far received on the new UI, we have indeed succeeded in creating one. Similar functionalities that were spread over different sections of a page or different pages altogether have been moved into the same section of a page; redundant or confusing options have been removed. Where things could remain as before – as even the old design received its fair share of praise despite its shortcomings – they remain roughly the same.

The previous blog posts have already provided some in-depth glances into the new UI. Importantly for this post, the design overhaul will include the removal of some redundant, badly thought-out or effectively counter-productive funtcionalities.

List of Features Not Yet Implemented

In the following, a simple list – in some cases with further remarks – of the features the old UI offered, that are not or not yet implemented in the new UI will follow. If some feature is not to be found in the list, it is most likely already implemented.

Features That Will Certainly Be Implemented, But Are Not Yet

Whole Pages

• The general search page for object images has not been implemented yet
• Visitor counting

Navigation

• Notification system

Misc Tools

There can be accessed in the old UI using the “Tools” subsection of the dashboard or the similarly named “Tools” submenu of the navigation.

• QR-code generator
• Calendar
• Process list
• To-Do list
• Knowledge management / mini-wiki
• Hyperlink checker
• URL Shortening
• Consistency checks for migrating data from free text fields to controlled ones

Features Available in Multiple Sections of musdb

• Notes (for any page)
• Nextcloud integration
• UI for users with less permissions
  The UI should hide submission buttons etc. where a user has no permissions to perform the relevant action. This is by far only implemented in very few cases (while permissions are enforced in the backend).

Object Pages

• Overview pages
  • Sorting menu
  • Watch list
  • List view / editing most fields in a tabular view
• Sidebar
  • Warning if an object is currently reserved
  • Object record editing status (e.g. open, locked, archived)
  • Export settings for XML exports; quick exports work
  • Improvement suggestions
• Basic data tab
  • Editing translations
  • Tag suggestions based on the object image
  • Tag suggestions based on the object description
  • Ability to add sources / citations for events
• Addendum tab
  • Data field-specific setting for publication status
• Administration tab
  • Deaccession (administration tab)
• Location tab
  • Links to loan management
• Most of the restoration / conservation tab
• Data history tab
  • Versioning UI
• Custom / user-defined object editing interace
• Comparing objects

Institution pages

• Institution editing page
  • Tab for self-categorization of institutions / survey
• Institution-level settings
  • Configuration of scheduled, automatic generation of exports

Collection pages

• Hierarchy & sorting
• Editing page
• Tagging for collections

Series

• Series editing pages
  • Hierarchy
  • Sorting objects of the series
  • Sidebar
    • Setting for visibility of the series on the public institution page
    • Option to group related vocabulary entries for revision in nodac

User Settings

• Editing pages for other users
  • Configuration of user permissions
• Own account settings
  • Export of own account data
  • WebDAV settings (for preparing independently run data imports)
  • Notification settings

Contacts / Address Book

• Contact editing pages
  • vCard export
  • List of linked literature entries (to be implemented on the literature search page)

Spaces

• Hierarchy
• Editing pages
  • Sensor data display

Exhibitions

• Editing pages
  • Tab for publications linked to the exhibition

Vocabularies / Backgrounds

• Unreleased catalogue raisonne feature
• Statements

Features That Are Implemented But Certainly Need to Get Better

• Object editing page
  • Linking recently linked entities on base tab

Features That Will Be Removed

• General page features
  • Bookmarks
    Browsers are much better at bookmarking than musdb. And musdb runs in a browser anyway.
• Object page
  • Tab: Provenance research
    Feedback from provenance researchers (and common sense and experience) informed us that the current implementation is altogether ill-conceived. A whole separate, general section of musdb for managing research projects might eventually be added to actually do what we aimed to do here.
  • Entering free text data from an appropriate prepared QR-code
  • “Ask an expert” tool
• Article section (never really used)
• Podcasting features (never published)
• Series page
  • Offline HTML catalogue
    This solution has not been properly maintained in a decade. With a generally wider availability of internet connections, API-based solutions are also much more flexible.
  • Configuration for display modes of published series pages
• Chat
• Setting for the start page of a user upon logging in
  The primary and most reliable means by which we can communicate updates to musdb (and other museum-digital tools) is the blog, whose feed is embedded into the dashboard. Making sure that everybody occassionally visits the dashboard thus also means that the likelihood that users know about updates is increased. On the other hand understanding users’ bug reports regarding login and startup is simplified if there is a more unified workflow around logging in.

Features to be Discussed

• Object page
  • Social media buttons setting. With the selection of available social media buttons and current usage trends, it might be wiser to simply scrap the option.

What Does That Mean For the Roadmap?

As one can quite easily see, a lot is left to be done. But the bulk of the work has been done and we are perfectly on schedule. For anybody who relies on the features that are still missing, the old version will remain available until around November 22. For the majority of users, the current state of musdb should actually already support most or all of their requirements better than the old UI.

Until November 22 both versions are available for users to slowly transition over to the new version. We are curious for feedback and continuously push new updates to the new version.

(Post image generated using Anima Base v1)

This content is licensed under a Creative Commons Attribution 4.0 International license.

Recap: Scope and Reasons of the Rewrite

musdb as a tool has been around since sometime around 2009 or 2010 – soon after the wider inception of museum-digital. Back then, its scope was to provide an input interface for the recording of publishable museum object data. Since then, the basic architecture to the user interface has remained largely unchanged. While the scope grew to musdb aiming to be a full-featured collection and museum management system, many features that were added were only added “on top” – far enough out of the way to not disturb legacy users or to break existing workflows. In a nutshell: musdb in its current form is quite good, but it’s an application whose user interface has grown without a clear roadmap and a systematic reconsideration and streamlining of the user interface for over a decade.

Since 2009, web development has changed. We do not need to worry about the Internet Explorer anymore, browser compatibility is much better, and web applications can be much more interactive with modern browsers and hardware. To fully make use of these larger advances in the development of musdb, a general change to its architecture is necessary however.

As the architecture changes from a traditional web site with many independent pages to a single page application, the rewrite is indeed that: Instead of generating pages on the server and only displaying them in the browser, musdb’s user interface will be fully generated in the browser, with only the relevant data being queried from the server using dedicated, documented APIs. Even to keep the same functionality, we would need to re-implement its presentation. This means that the rewrite is also a perfect opportunity to streamline the user interface and to reconsider previously unused or overseen functionalities.

On the other hand: With the new user interface changing everything display-related on a technical level, that does not mean that much has to change on a conceptual level. The rewritten musdb will feature the same basic division of pages – lists, editing pages, and pages for adding new entries -, it features the same idea of a tabbed editing pages and – with few exceptions – the same data fields and order thereof.

A Sneak Preview

Where much stays the same, taking a small glimpse into things that will change might be in order. In the following there are a few such previews with small screen casts. Most of these focus on overview / list pages, which are already in a release-ready form.

Navigation

New Filter Options

Display Modes of Overview Pages

Batch Editing On All Overview Pages

Interactive Overviews

Timeline

Now, when will the new UI be done? And when will it be released?

As of now, the development is progressing smoothly and the required building blocks for almost all functionalities are there. Depending on what features one uses, the new UI can already replace the old for most common use cases. To fully do so for common use cases means that at least object editing pages need to be implemented fully. This should be the case as of late September. In the following, seminars and a time period where both versions can be used side-by-side will be had to ensure a somewhat smooth transition from the old to the new UI.

This content is licensed under a Creative Commons Attribution 4.0 International license.

(Post image generated using Anima Base v1)

Over all that time however, the basic architecture of musdb remained basically unchanged, with new functionality fit in or added on top where it seemed suitable. Additionally we have been struggling with the duplicate implementation of several functionalities at different times, where one implementation offers a very simple access to the respective data (e.g. a free text field), while the other offers a representation of the same data in a much more fine-grained, precise, and more interlinked way, whose requirements are however harder to fulfill during imports and that requires a more thorough documentation generally.

We routinely get very positive feedback from colleagues working in museums, that musdb is comparatively easy to use and logical. The age of the architecture as well as the many small inconsistencies in terms of the user interface that have crept in over time however suggest that it is time for a more radical reimagining of what musdb looks and works like.

This post is a first small preview to what musdb should look like starting in the third quarter of 2026. After a more thorough discussion of the reasons for rewriting significant parts of the publications, some first glimpses into the new version are provided at the end of this post.

Reasons

Technical

musdb is primarily a PHP application. As such, the internal logic as well as most of the user interface rest on the server. Almost all of the user interface is rendered into HTML on the server and then sent to the browser to generate what a user can view. Some functionalities that require additional interactivity, are presented in overlays, etc. are implemented in JavaScript and processed directly in the browser, with the JavaScript drawing its data both from the HTML as well as APIs (some internal, some documented for internal and external use).

Speaking in modern web development terms, PHP covers model, view, and controller on the server, with JavaScript extending the view somewhere half-way. Logically, this is problematic. In brutal practicality, it means that musdb’s JavaScript components are a collection of small scripts always dependent of some external (PHP/server-side) preparation and not logically interacting with each other. This prevents lots of the benefits of a more structured programming from being used altogether.

Our aim with the technical architecture of a reimagined musdb is to design musdb as a modern single page application with all data being provided by clearly documented APIs. This will come with several benefits:

• The view layer is entirely processed entirely in the client, providing a clear logical division between the different tasks
• With a modern build infrastructure, code can be reused much more than thus far, providing a better maintainability in the long run
• musdb will come with barely any browser-level page reloads. If an API is queried, the application can clearly communicate what is being loaded and for what purpose.
• As musdb’s UI will be entirely built based on the same API as can be used by external developers, that API will be fully feature-complete by the end of the process.

User interface

The general structure of musdb’s user interface has proven itself to be quite good. However, as musdb has grown over the years, a lot of small inconsistencies and opportunities for improvement have crept in.

Take the navigation alone:

• To access object groups, one can always hover over the three dots sub-menu of the navigation section on the very top right. If one enables the menu option “object group” in the personal settings (accessible by clicking on one’s name), one can also quickly access it via the quick access navigation (the second line of the navigation). As such, simply accessing a central part of musdb is spread over three different sections of the navigation.
• There are two question mark icons in the navigation. One, in a speech bubble, links to communication channels and contact persons that one can use to seek help. The other provides links to the handbook. On the one hand, there is a clear logical differentiation between the two. On the other, both look similar and follow rather similar aims. That the two sub-menus are divided rather than being merged into a single sub-menu is not intuitively apparent.
• Clicking on the page title or the colored circle at the right of the navigation opens the context-independent search function. Both bear no resemblance to an icon associated with searching and actually follow entirely different primary aims by themselves.

Many similar issues exist around musdb. In most cases, these somehow make sense and are logical in themselves – but they still contradict a seamless, intuitive use of the application.

Additionally, there are some functionalities that in themselves contradict a more streamlined use of musdb. One such example also accessible via the navigation is the option for users to set their preferred page to be referred to right after they log in. musdb has been built more and more on the premise, that users will start their use of the application on the dashboard, and thus some features – like requiring users to enable two-factor authentication for regional administrators – can be bypassed by selecting any non-dashboard start page. The option to select a non-standard start page will hence almost certainly be scrapped in the new version of musdb.

Preview

For now, only the basic scaffolding, most of the navigation and some of the overview pages have been implemented in the new version. Screenshots of these can be seen below. Importantly and yet missing, each different section of musdb is planned to come with an icon representing it in the navigation (top left).

Note also, that nothing about the new UI is set in stone. These are first drafts. More will follow soon.

musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

In the last post, I detailed how we severely restricted the availability of the public PDF generation functionalities in museum-digital according to available system resources. That, as it turned out, was not enough to bring reliable stability to our systems. After the server fell over on December 26th once more, we hence moved the IIIF Image API into the same PHP setup used for PDF generation – meaning that any user/IP can only request the API 10 times a minute and that for any instance of museum-digital, only one PHP worker serves it. This allowed us to severely reduce the maximum available resources per worker for the frontend outside of those two use cases (where the IIIF Image API may use up to 80 MB of RAM, no other part of the frontend will go beyond 5). Since then, the system runs as smoothly as if AI scraping had never become an issue.

A Limited Goodbye to IIIF & Server-Side Image Manipulation

Now, what does that mean in practice? On the one hand, we have not fully removed the IIIF image API. All links generated using it remain valid and will be served, even if comparatively slowly.

On the other hand the user experience with viewing the images in a IIIF viewer will be significantly worse, even though this strongly depends on the IIIF viewer. The most popular “full” IIIF viewers being Mirador and Universal Viewer, significant problems (or a complete inability to use an object’s images) are to be expected with Mirador. Mirador in its default configuration loads multiple segments of an image separately to then assemble the displayed image from those – with the creation of the segments happening on the server, thus consuming resources centrally. It also seems to set extremely low limits on accepted response times, which museum-digital’s IIIF Image API now regularly exceeds due to the aggressive rate limiting. Simply looking at the demo installation of Universal Viewer, the software seems to be much more targeted in its API calls and might still work well despite the restrictions.

As far as I know, there are no published numbers on the market share of the different IIIF image viewers. And about whether IIIF viewers external to whoever provides the API are actually regularly used or not. The most jaded – and likely true – assumption would be that the share of users who use IIIF without a viewer hosted next to the API is miniscule and that most users will use one of the abovementioned. Our experience, once again, seems to support that hypothesis: We released our implementation of IIIF 2 in 2020, but essentially nobody noticed before we also started hosting a IIIF viewer.

As we do use Mirador as a viewer, assume the “visible” IIIF image API at museum-digital to be more or less broken. Developers and those making direct use of the API without our installation of Mirador can still benefit from the API. But those are comparatively few.

The radical restriction of resources provided to the IIIF Image API is thus likely indeed a goodbye to IIIF, if a limited one. The basic idea is great – to create a unified way to reference parts of an image (or later a wider media file) and annotate it. In times of significantly increased bot activity, reduced funds, and foreseeably rising hosting costs, our example may be an early sign that the decision to realize that aim by specifying an API to be implemented by the data providers restricts the ability to fully support IIIF to very well resourced institutions. And as funds are shrinking, that is less and less institutions. Let’s hope that the most basic need IIIF wished to fulfill can be achieved in a different way in the future; one that is accessible to anybody. Realistically this means that computing would need to happen on the client PCs, not on a server.

To end the saga on a more positive note: Since we limited the IIIF Image API, our systems run wonderfully smoothly again and we were able to reduce the overall rate limiting on the rest of museum-digital’s portals. We will monitor the situation and increase the limit slowly to allow more simultaneous API requests without risking stability.

Communications

Second, the whole ordeal posed a challenge to our communication channels. If any significant error occurs anywhere on museum-digital, I personally am sent an encrypted error message via mail. Usually. In this case, the primary component falling over was the PHP server, which is also responsible for managing the sending of mails. If a service fell over, the primary way to learn of it was receiving mails about that instead. Reaction times were thus worse than they needed to be. This means that we need to improve our monitoring.

On the other hand there was the issue of explaining what was going on. We had a thread about it in the forum, which few people read. We had the blog posts. Which few people read. We lack (or lacked) a unified source of information about current events that we can assume people to read. The blog could and should be exactly that.

At the top right of the login screen of musdb, the two most recent blog posts from the respective region as well as from the “development” category of the blog have been shown for years. Then we turned on the “remember me” feature by default, which means that people only very rarely see the login page at all anymore.

The first page most users see upon logging in or opening musdb while logged in is the dashboard, the default subsection of which previously offered a summary of the database contents a user has access to, a tile for writing personal notes to oneself, a tile with messages from the respective regional administrators, a tile for the integration of a discourse forum, and links to the museum elsewhere on the web.

The summary of database contents and the links to the museum elsewhere are certainly useful. The other features not so much. Checking their actual use revealed that barely anybody used any of the note-taking features (likely also because musdb itself offers better alternatives elsewhere), while the discourse integration has not been in use for years. The very first features one sees when opening musdb were thus largely unused, wasting space that could be filled with a feed of relevant blog entries.

And so we removed the unused features and replaced them with a more prettily designed feed. This feed now contains the two newest blog posts from the development feed in the user’s language, the regional or national feed (again in the user’s language) as well as – importantly – the English-language development feed. None of the most recent development-related posts were translated to any language other than the original English, mainly because the time was better spent trying to alleviate or fix the issues than describing them in yet another language. Besides, most people know enough English to grasp the posts. And for those who do not: Community contributions to the blog – also translations for those who do not – are always welcome.

This content is licensed under a Creative Commons Attribution 4.0 International license.

As such, we have expanded our efforts in rewriting and reformulating key resource-intensive functionalities for increased stability. Different from before, we have also started to fully remove or disable functionalities that are simply not tenable anymore under the current conditions.

PDF Generation

Thus far, there were two basic types of PDFs that were generated (on the server side) in museum-digital’s portals: PDF representations of object pages (“data sheet”) on the one hand and PDFs encapsulating all images of an object in one document for easy printing.

The latter was – simply by nature of its envisioned task – extremely resource-intensive. All image files had to be loaded from disk, embedded into the PDF, compressed and served. The option had thus been available for fewer and fewer objects. Where it was originally available in case of any object with more than three images, it was later limited to objects of less than 40 images. As such, its availability was increasingly hard to communicate clearly, while its usefulness was relatively reduced with the introduction of a new download option for all images of an object. Its natural resource-intensiveness remained a problem however, and as scrapers will click any link they can find, this type of PDF generation continued to be used quite regularly (every few seconds before the recent surge in bot activity). As of last week, the functionality has been entirely removed.

The “data sheet” PDF generation has been further limited as well. As stated in the previous blog post, its usefulness is significantly reduced with the introduction of a print stylesheet (you will get better results simply pressing CTRL + P on an object page and printing the page to PDF). Nevertheless, it remained rather popular and has not been removed entirely. To reduce its impact on server stability, we however further limited its availability: If the server load is any higher than comfortable, the PDF will not be generated and an error message will appear. If the load is high (up from around 70% of comfortable) and the user’s browser language is not the default language of an instance of museum-digital, the same error message will appear.

Failed Search Pages

If a search query for objects fails, users are forwarded to a failed search page, on suggestions for alternative search queries are made. This is essentially the same as Google automatically suggesting corrections when search terms contain typos. Identifying the alternatives and offering previews for each is not free. As it is simply suggestions, the benefit or general accuracy of the suggestions fluctuates from case to case.

Now, looking at the logs, we had a large number of queries for non-existing entities – obviously scrapers who were trying out different IDs after analyzing the URL scheme. Each of those queries was executed and then forwarded to the failed search page, triggering the loading of suggestions and previews and thus further using resources on the server for little benefit (besides getting more links to scrape). We have now introduced a similar logic to the limitations on the data sheet PDF generation. Suggestions and previews are only generated when server load is comparatively low, with non-primary language users being slightly disadvantaged vis-a-vis primary-language users in an instance.

Timelines

Timelines remain popular – and a problem. A very common type of query we would see in our logs would combine timelines with searches by start and end date. This was likely due to another possible loop of endless URL generation for scrapers – specify a timeline until it forwards to search pages for a given timespan, then open the timeline for that timespan. Exactly that behavior has now been made impossible. If a search by a timeline (“start after”, “end before”) has been set, timelines will not be offered in the sidebar anymore. Trying to generate them for such a search using URL manipulation or the API will return an error page.

Search: Cleanup, Image Search & Checking Entity Existence Early

A more messy way of optimizations hit the core of the object search. In around 2021, we introduced a new search logic. Almost all pages relying on the core search logic – search overview pages, maps for objects, timelines, were adjusted to work with the new logic. The only exception from this was the image search. Still, as the new search logic re-used some of the old search logic’s functions, we kept both as separate classes, which grew over time. Simply loading the new search logic took about one ms (without OPCache enabled, measured through PHPBench). This sounds like little, but hints at a lack of modularization of the code and gains relevance with many unpredictable requests with servers automatically spinning up and down.

And indeed, in writing the new search logic, we did not modularize thoroughly HTML generation, query building and database querying. With last weeks updates, there are now separate classes for each of these and functionalities relevant only to the old search functions have been moved to class managing the image search logic. This reduces startup time for only the new / main search logic by about half (ca. 0.6 ms).

Second, we reduced the available search options for image searches. The remaining search parameters are either those actually relevant to the images or those linked to the controlled vocabularies. As a positive side effect, this also solves some issues in communication: Making it legible what the difference between searching images by their own license and by the license of (unrelated) metadata of objects the images are linked to is, is complicated.

Finally, as stated above, the logs revealed a lot of queries for objects linked to e.g. either entirely non-existent places or places that are not linked to any object in the instance of museum-digital altogether. When a place or tag is queried, we hence check whether there exists any public mention of the entity in the current instance of museum-digital during query building. If there is no link at all, it is clear early on that a more detailed (i.e. costly) query combining the search by that entity with other parameters will not return any results.

The Current Situation

All these improvements help, but a look at the current real-world numbers is warranted. On the one hand, the database server now often falls down to half or even less of the expected server load. This is a positive sign for system stability outside of peak times.

On the other hand, there are noticably spikes in the morning (around 10:20 in Germany) and in the afternoon (starting around 5 p.m.). The spike in the morning is likely related to the start of workdays and has led to the server falling over multiple times last week. This can likely be fixed only with a further tuning of the PHP-FPM settings. The spikes in the afternoon and early evening on the other hand remain hard to explain, but are altogether much less critical.

We’re on it.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Background

Scrapers

There were (or are) two factors simultaneously pushing our servers to their limits and requiring changes. On the one hand, scraping of museum-digital has gotten even more aggressive. Where we usually has something around 10-30 requests per second across all of museum-digital a year ago, we had around 300 two weeks ago. Right now it’s often between 500 and 700. This number excludes any access to static files.

As I’ve written elsewhere, the scrapers are mostly noticable by coming from IP ranges in Asia or (to a lesser extent) the US. On the other hand, the IPs change constantly and user-agents etc. resemble regular users. Likely they simply use an actual chrome browser for scraping. Which is to say, attempting to block them is futile. Worse yet, attempts to block scrapers would likely also impact some real users.

Fortunately museum-digital is run on dedicates servers paid by time rather than by compute. The onslaught of scrapers thus has no financial impact on us. But the scrapers still use resources, and as they try to scrape as many different pages as possible, it is much harder to optimize for them than it is to optimize for actual human users (see this article on a similar issue at Wikimedia).

Either way, AI scrapers can result in improvements. Viewed positively, they essentially act as a free stress test on a service and enforce efficiency in all aspects. If most pages are optimized for performance already, scrapers will find the unoptimized ones and bring down a service by overusing those. Which is to say, they help to identify yet unoptimized scripts/pages/classes and enforce that necessary changes are made. At museum-digital, there are three main weak spots that are hard to optimize: timelines, image manipulation (including the IIIF API), and PDF generation.

PHP

On November 20th PHP 8.5 was released. Thus far, museum-digital had been running on PHP 8.3 for web hosting and PHP 8.4 on the command line. When we attempted to update to 8.4 last year, the server fell over. This was mainly caused by the IIIF API (and thus, image manipulation via libvips).

Dependencies at museum-digital are (like pretty much universal with PHP) handled using the package manager composer. Setting up a new instance of museum-digital, composer (managed on version 8.4) required PHP 8.4 or later to run – the new instance was thus unable, being stuck on version 8.3 for hosting.

That leaves two options: Either to set up composer using PHP 8.3 again, or to simply update everything to the current version. While PHP 8.3 will be supported until 2027, it is generally advisable to update when possible. So updating it was.

Importantly, PHP at museum-digital is run via PHP-FPM. Before the update, we had one socket running per subdomain. This means, that if a PHP process serving the frontend stopped working for any reason, users in musdb were impacted as well.

Upgrading PHP to version 8.5

Once we upgraded the PHP version to 8.5 on Thursday, the same problems we faced with PHP 8.4 appeared again. The server would run rather smoothly for some hours, then more and more PHP processes would die and PHP-FPM would fall over for a given subdomain, and users would get a 504 gateway timeout error. Again, the IIIF API and image manipulation were the main causes of PHP-FPM getting stuck. Of course, the number of AI scrappers continuing to use the site did not help.

PHP-FPM settings

A natural first point to consider was the configuration of PHP-FPM. PHP-FPM knows three basic modes for running an application:

• ondemand You define a maximum number of processes the application may use. When a new request is made, idle processes get used. If there is no idle process, PHP-FPM starts a new one. After a specified number of requests or a given number of seconds, an old process is closed. This is primarily aimed at being able to scale way down – if there is no requests, there will be no processes (which is to say, less resources used). On the other hand, starting new processes takes time.
• static You define a number of processes that should always be running for the application. This means that there should always be processes already started and ready for usage, but it also means that those processes take up resources even when they are little used. Which is to say, this is useful if one has a high and constant stream of users.
• dynamic You define a maximum number of processes, as well as how many processes should be always running for immediate use, and a (minimum and maximum) number of spare processes to keep running. PHP-FPM then manages if more processes should be started or if one of the already running ones shall be used. This, in theory, is useful if one wants to reliably and quickly serve users, expects some use all the time, but wants the server to dynamically scale up and down as needed.

With museum-digital spread out over around 80 subdomains, we had thus far used the ondemand mode for most subdomains. Only the largest and most used instances / subdomains of museum-digital were run using dynamic mode. With the update to PHP 8.4 and then 8.5, the behavior of the ondemand mode seems to have changed. If one process dies, the whole subdomain goes seems to go down with it (I have not found a documentation on this, but it’s evident from the last two weeks).

We hence moved critical subdomains impacted by the errors (which is to say, any “regular” instance of museum-digital) to dynamic mode. As dynamic mode enforces stricter limits on how many processes can be run respective to the available hardware (which is to say, dynamic mode requires a better-written configuration), this also meant that we needed to adjust the specified numbers of processes per subdomain according to their use.

To actually grasp real use of a subdomain including bots, we turned to the logs we keep for about a week (and then rotate out). In server logs, usually one line corresponds to a single request. With a small script, we loop all the different subdomains and check how many requests were made. To be really sure that only requests to relevant PHP scripts are processed, we filter them by the presence of the substring “php” before counting. The result for today between 1 a.m. and 4 p.m. looks as follows:

| Requests count in instance                         |      Total |      musdb |        PDF | 
| -----                                              |      ----- |      ----- |      ----- | 
| agrargeschichte.museum-digital.de                  |     341508 |       1245 |        719 | 
| bawue.museum-digital.de                            |     454228 |      12559 |       6819 | 
| bayern.museum-digital.de                           |     176291 |          0 |        158 | 
| berlin.museum-digital.de                           |     223280 |      14917 |       6814 | 
| brandenburg.museum-digital.de                      |      63286 |       6927 |       3873 | 
| bremen.museum-digital.de                           |     221208 |          0 |       2026 | 
| bund.museum-digital.de                             |        261 |        167 |          5 | 
| collectors.museum-digital.de                       |     108398 |        449 |        648 | 
| hamburg.museum-digital.de                          |      35489 |          0 |         11 | 
| hessen.museum-digital.de                           |      50932 |       7962 |       2486 | 
| meckpomm.museum-digital.de                         |      94177 |         11 |        139 | 
| nds.museum-digital.de                              |     137703 |       4105 |       4134 | 
| owl.museum-digital.de                              |     427667 |       1258 |       2412 | 
| rheinland.museum-digital.de                        |      64838 |       1753 |       1276 | 
| rlp.museum-digital.de                              |     207944 |       7405 |       7532 | 
| sachsen.museum-digital.de                          |     120931 |      16117 |       6034 | 
| saarland.museum-digital.de                         |        210 |          0 |          1 | 
| smb.museum-digital.de                              |     228542 |          0 |      11517 | 
| sh.museum-digital.de                               |      21098 |          0 |         48 | 
| st.museum-digital.de                               |     317913 |       6243 |       6217 | 
| thue.museum-digital.de                             |     117893 |          0 |        495 | 
| westfalen.museum-digital.de                        |     101584 |       2033 |       3310 | 
| br.museum-digital.org                              |      43413 |          0 |         16 | 
| jateng.id.museum-digital.org                       |        211 |          0 |          0 | 
| jatim.id.museum-digital.org                        |      23410 |          0 |        159 | 
| lazio.it.museum-digital.org                        |        295 |          0 |          0 | 
| ma.pl.museum-digital.org                           |        385 |          0 |          0 | 
| noe.at.museum-digital.org                          |     906386 |          0 |        369 | 
| tirol.at.museum-digital.org                        |        537 |          0 |          7 | 
| vbg.at.museum-digital.org                          |         96 |          0 |          0 | 
| wien.at.museum-digital.org                         |     472305 |        586 |       3243 | 
| ulster.ie.museum-digital.org                       |      28869 |          0 |          2 | 
| connacht.ie.museum-digital.org                     |        392 |          0 |          0 | 
| va.srb.museum-digital.org                          |       5599 |          0 |         22 | 
| ko.rou.museum-digital.org                          |       9036 |        635 |        567 | 
| mm.rou.museum-digital.org                          |        235 |          0 |          0 | 
| ca.usa.museum-digital.org                          |       3946 |          0 |          0 | 
| ma.usa.museum-digital.org                          |        357 |          0 |          0 | 
| ny.usa.museum-digital.org                          |      19576 |          0 |        294 | 
| syddanmark.dk.museum-digital.org                   |        675 |          0 |          9 | 
| de.pt.museum-digital.org                           |       1241 |          0 |         29 | 
| zh.ch.museum-digital.org                           |     233280 |        512 |        650 | 
| ba.hu.museum-digital.org                           |      99927 |       1901 |         72 | 
| be.hu.museum-digital.org                           |     100830 |        244 |       3005 | 
| bk.hu.museum-digital.org                           |     489446 |         55 |       3985 | 
| bu.hu.museum-digital.org                           |     213616 |       6206 |       5753 | 
| bz.hu.museum-digital.org                           |     598550 |        680 |       1788 | 
| cs.hu.museum-digital.org                           |      88585 |          0 |       1054 | 
| fe.hu.museum-digital.org                           |     199812 |          7 |        215 | 
| gs.hu.museum-digital.org                           |     216680 |       4215 |        912 | 
| hb.hu.museum-digital.org                           |      61250 |          0 |         65 | 
| he.hu.museum-digital.org                           |      26312 |          7 |         26 | 
| jn.hu.museum-digital.org                           |      11970 |          0 |        131 | 
| ke.hu.museum-digital.org                           |     370219 |       2959 |       1680 | 
| no.hu.museum-digital.org                           |     119487 |          0 |       1545 | 
| pe.hu.museum-digital.org                           |     603846 |       2957 |       1446 | 
| so.hu.museum-digital.org                           |     308116 |       6151 |       6698 | 
| sz.hu.museum-digital.org                           |        116 |          0 |          0 | 
| to.hu.museum-digital.org                           |      52406 |          0 |       1229 | 
| va.hu.museum-digital.org                           |     184231 |       2839 |       1666 | 
| ve.hu.museum-digital.org                           |    1015509 |       3672 |        296 | 
| za.hu.museum-digital.org                           |        199 |          0 |          6 | 
| ce.cz.museum-digital.org                           |          3 |          0 |          0 | 
| ccc.cz.museum-digital.org                          |         17 |          0 |          0 | 
| academia.hu.museum-digital.org                     |       9158 |          0 |         13 | 
| cherkasy.ua.museum-digital.org                     |      25567 |          0 |         26 | 
| chernihiv.ua.museum-digital.org                    |       3258 |         99 |        156 | 
| dnipro.ua.museum-digital.org                       |      26725 |          0 |        109 | 
| donetsk.ua.museum-digital.org                      |         17 |          0 |          0 | 
| ivfr.ua.museum-digital.org                         |        722 |          0 |          9 | 
| kharkiv.ua.museum-digital.org                      |      12932 |          0 |         39 | 
| kyiv.ua.museum-digital.org                         |     436482 |       5967 |       1351 | 
| kyivska.ua.museum-digital.org                      |       2159 |          0 |         79 | 
| lviv.ua.museum-digital.org                         |     163358 |        188 |        274 | 
| poltava.ua.museum-digital.org                      |       7657 |        284 |          3 | 
| odesa.ua.museum-digital.org                        |         93 |          0 |          1 | 
| rivne.ua.museum-digital.org                        |      59510 |         65 |        156 | 
| sumy.ua.museum-digital.org                         |      35890 |        303 |          3 | 
| ternopil.ua.museum-digital.org                     |     150700 |         37 |        184 | 
| zhytomyr.ua.museum-digital.org                     |          3 |          0 |          0 | 
| vinnytsia.ua.museum-digital.org                    |      14229 |          0 |          0 | 
| volyn.ua.museum-digital.org                        |      16705 |          0 |        485 | 
| zakarpattia.ua.museum-digital.org                  |       2865 |          0 |         30 | 
| zaporizhzhia.ua.museum-digital.org                 |      24348 |        338 |         56 | 
| scotland.museum-digital.org                        |          0 |          0 |          0 | 
| md.museum-digital.org                              |          0 |          0 |          0 | 
| demo.museum-digital.org                            |         12 |          2 |          0 | 
| goethehaus.museum-digital.de                       |     260072 |          0 |         85 | 
| lmw.museum-digital.de                              |     326724 |          0 |         65 | 
| gedenkstaetten.museum-digital.de                   |       3474 |          0 |          0 | 
| turcica.museum-digital.de                          |      75533 |          0 |          1 | 
| nat.museum-digital.de                              |    1238860 |          0 |       4657 | 
| at.museum-digital.org                              |     631578 |          0 |         89 | 
| cz.museum-digital.org                              |          2 |          0 |          0 | 
| dk.museum-digital.org                              |       5415 |          0 |          4 | 
| hu.museum-digital.org                              |     359619 |          0 |       2827 | 
| id.museum-digital.org                              |       8030 |          0 |          0 | 
| ie.museum-digital.org                              |       2073 |          0 |          0 | 
| it.museum-digital.org                              |         78 |          0 |          0 | 
| rou.museum-digital.org                             |       8277 |          0 |        466 | 
| pl.museum-digital.org                              |        142 |          0 |          0 | 
| pt.museum-digital.org                              |          0 |          0 |          0 | 
| srb.museum-digital.org                             |        565 |          0 |          0 | 
| ua.museum-digital.org                              |     232115 |          0 |        805 | 
| usa.museum-digital.org                             |       3752 |          0 |         34 | 
| ch.museum-digital.org                              |      53417 |          0 |          1 | 
| global.museum-digital.org                          |     727690 |          0 |       2199 |

Note that the number of requests obviously is also impacted by bots changing attention – once a scraper is done with one subdomain, they turn to the next. The elevated number of requests in ve.hu.museum-digital.org is normal, but still starkly exaggerated when compared to other days. The Germany-wide instance is persistently the most frequented one, usually the global one is second at around 80% of requests.

Now equipped with actual numbers, we could scale the PHP-FPM to a much more suitable configuration than before (we had thus far never bothered counting actual requests, instead relying on the number of objects).

A second step in the PHP-FPM configuration was to reduce the impact the problems had. Previously there was one shared configuration and socket per subdomain. On the one hand, this meant that stuck processes in the frontend impacted users in musdb (and vice-versa). On the other hand, some constraints on resource usage cannot be set on a per-directory level but must be set per PHP-FPM socket / server (see the PHP documentation on user.ini and the list of php.ini directives). As the frontend and musdb have different requirements (frontend: low maximum memory use, short timeouts, no file uploads, generally strict settings; musdb: long timeouts for uploads, generally more lenient), being able to configure them independent of each other is useful in general.

We thus separated the configuration for the frontend, musdb, and PDF generation in the frontend; providing dedicated sockets for each. The frontend has a reduced priority on the system overall, strict constraints on how it may be used, etc. The settings are stricter than they were before. musdb has an elevated priority and more lenient settings (file uploads, longer timeouts), in fact more lenient than before. Finally, PDF generation is a special case as it offers no real benefit over the browser’s print tool (see MDN on print CSS), while being resource-intensive. As such, it has a far reduced priority and very strict settings.

With the separated configuration and sockets, we can now better tailor the configuration to each application’s needs and have the added benefit of problems in one application not impacting the other.

Code

As we had already prepared the codebase for PHP 8.4 awaiting an eventual upgrade, the upgrade to PHP 8.5 only required minimal changes. Aside from the deprecation of the functions finfo_close() and curl_close(), references to which were accordingly removed from the code, the update necessitated no further work.

Scaling in Software

Improving the PHP configuration was not enough to fix the issues, especially with the now increased number of requests from bots. To get some breathing room, we adjusted the most resource-intensive pages.

Frontend

In the frontend these are, again, the IIIF API, PDF generation, and timelines. Finally, we made changes to the pages for failed searches to better handle high load situations.

Image pages

The IIIF API was used for the main image pages in the frontend. We used (and use) Mirador as a IIIF viewer. Simply opening an image page thus meant three requests to fetch different regions of an image. Zooming into the image triggered further requests to fetch the relevant parts of the image. Cropping the image to the requested region with IIIF happens on the server (which is no problem if there are few users, but is turning into a problem when you have hundreds of requests per second).

We thus changed the default of image pages: The new default image page is the old, non-IIIF one. As features like zooming into images, that Mirador comes with, are popular and useful and the old image page did not support those, we worked to improve the page. To do so, we rely on OpenLayers, a library we already use for maps. Besides including maps from tile servers, OpenLayers also supports loading simple image files – which we do here. The image is hence loaded once in full size and zooming etc. happen entirely in the browser.

Taking the opportunity, we improved the page overall. An often noticed problem of image pages thus far was, that users who opened image pages coming from external services (think Google Images) had problems identifying that the image was an object image and that there is further object data to be found on object pages. The updated image pages now come with a header stating reflecting the name of the image, the name of the object and the name of the institution. Note that many images do not feature a dedicated title, musdb uses the object name as a default image title in that case, which is why the object title will often appear twice in the header. Maybe this can be used as an encouragement for the colleagues working in musdb to more consistently set expressive image titles in the future.

Also new is a mini map at the bottom left, displaying where in the wider context of the image one has currently zoomed in, as well as the ability to link exactly the region one has currently zoomed into. To enable the latter, the URL updates as one zooms or navigates around the image. Somebody else opening the same URL will then open exactly the same image region the linking person was viewing when copying the URL. Finally, we finally set specific Content Security Policies relevant to the currently opened media. If the displayed media entry is an internally stored image, no external images need to be allowed to load. If the displayed media entry is an audio file stored on archive.org, archive.org needs to be whitelisted as a source for audio files – but only archive.org and no other page. Previously, embedding images from anywhere on the net was allowed, increasing the potential damage a potential attacker may cause.

Making the use of Mirador a secondary, non-default option reduced the need for server-side image manipulation and the corresponding resource use significantly. The IIIF remains largely unchanged, but its use must now be requested explicitly.

PDF generation

As stated above, PDF generation brings little advantages to the browser’s print functionality in combination with object pages. On the contrary, the PDFs generated using the frontend’s templates feature less information. But they come with the file ending “.pdf” and seem to be extremely popular with bots. On the other hand, PDF generation means, among others, loading whatever images are to be embedded into the PDF and manipulating them fit into the PDF. The resulting files are significantly larger than the corresponding HTML files and thus also use more of the available bandwidth.

The update to handle PDF generation respective to resource usage was already introduced in the last months: publicly linked PDFs are now only generated if overall load on the server is low, if a user has set their browser language to any language different from a museum-digital instance’s default language. As most scrapers do not bother to change their browser language (which means they come with either none, English or Chinese), this means they will mostly be unable to trigger the generation of PDFs. They see an error page instead.

Failed Search Pages

If a user tries to execute a search query without any results, they will get suggestions for similar search terms – similar to how Google will ask one searching for “Berrlin”, if they meant “Berlin”. Trying to identify suitable suggestions obviously costs resources and whether the suggestions are actually what a user wanted is by nature hit or miss – it’s suggestions after all. In the case of scrapers, suggesting alternative search queries offers them a never-ending stream of possible search queries to run and keep scraping the subdomain with – to nobody’s benefit (not even the scrapers’, as they likely got the same content with other search queries already).

We thus now use the same function used to identify whether PDFs should be generated for a user to check if search suggestions should be provided. It a user comes with a non-default browser language and resource use is high, no suggestions will be provided.

Timelines

Timeline pages as implemented in museum-digital’s frontend offer another source of endless links and search queries, as they link to further and further specifications of the time searched by. Again, an improvement already introduced months ago, was to better parse queries by time: If a user searches for objects that are linked to times “after 1920” and “after 1930”, the latter already includes the former. “After 1920 and after 1930” means exactly the same as “after 1930”. Which is one join instead of two – half the resource usage.

A minor improvement we noticed on the side was impact of automatic redirects in the timelines. Say, a user searches objects by their link to a given tag and then generates a timeline for said objects. If all objects were created in the 20th century, the timeline will automatically redirect so as to “zoom” into a more appropriate time scale than from the big bang to now. Until the last weekend, script execution was not stopped when that redirect happened – which means that all database queries for time time from the big bang to now were still executed even though the user never got to see them. That is now fixed.

The Anti-Climactical Solution

All of those changes got the frontend more or less stable. Problems with uploading images remained however. Finally, the only thing that helped was uninstalling libvips (which we use for image manipulation) and reinstalling it. That seems to have fixed the issues.

Especially as the number of requests from scrapers continues to increase, the current strategy outlined above seems to be fruitful. By reducing the use (and sometimes the availability altogether) of especially resource-intensive and – depending on the context – little useful functionalities, much stability and can be gained.

The update seems to finally be largely completed (aside from maybe some further fine-tuning of the PHP-FPM configuration) and museum-digital is stable despite the bot problem, while we haven’t had to take more drastic or costly actions yet – such as blocking or adding additional servers.

This content is licensed under a Creative Commons Attribution 4.0 International license.

• On source / reference pages, linked objects are now sorted by the position within the source work on which they are referenced or which they do themselves reference
• The target URL of the regular / unspecified search bar for objects now follows the new, prettier URL schema
• Support for an OAI-PMH API for object metadata
  • Standardized endpoint for aggregators seeking to retrieve data in batch
  • Metadata formats thus far supported:
    • LIDO
    • OAI-DC (mandatory)
  • See also: Blog
• PDFs are only generated for users with a browser set to a non-default language if load on the server is low
  • The resource use caused by AI bots scraping museum-digital has been growing and growing. Generally, we see bots included in our mission to enable access to cultural heritage. On the other hand, nobody is served if the service is bogged down by bots. One functionality that is commonly used among bots and resource intensive is the generation of PDFs for object pages. The same information can be loaded from the object page itself and printed to a PDF using the browser’s print option. There are thus rather few downsides to limiting access to PDF generation to timmes, when server load is low. So that’s what we did.
• Collection-specific ISIL identifiers are now also used in the LIDO API
• Alternative numbers of an object can now be displayed on object pages
  • This includes tooltips for types of alternative numbers, that can be set by the museum on the institution-wide settings pages of musdb

musdb

• Search for objects
  • Type-ahead search for languages (of the object’s content)
  • Search by object’s revision status (open, read-ony, archived, etc.)
• Batch editing of objects’ revision status
• Parameters of the full text search index were updated to improve the search of word compounds in German

Importer

Core

• The dry-run mode now does not abort an import anymore, if an unmapped value is encountered. Unmapped entries are collected and displayed together afterwards.
  • This means, that unmapped entries can now much more easily be copied to and mapped in concordance.museum-digital.org
• Support for the import of alternative numbers (of objects)
• Support for the import of space hierarchies

Parser

• AdlibXml
  • Support for importing objects’ alternative numbers
• CsvXml
  • Support for importing objects’ alternative numbers
• CsvLocations
  • New parser for csv-based imports of space hierarchies
• ImageByInvno
  • New setting: append_chars (Adds suffixes, that exist in the inventory number, but not in file names)

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Significantly reworked the display of transcriptions on object pages
  • Titles of transcriptions are now displayed
    • If none is set, the type of the transcription (original or translation) is used as a replacement
  • Transcriptions are sorted by their titles
  • Improved the display of transcriptions in tiles
    • Problems with vertical scrolling are now solved
    • If only one transcription has been recorded, it will be displayed on the full width of the page
    • If there are more than two transcriptions for an object, they are folded in by default and can be unfolded later on
• Batch export of object metadata via the API
  • Thus far available in JSON & LIDO
  • API documentation
  • See also
• Dots as a separator in floating point numbers for object measurements are replaced with a comma in languages that require that
• Collection-specific ISIL IDs are used in the LIDO API

musdb

• Added a field for recording titles / names of transcriptions
• Added the option to set collection-specific ISIL IDs
• Setting object type tags via the improvement suggestions now correctly classifies the thus created link between object and tag
• Additional shapes are now available
  • E.g.: round, square
• Object groups can now be filtered by whether they have a superordinate one or not

Dissemination

• 2025-10-08: Presentation at the Autumn Conference of the Working Group Documentation of the German Museum Association: “Interoperabilität schaffen – Geschichten aus 1001 Importen”
  • PDF
  • ODP
• 2025-10-14: Talk on a workshop of the project CiVers (Citation of Versioned Web Pages by PID)
  • PDF
  • ODP
• 2025-10-17: museum-digital Usertagung 2025

This content is licensed under a Creative Commons Attribution 4.0 International license.