musdb

Object overview

In the default tile view of the object overview page, hovering over an object image thus far revealed the object’s name. As object names are often too long to display fully and inventory numbers are the primary means of identifying an object in most museums, this preview text has now been extended to include the inventory numer.

User management

New Options for Managing User Accounts: Disabling Accounts & Setting Account Expiry Dates

Two new options on user editing pages allow disabling logins on an account and setting an expiry date for the account. Both can be useful for administration: If a new worker joins the museum for a project with a clear-cut limitation on funding and time, one can now set the account expiry at the beginning of the project to the end of it. The accounts will then automatically be deleted when the project ends. Similarly, colleagues that leave service temporarily but for a prolonged time (e.g. for a sabbatical) and will not need to use their accounts for that time can have their accounts disabled.

List of Terms of Use

A new tab on a user’s (own) account settings page provides the option to list all usage agreements / terms of use a user has agreed to in the context of their use of museum-digital / musdb and when they agreed to them.

Imports

Limiting Report Mail Size

When a user runs imports themselves using the WebDAV upload, the end of the import process – no matter if it is successful or fails – is marked by the sending of a report via mail. This report usually contains a list of noteworthy operations that happened during the import, e.g. which objects of which inventory number were imported to which object in musdb, identified by its ID. As imports grow, this list of operation grows. To not encounter issues sending the report, it is henceforth limited to a maximum of 2 MB or 10000 lines.

Dry-run Mode

Sometimes it is useful to try running an import to see if it will actually work but not actually process any data. This option has been available in the importer command line interface for a while, among others powering museum-digital:qa. It is now available in the import configuration for self-run imports as well using the setting dry-run. Enabling the setting accordingly stops the importer from actually writing the data into the database and changes the behavior if values that need to be mapped to values in controlled lists at museum-digital are encountered. Usually an import stops the moment such data is to be imported and not yet mapped. During a dry run, the error is collected and the import proceeds. All unmapped entries are listed together at the end of the import, allowing for a simpler mapping (possibly aided by concordance.museum-digital.org).

Dashboard

The first page of the dashboard, which for almost all users also means the start page of musdb right after the login process, was significantly reworked during the last month. The almost entirely unused notetaking features and discourse integration were removed in favor of a feed of recent blog posts. See also the section “Communications” in the respective blog post.

Annotations for the Vocabulary Editing Team

Each event, displayed as a tile on object editing pages, featured speech bubble icons behind each time / actor / place to provide additional comments and hints for the central vocabulary editing team. This positioning of the annotation feature led to confusion over the years, with some users using the feature to comment on the relationship between the entity and the object (for which the event notes should be used). We hence repositioned the links and moved them to the respective entity’s page (e.g. a place page for giving hints and comments on a place entry). The hinting / commenting feature for times has been altogether removed, as providing comments to clarify the meaning of e.g. a year never made much sense.

Smaller Updates and Bugfixes

• Fixed a bug in the HTML generated for listing other objects linked to an object. Links to the other object were broken and are not anymore.
• Image editing pages now embed the image directly instead of using the IIIF API. This reduces resource usage and increases stability at no cost.
• Removed option to manually trigger the rewriting of EXIF and IPTC metadata of object images. Rewriting takes place in the background whenever an image or a linked object is updated, making user-triggered updates obsolete.
• Re-introduce option to repeat linking to the last used linked object
• Updated Swagger UI to version 5.30.3

Frontend

As stated above and lengthily described in the previous blog posts (here, here, and here) we struggled with stability over the last month. This means that most changes in the frontend are aimed at improving stability.

Reworked Default Image Page

Thoroughly described in Updates, AI scrapers, and Resilience, we replaced the default view for single object image pages. While the default view was previously built around the IIIF viewer Mirador, the new default view uses OpenLayers and the unmediated image file for capabilities such as zooming. The new view also brings with it some new features, such as an option to reference specific sections of an image.

Serving Resource-Intensive Pages / Functionalities Only When Resources Are Available

PDF generation, the IIIF Image API, and the suggestions for alternative search queries on failed search pages are now limited to reduce their impact on the overall system stability. This follows two strategies:

• Suggestions on failed search pages and PDF generation will only appear if the overall load on the system is low. The threshold for when or when they are not provided is influenced by the user’s browser language: If a user uses a browser set to the primary language of a given instance of museum-digital (e.g. German in Hesse, Hungarian in Budapest), the threshold is much higher, meaning users will be able to access the pages at a medium server load. In the case of PDFs, high server load will forward users to the print dialogue for object pages instead of receiving a PDF generated on the server side.
• PDF generation and the IIIF Image API are served with a different PHP configuration and set of processes than the rest of the frontend. This configuration significantly reduces available resources for these two functionalities.
• The option to generate PDFs featuring all images of an object with between 10 and 40 images has been entirely removed. Given its constraints, the feature was hard to explain and rarely accessible anyway. The primary “users” were noticeably AI scrapers.

Image Search

The image search feature was refactored and reduced to further separate it from the primary object search. The number of available search options has been reduced to be more easily explainable and reduce possibilities for very resource-intensive queries.

Batch Export of Object Metadata / OAI

Updated the LIDO API to almost entirely match the LIDO as generated during exports from musdb

Smaller Updates and Bugfixes

• Improved performance of object search by tags and places by filtering searched entities to those who are actually linked in the given instance of museum-digital.
• Object groups with only one object are henceforth not displayed and linked on object pages anymore
• Fixed link in footer: Clicking on “museum-digital” should lead to the home / start page of the given instance of musdb.
• Updated Swagger UI to version 5.30.3

nodac

• User-provided comments / hints have been removed for times (see above)
• Tooltips for linked objects now display which institution an object belongs to
  • This is particularly important for vocabulary editors who do not have access to the museums’ data. This way they get a limited preview with the required information for unpublished objects despite their otherwise lacking permissions.

This content is licensed under a Creative Commons Attribution 4.0 International license.

In the last post, I detailed how we severely restricted the availability of the public PDF generation functionalities in museum-digital according to available system resources. That, as it turned out, was not enough to bring reliable stability to our systems. After the server fell over on December 26th once more, we hence moved the IIIF Image API into the same PHP setup used for PDF generation – meaning that any user/IP can only request the API 10 times a minute and that for any instance of museum-digital, only one PHP worker serves it. This allowed us to severely reduce the maximum available resources per worker for the frontend outside of those two use cases (where the IIIF Image API may use up to 80 MB of RAM, no other part of the frontend will go beyond 5). Since then, the system runs as smoothly as if AI scraping had never become an issue.

A Limited Goodbye to IIIF & Server-Side Image Manipulation

Now, what does that mean in practice? On the one hand, we have not fully removed the IIIF image API. All links generated using it remain valid and will be served, even if comparatively slowly.

On the other hand the user experience with viewing the images in a IIIF viewer will be significantly worse, even though this strongly depends on the IIIF viewer. The most popular “full” IIIF viewers being Mirador and Universal Viewer, significant problems (or a complete inability to use an object’s images) are to be expected with Mirador. Mirador in its default configuration loads multiple segments of an image separately to then assemble the displayed image from those – with the creation of the segments happening on the server, thus consuming resources centrally. It also seems to set extremely low limits on accepted response times, which museum-digital’s IIIF Image API now regularly exceeds due to the aggressive rate limiting. Simply looking at the demo installation of Universal Viewer, the software seems to be much more targeted in its API calls and might still work well despite the restrictions.

As far as I know, there are no published numbers on the market share of the different IIIF image viewers. And about whether IIIF viewers external to whoever provides the API are actually regularly used or not. The most jaded – and likely true – assumption would be that the share of users who use IIIF without a viewer hosted next to the API is miniscule and that most users will use one of the abovementioned. Our experience, once again, seems to support that hypothesis: We released our implementation of IIIF 2 in 2020, but essentially nobody noticed before we also started hosting a IIIF viewer.

As we do use Mirador as a viewer, assume the “visible” IIIF image API at museum-digital to be more or less broken. Developers and those making direct use of the API without our installation of Mirador can still benefit from the API. But those are comparatively few.

The radical restriction of resources provided to the IIIF Image API is thus likely indeed a goodbye to IIIF, if a limited one. The basic idea is great – to create a unified way to reference parts of an image (or later a wider media file) and annotate it. In times of significantly increased bot activity, reduced funds, and foreseeably rising hosting costs, our example may be an early sign that the decision to realize that aim by specifying an API to be implemented by the data providers restricts the ability to fully support IIIF to very well resourced institutions. And as funds are shrinking, that is less and less institutions. Let’s hope that the most basic need IIIF wished to fulfill can be achieved in a different way in the future; one that is accessible to anybody. Realistically this means that computing would need to happen on the client PCs, not on a server.

To end the saga on a more positive note: Since we limited the IIIF Image API, our systems run wonderfully smoothly again and we were able to reduce the overall rate limiting on the rest of museum-digital’s portals. We will monitor the situation and increase the limit slowly to allow more simultaneous API requests without risking stability.

Communications

Second, the whole ordeal posed a challenge to our communication channels. If any significant error occurs anywhere on museum-digital, I personally am sent an encrypted error message via mail. Usually. In this case, the primary component falling over was the PHP server, which is also responsible for managing the sending of mails. If a service fell over, the primary way to learn of it was receiving mails about that instead. Reaction times were thus worse than they needed to be. This means that we need to improve our monitoring.

On the other hand there was the issue of explaining what was going on. We had a thread about it in the forum, which few people read. We had the blog posts. Which few people read. We lack (or lacked) a unified source of information about current events that we can assume people to read. The blog could and should be exactly that.

At the top right of the login screen of musdb, the two most recent blog posts from the respective region as well as from the “development” category of the blog have been shown for years. Then we turned on the “remember me” feature by default, which means that people only very rarely see the login page at all anymore.

The first page most users see upon logging in or opening musdb while logged in is the dashboard, the default subsection of which previously offered a summary of the database contents a user has access to, a tile for writing personal notes to oneself, a tile with messages from the respective regional administrators, a tile for the integration of a discourse forum, and links to the museum elsewhere on the web.

The summary of database contents and the links to the museum elsewhere are certainly useful. The other features not so much. Checking their actual use revealed that barely anybody used any of the note-taking features (likely also because musdb itself offers better alternatives elsewhere), while the discourse integration has not been in use for years. The very first features one sees when opening musdb were thus largely unused, wasting space that could be filled with a feed of relevant blog entries.

And so we removed the unused features and replaced them with a more prettily designed feed. This feed now contains the two newest blog posts from the development feed in the user’s language, the regional or national feed (again in the user’s language) as well as – importantly – the English-language development feed. None of the most recent development-related posts were translated to any language other than the original English, mainly because the time was better spent trying to alleviate or fix the issues than describing them in yet another language. Besides, most people know enough English to grasp the posts. And for those who do not: Community contributions to the blog – also translations for those who do not – are always welcome.

This content is licensed under a Creative Commons Attribution 4.0 International license.

As such, we have expanded our efforts in rewriting and reformulating key resource-intensive functionalities for increased stability. Different from before, we have also started to fully remove or disable functionalities that are simply not tenable anymore under the current conditions.

PDF Generation

Thus far, there were two basic types of PDFs that were generated (on the server side) in museum-digital’s portals: PDF representations of object pages (“data sheet”) on the one hand and PDFs encapsulating all images of an object in one document for easy printing.

The latter was – simply by nature of its envisioned task – extremely resource-intensive. All image files had to be loaded from disk, embedded into the PDF, compressed and served. The option had thus been available for fewer and fewer objects. Where it was originally available in case of any object with more than three images, it was later limited to objects of less than 40 images. As such, its availability was increasingly hard to communicate clearly, while its usefulness was relatively reduced with the introduction of a new download option for all images of an object. Its natural resource-intensiveness remained a problem however, and as scrapers will click any link they can find, this type of PDF generation continued to be used quite regularly (every few seconds before the recent surge in bot activity). As of last week, the functionality has been entirely removed.

The “data sheet” PDF generation has been further limited as well. As stated in the previous blog post, its usefulness is significantly reduced with the introduction of a print stylesheet (you will get better results simply pressing CTRL + P on an object page and printing the page to PDF). Nevertheless, it remained rather popular and has not been removed entirely. To reduce its impact on server stability, we however further limited its availability: If the server load is any higher than comfortable, the PDF will not be generated and an error message will appear. If the load is high (up from around 70% of comfortable) and the user’s browser language is not the default language of an instance of museum-digital, the same error message will appear.

Failed Search Pages

If a search query for objects fails, users are forwarded to a failed search page, on suggestions for alternative search queries are made. This is essentially the same as Google automatically suggesting corrections when search terms contain typos. Identifying the alternatives and offering previews for each is not free. As it is simply suggestions, the benefit or general accuracy of the suggestions fluctuates from case to case.

Now, looking at the logs, we had a large number of queries for non-existing entities – obviously scrapers who were trying out different IDs after analyzing the URL scheme. Each of those queries was executed and then forwarded to the failed search page, triggering the loading of suggestions and previews and thus further using resources on the server for little benefit (besides getting more links to scrape). We have now introduced a similar logic to the limitations on the data sheet PDF generation. Suggestions and previews are only generated when server load is comparatively low, with non-primary language users being slightly disadvantaged vis-a-vis primary-language users in an instance.

Timelines

Timelines remain popular – and a problem. A very common type of query we would see in our logs would combine timelines with searches by start and end date. This was likely due to another possible loop of endless URL generation for scrapers – specify a timeline until it forwards to search pages for a given timespan, then open the timeline for that timespan. Exactly that behavior has now been made impossible. If a search by a timeline (“start after”, “end before”) has been set, timelines will not be offered in the sidebar anymore. Trying to generate them for such a search using URL manipulation or the API will return an error page.

Search: Cleanup, Image Search & Checking Entity Existence Early

A more messy way of optimizations hit the core of the object search. In around 2021, we introduced a new search logic. Almost all pages relying on the core search logic – search overview pages, maps for objects, timelines, were adjusted to work with the new logic. The only exception from this was the image search. Still, as the new search logic re-used some of the old search logic’s functions, we kept both as separate classes, which grew over time. Simply loading the new search logic took about one ms (without OPCache enabled, measured through PHPBench). This sounds like little, but hints at a lack of modularization of the code and gains relevance with many unpredictable requests with servers automatically spinning up and down.

And indeed, in writing the new search logic, we did not modularize thoroughly HTML generation, query building and database querying. With last weeks updates, there are now separate classes for each of these and functionalities relevant only to the old search functions have been moved to class managing the image search logic. This reduces startup time for only the new / main search logic by about half (ca. 0.6 ms).

Second, we reduced the available search options for image searches. The remaining search parameters are either those actually relevant to the images or those linked to the controlled vocabularies. As a positive side effect, this also solves some issues in communication: Making it legible what the difference between searching images by their own license and by the license of (unrelated) metadata of objects the images are linked to is, is complicated.

Finally, as stated above, the logs revealed a lot of queries for objects linked to e.g. either entirely non-existent places or places that are not linked to any object in the instance of museum-digital altogether. When a place or tag is queried, we hence check whether there exists any public mention of the entity in the current instance of museum-digital during query building. If there is no link at all, it is clear early on that a more detailed (i.e. costly) query combining the search by that entity with other parameters will not return any results.

The Current Situation

All these improvements help, but a look at the current real-world numbers is warranted. On the one hand, the database server now often falls down to half or even less of the expected server load. This is a positive sign for system stability outside of peak times.

On the other hand, there are noticably spikes in the morning (around 10:20 in Germany) and in the afternoon (starting around 5 p.m.). The spike in the morning is likely related to the start of workdays and has led to the server falling over multiple times last week. This can likely be fixed only with a further tuning of the PHP-FPM settings. The spikes in the afternoon and early evening on the other hand remain hard to explain, but are altogether much less critical.

We’re on it.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Background

Scrapers

There were (or are) two factors simultaneously pushing our servers to their limits and requiring changes. On the one hand, scraping of museum-digital has gotten even more aggressive. Where we usually has something around 10-30 requests per second across all of museum-digital a year ago, we had around 300 two weeks ago. Right now it’s often between 500 and 700. This number excludes any access to static files.

As I’ve written elsewhere, the scrapers are mostly noticable by coming from IP ranges in Asia or (to a lesser extent) the US. On the other hand, the IPs change constantly and user-agents etc. resemble regular users. Likely they simply use an actual chrome browser for scraping. Which is to say, attempting to block them is futile. Worse yet, attempts to block scrapers would likely also impact some real users.

Fortunately museum-digital is run on dedicates servers paid by time rather than by compute. The onslaught of scrapers thus has no financial impact on us. But the scrapers still use resources, and as they try to scrape as many different pages as possible, it is much harder to optimize for them than it is to optimize for actual human users (see this article on a similar issue at Wikimedia).

Either way, AI scrapers can result in improvements. Viewed positively, they essentially act as a free stress test on a service and enforce efficiency in all aspects. If most pages are optimized for performance already, scrapers will find the unoptimized ones and bring down a service by overusing those. Which is to say, they help to identify yet unoptimized scripts/pages/classes and enforce that necessary changes are made. At museum-digital, there are three main weak spots that are hard to optimize: timelines, image manipulation (including the IIIF API), and PDF generation.

PHP

On November 20th PHP 8.5 was released. Thus far, museum-digital had been running on PHP 8.3 for web hosting and PHP 8.4 on the command line. When we attempted to update to 8.4 last year, the server fell over. This was mainly caused by the IIIF API (and thus, image manipulation via libvips).

Dependencies at museum-digital are (like pretty much universal with PHP) handled using the package manager composer. Setting up a new instance of museum-digital, composer (managed on version 8.4) required PHP 8.4 or later to run – the new instance was thus unable, being stuck on version 8.3 for hosting.

That leaves two options: Either to set up composer using PHP 8.3 again, or to simply update everything to the current version. While PHP 8.3 will be supported until 2027, it is generally advisable to update when possible. So updating it was.

Importantly, PHP at museum-digital is run via PHP-FPM. Before the update, we had one socket running per subdomain. This means, that if a PHP process serving the frontend stopped working for any reason, users in musdb were impacted as well.

Upgrading PHP to version 8.5

Once we upgraded the PHP version to 8.5 on Thursday, the same problems we faced with PHP 8.4 appeared again. The server would run rather smoothly for some hours, then more and more PHP processes would die and PHP-FPM would fall over for a given subdomain, and users would get a 504 gateway timeout error. Again, the IIIF API and image manipulation were the main causes of PHP-FPM getting stuck. Of course, the number of AI scrappers continuing to use the site did not help.

PHP-FPM settings

A natural first point to consider was the configuration of PHP-FPM. PHP-FPM knows three basic modes for running an application:

• ondemand You define a maximum number of processes the application may use. When a new request is made, idle processes get used. If there is no idle process, PHP-FPM starts a new one. After a specified number of requests or a given number of seconds, an old process is closed. This is primarily aimed at being able to scale way down – if there is no requests, there will be no processes (which is to say, less resources used). On the other hand, starting new processes takes time.
• static You define a number of processes that should always be running for the application. This means that there should always be processes already started and ready for usage, but it also means that those processes take up resources even when they are little used. Which is to say, this is useful if one has a high and constant stream of users.
• dynamic You define a maximum number of processes, as well as how many processes should be always running for immediate use, and a (minimum and maximum) number of spare processes to keep running. PHP-FPM then manages if more processes should be started or if one of the already running ones shall be used. This, in theory, is useful if one wants to reliably and quickly serve users, expects some use all the time, but wants the server to dynamically scale up and down as needed.

With museum-digital spread out over around 80 subdomains, we had thus far used the ondemand mode for most subdomains. Only the largest and most used instances / subdomains of museum-digital were run using dynamic mode. With the update to PHP 8.4 and then 8.5, the behavior of the ondemand mode seems to have changed. If one process dies, the whole subdomain goes seems to go down with it (I have not found a documentation on this, but it’s evident from the last two weeks).

We hence moved critical subdomains impacted by the errors (which is to say, any “regular” instance of museum-digital) to dynamic mode. As dynamic mode enforces stricter limits on how many processes can be run respective to the available hardware (which is to say, dynamic mode requires a better-written configuration), this also meant that we needed to adjust the specified numbers of processes per subdomain according to their use.

To actually grasp real use of a subdomain including bots, we turned to the logs we keep for about a week (and then rotate out). In server logs, usually one line corresponds to a single request. With a small script, we loop all the different subdomains and check how many requests were made. To be really sure that only requests to relevant PHP scripts are processed, we filter them by the presence of the substring “php” before counting. The result for today between 1 a.m. and 4 p.m. looks as follows:

| Requests count in instance                         |      Total |      musdb |        PDF | 
| -----                                              |      ----- |      ----- |      ----- | 
| agrargeschichte.museum-digital.de                  |     341508 |       1245 |        719 | 
| bawue.museum-digital.de                            |     454228 |      12559 |       6819 | 
| bayern.museum-digital.de                           |     176291 |          0 |        158 | 
| berlin.museum-digital.de                           |     223280 |      14917 |       6814 | 
| brandenburg.museum-digital.de                      |      63286 |       6927 |       3873 | 
| bremen.museum-digital.de                           |     221208 |          0 |       2026 | 
| bund.museum-digital.de                             |        261 |        167 |          5 | 
| collectors.museum-digital.de                       |     108398 |        449 |        648 | 
| hamburg.museum-digital.de                          |      35489 |          0 |         11 | 
| hessen.museum-digital.de                           |      50932 |       7962 |       2486 | 
| meckpomm.museum-digital.de                         |      94177 |         11 |        139 | 
| nds.museum-digital.de                              |     137703 |       4105 |       4134 | 
| owl.museum-digital.de                              |     427667 |       1258 |       2412 | 
| rheinland.museum-digital.de                        |      64838 |       1753 |       1276 | 
| rlp.museum-digital.de                              |     207944 |       7405 |       7532 | 
| sachsen.museum-digital.de                          |     120931 |      16117 |       6034 | 
| saarland.museum-digital.de                         |        210 |          0 |          1 | 
| smb.museum-digital.de                              |     228542 |          0 |      11517 | 
| sh.museum-digital.de                               |      21098 |          0 |         48 | 
| st.museum-digital.de                               |     317913 |       6243 |       6217 | 
| thue.museum-digital.de                             |     117893 |          0 |        495 | 
| westfalen.museum-digital.de                        |     101584 |       2033 |       3310 | 
| br.museum-digital.org                              |      43413 |          0 |         16 | 
| jateng.id.museum-digital.org                       |        211 |          0 |          0 | 
| jatim.id.museum-digital.org                        |      23410 |          0 |        159 | 
| lazio.it.museum-digital.org                        |        295 |          0 |          0 | 
| ma.pl.museum-digital.org                           |        385 |          0 |          0 | 
| noe.at.museum-digital.org                          |     906386 |          0 |        369 | 
| tirol.at.museum-digital.org                        |        537 |          0 |          7 | 
| vbg.at.museum-digital.org                          |         96 |          0 |          0 | 
| wien.at.museum-digital.org                         |     472305 |        586 |       3243 | 
| ulster.ie.museum-digital.org                       |      28869 |          0 |          2 | 
| connacht.ie.museum-digital.org                     |        392 |          0 |          0 | 
| va.srb.museum-digital.org                          |       5599 |          0 |         22 | 
| ko.rou.museum-digital.org                          |       9036 |        635 |        567 | 
| mm.rou.museum-digital.org                          |        235 |          0 |          0 | 
| ca.usa.museum-digital.org                          |       3946 |          0 |          0 | 
| ma.usa.museum-digital.org                          |        357 |          0 |          0 | 
| ny.usa.museum-digital.org                          |      19576 |          0 |        294 | 
| syddanmark.dk.museum-digital.org                   |        675 |          0 |          9 | 
| de.pt.museum-digital.org                           |       1241 |          0 |         29 | 
| zh.ch.museum-digital.org                           |     233280 |        512 |        650 | 
| ba.hu.museum-digital.org                           |      99927 |       1901 |         72 | 
| be.hu.museum-digital.org                           |     100830 |        244 |       3005 | 
| bk.hu.museum-digital.org                           |     489446 |         55 |       3985 | 
| bu.hu.museum-digital.org                           |     213616 |       6206 |       5753 | 
| bz.hu.museum-digital.org                           |     598550 |        680 |       1788 | 
| cs.hu.museum-digital.org                           |      88585 |          0 |       1054 | 
| fe.hu.museum-digital.org                           |     199812 |          7 |        215 | 
| gs.hu.museum-digital.org                           |     216680 |       4215 |        912 | 
| hb.hu.museum-digital.org                           |      61250 |          0 |         65 | 
| he.hu.museum-digital.org                           |      26312 |          7 |         26 | 
| jn.hu.museum-digital.org                           |      11970 |          0 |        131 | 
| ke.hu.museum-digital.org                           |     370219 |       2959 |       1680 | 
| no.hu.museum-digital.org                           |     119487 |          0 |       1545 | 
| pe.hu.museum-digital.org                           |     603846 |       2957 |       1446 | 
| so.hu.museum-digital.org                           |     308116 |       6151 |       6698 | 
| sz.hu.museum-digital.org                           |        116 |          0 |          0 | 
| to.hu.museum-digital.org                           |      52406 |          0 |       1229 | 
| va.hu.museum-digital.org                           |     184231 |       2839 |       1666 | 
| ve.hu.museum-digital.org                           |    1015509 |       3672 |        296 | 
| za.hu.museum-digital.org                           |        199 |          0 |          6 | 
| ce.cz.museum-digital.org                           |          3 |          0 |          0 | 
| ccc.cz.museum-digital.org                          |         17 |          0 |          0 | 
| academia.hu.museum-digital.org                     |       9158 |          0 |         13 | 
| cherkasy.ua.museum-digital.org                     |      25567 |          0 |         26 | 
| chernihiv.ua.museum-digital.org                    |       3258 |         99 |        156 | 
| dnipro.ua.museum-digital.org                       |      26725 |          0 |        109 | 
| donetsk.ua.museum-digital.org                      |         17 |          0 |          0 | 
| ivfr.ua.museum-digital.org                         |        722 |          0 |          9 | 
| kharkiv.ua.museum-digital.org                      |      12932 |          0 |         39 | 
| kyiv.ua.museum-digital.org                         |     436482 |       5967 |       1351 | 
| kyivska.ua.museum-digital.org                      |       2159 |          0 |         79 | 
| lviv.ua.museum-digital.org                         |     163358 |        188 |        274 | 
| poltava.ua.museum-digital.org                      |       7657 |        284 |          3 | 
| odesa.ua.museum-digital.org                        |         93 |          0 |          1 | 
| rivne.ua.museum-digital.org                        |      59510 |         65 |        156 | 
| sumy.ua.museum-digital.org                         |      35890 |        303 |          3 | 
| ternopil.ua.museum-digital.org                     |     150700 |         37 |        184 | 
| zhytomyr.ua.museum-digital.org                     |          3 |          0 |          0 | 
| vinnytsia.ua.museum-digital.org                    |      14229 |          0 |          0 | 
| volyn.ua.museum-digital.org                        |      16705 |          0 |        485 | 
| zakarpattia.ua.museum-digital.org                  |       2865 |          0 |         30 | 
| zaporizhzhia.ua.museum-digital.org                 |      24348 |        338 |         56 | 
| scotland.museum-digital.org                        |          0 |          0 |          0 | 
| md.museum-digital.org                              |          0 |          0 |          0 | 
| demo.museum-digital.org                            |         12 |          2 |          0 | 
| goethehaus.museum-digital.de                       |     260072 |          0 |         85 | 
| lmw.museum-digital.de                              |     326724 |          0 |         65 | 
| gedenkstaetten.museum-digital.de                   |       3474 |          0 |          0 | 
| turcica.museum-digital.de                          |      75533 |          0 |          1 | 
| nat.museum-digital.de                              |    1238860 |          0 |       4657 | 
| at.museum-digital.org                              |     631578 |          0 |         89 | 
| cz.museum-digital.org                              |          2 |          0 |          0 | 
| dk.museum-digital.org                              |       5415 |          0 |          4 | 
| hu.museum-digital.org                              |     359619 |          0 |       2827 | 
| id.museum-digital.org                              |       8030 |          0 |          0 | 
| ie.museum-digital.org                              |       2073 |          0 |          0 | 
| it.museum-digital.org                              |         78 |          0 |          0 | 
| rou.museum-digital.org                             |       8277 |          0 |        466 | 
| pl.museum-digital.org                              |        142 |          0 |          0 | 
| pt.museum-digital.org                              |          0 |          0 |          0 | 
| srb.museum-digital.org                             |        565 |          0 |          0 | 
| ua.museum-digital.org                              |     232115 |          0 |        805 | 
| usa.museum-digital.org                             |       3752 |          0 |         34 | 
| ch.museum-digital.org                              |      53417 |          0 |          1 | 
| global.museum-digital.org                          |     727690 |          0 |       2199 |

Note that the number of requests obviously is also impacted by bots changing attention – once a scraper is done with one subdomain, they turn to the next. The elevated number of requests in ve.hu.museum-digital.org is normal, but still starkly exaggerated when compared to other days. The Germany-wide instance is persistently the most frequented one, usually the global one is second at around 80% of requests.

Now equipped with actual numbers, we could scale the PHP-FPM to a much more suitable configuration than before (we had thus far never bothered counting actual requests, instead relying on the number of objects).

A second step in the PHP-FPM configuration was to reduce the impact the problems had. Previously there was one shared configuration and socket per subdomain. On the one hand, this meant that stuck processes in the frontend impacted users in musdb (and vice-versa). On the other hand, some constraints on resource usage cannot be set on a per-directory level but must be set per PHP-FPM socket / server (see the PHP documentation on user.ini and the list of php.ini directives). As the frontend and musdb have different requirements (frontend: low maximum memory use, short timeouts, no file uploads, generally strict settings; musdb: long timeouts for uploads, generally more lenient), being able to configure them independent of each other is useful in general.

We thus separated the configuration for the frontend, musdb, and PDF generation in the frontend; providing dedicated sockets for each. The frontend has a reduced priority on the system overall, strict constraints on how it may be used, etc. The settings are stricter than they were before. musdb has an elevated priority and more lenient settings (file uploads, longer timeouts), in fact more lenient than before. Finally, PDF generation is a special case as it offers no real benefit over the browser’s print tool (see MDN on print CSS), while being resource-intensive. As such, it has a far reduced priority and very strict settings.

With the separated configuration and sockets, we can now better tailor the configuration to each application’s needs and have the added benefit of problems in one application not impacting the other.

Code

As we had already prepared the codebase for PHP 8.4 awaiting an eventual upgrade, the upgrade to PHP 8.5 only required minimal changes. Aside from the deprecation of the functions finfo_close() and curl_close(), references to which were accordingly removed from the code, the update necessitated no further work.

Scaling in Software

Improving the PHP configuration was not enough to fix the issues, especially with the now increased number of requests from bots. To get some breathing room, we adjusted the most resource-intensive pages.

Frontend

In the frontend these are, again, the IIIF API, PDF generation, and timelines. Finally, we made changes to the pages for failed searches to better handle high load situations.

Image pages

The IIIF API was used for the main image pages in the frontend. We used (and use) Mirador as a IIIF viewer. Simply opening an image page thus meant three requests to fetch different regions of an image. Zooming into the image triggered further requests to fetch the relevant parts of the image. Cropping the image to the requested region with IIIF happens on the server (which is no problem if there are few users, but is turning into a problem when you have hundreds of requests per second).

We thus changed the default of image pages: The new default image page is the old, non-IIIF one. As features like zooming into images, that Mirador comes with, are popular and useful and the old image page did not support those, we worked to improve the page. To do so, we rely on OpenLayers, a library we already use for maps. Besides including maps from tile servers, OpenLayers also supports loading simple image files – which we do here. The image is hence loaded once in full size and zooming etc. happen entirely in the browser.

Taking the opportunity, we improved the page overall. An often noticed problem of image pages thus far was, that users who opened image pages coming from external services (think Google Images) had problems identifying that the image was an object image and that there is further object data to be found on object pages. The updated image pages now come with a header stating reflecting the name of the image, the name of the object and the name of the institution. Note that many images do not feature a dedicated title, musdb uses the object name as a default image title in that case, which is why the object title will often appear twice in the header. Maybe this can be used as an encouragement for the colleagues working in musdb to more consistently set expressive image titles in the future.

Also new is a mini map at the bottom left, displaying where in the wider context of the image one has currently zoomed in, as well as the ability to link exactly the region one has currently zoomed into. To enable the latter, the URL updates as one zooms or navigates around the image. Somebody else opening the same URL will then open exactly the same image region the linking person was viewing when copying the URL. Finally, we finally set specific Content Security Policies relevant to the currently opened media. If the displayed media entry is an internally stored image, no external images need to be allowed to load. If the displayed media entry is an audio file stored on archive.org, archive.org needs to be whitelisted as a source for audio files – but only archive.org and no other page. Previously, embedding images from anywhere on the net was allowed, increasing the potential damage a potential attacker may cause.

Making the use of Mirador a secondary, non-default option reduced the need for server-side image manipulation and the corresponding resource use significantly. The IIIF remains largely unchanged, but its use must now be requested explicitly.

PDF generation

As stated above, PDF generation brings little advantages to the browser’s print functionality in combination with object pages. On the contrary, the PDFs generated using the frontend’s templates feature less information. But they come with the file ending “.pdf” and seem to be extremely popular with bots. On the other hand, PDF generation means, among others, loading whatever images are to be embedded into the PDF and manipulating them fit into the PDF. The resulting files are significantly larger than the corresponding HTML files and thus also use more of the available bandwidth.

The update to handle PDF generation respective to resource usage was already introduced in the last months: publicly linked PDFs are now only generated if overall load on the server is low, if a user has set their browser language to any language different from a museum-digital instance’s default language. As most scrapers do not bother to change their browser language (which means they come with either none, English or Chinese), this means they will mostly be unable to trigger the generation of PDFs. They see an error page instead.

Failed Search Pages

If a user tries to execute a search query without any results, they will get suggestions for similar search terms – similar to how Google will ask one searching for “Berrlin”, if they meant “Berlin”. Trying to identify suitable suggestions obviously costs resources and whether the suggestions are actually what a user wanted is by nature hit or miss – it’s suggestions after all. In the case of scrapers, suggesting alternative search queries offers them a never-ending stream of possible search queries to run and keep scraping the subdomain with – to nobody’s benefit (not even the scrapers’, as they likely got the same content with other search queries already).

We thus now use the same function used to identify whether PDFs should be generated for a user to check if search suggestions should be provided. It a user comes with a non-default browser language and resource use is high, no suggestions will be provided.

Timelines

Timeline pages as implemented in museum-digital’s frontend offer another source of endless links and search queries, as they link to further and further specifications of the time searched by. Again, an improvement already introduced months ago, was to better parse queries by time: If a user searches for objects that are linked to times “after 1920” and “after 1930”, the latter already includes the former. “After 1920 and after 1930” means exactly the same as “after 1930”. Which is one join instead of two – half the resource usage.

A minor improvement we noticed on the side was impact of automatic redirects in the timelines. Say, a user searches objects by their link to a given tag and then generates a timeline for said objects. If all objects were created in the 20th century, the timeline will automatically redirect so as to “zoom” into a more appropriate time scale than from the big bang to now. Until the last weekend, script execution was not stopped when that redirect happened – which means that all database queries for time time from the big bang to now were still executed even though the user never got to see them. That is now fixed.

The Anti-Climactical Solution

All of those changes got the frontend more or less stable. Problems with uploading images remained however. Finally, the only thing that helped was uninstalling libvips (which we use for image manipulation) and reinstalling it. That seems to have fixed the issues.

Especially as the number of requests from scrapers continues to increase, the current strategy outlined above seems to be fruitful. By reducing the use (and sometimes the availability altogether) of especially resource-intensive and – depending on the context – little useful functionalities, much stability and can be gained.

The update seems to finally be largely completed (aside from maybe some further fine-tuning of the PHP-FPM configuration) and museum-digital is stable despite the bot problem, while we haven’t had to take more drastic or costly actions yet – such as blocking or adding additional servers.

This content is licensed under a Creative Commons Attribution 4.0 International license.

• On source / reference pages, linked objects are now sorted by the position within the source work on which they are referenced or which they do themselves reference
• The target URL of the regular / unspecified search bar for objects now follows the new, prettier URL schema
• Support for an OAI-PMH API for object metadata
  • Standardized endpoint for aggregators seeking to retrieve data in batch
  • Metadata formats thus far supported:
    • LIDO
    • OAI-DC (mandatory)
  • See also: Blog
• PDFs are only generated for users with a browser set to a non-default language if load on the server is low
  • The resource use caused by AI bots scraping museum-digital has been growing and growing. Generally, we see bots included in our mission to enable access to cultural heritage. On the other hand, nobody is served if the service is bogged down by bots. One functionality that is commonly used among bots and resource intensive is the generation of PDFs for object pages. The same information can be loaded from the object page itself and printed to a PDF using the browser’s print option. There are thus rather few downsides to limiting access to PDF generation to timmes, when server load is low. So that’s what we did.
• Collection-specific ISIL identifiers are now also used in the LIDO API
• Alternative numbers of an object can now be displayed on object pages
  • This includes tooltips for types of alternative numbers, that can be set by the museum on the institution-wide settings pages of musdb

musdb

• Search for objects
  • Type-ahead search for languages (of the object’s content)
  • Search by object’s revision status (open, read-ony, archived, etc.)
• Batch editing of objects’ revision status
• Parameters of the full text search index were updated to improve the search of word compounds in German

Importer

Core

• The dry-run mode now does not abort an import anymore, if an unmapped value is encountered. Unmapped entries are collected and displayed together afterwards.
  • This means, that unmapped entries can now much more easily be copied to and mapped in concordance.museum-digital.org
• Support for the import of alternative numbers (of objects)
• Support for the import of space hierarchies

Parser

• AdlibXml
  • Support for importing objects’ alternative numbers
• CsvXml
  • Support for importing objects’ alternative numbers
• CsvLocations
  • New parser for csv-based imports of space hierarchies
• ImageByInvno
  • New setting: append_chars (Adds suffixes, that exist in the inventory number, but not in file names)

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Significantly reworked the display of transcriptions on object pages
  • Titles of transcriptions are now displayed
    • If none is set, the type of the transcription (original or translation) is used as a replacement
  • Transcriptions are sorted by their titles
  • Improved the display of transcriptions in tiles
    • Problems with vertical scrolling are now solved
    • If only one transcription has been recorded, it will be displayed on the full width of the page
    • If there are more than two transcriptions for an object, they are folded in by default and can be unfolded later on
• Batch export of object metadata via the API
  • Thus far available in JSON & LIDO
  • API documentation
  • See also
• Dots as a separator in floating point numbers for object measurements are replaced with a comma in languages that require that
• Collection-specific ISIL IDs are used in the LIDO API

musdb

• Added a field for recording titles / names of transcriptions
• Added the option to set collection-specific ISIL IDs
• Setting object type tags via the improvement suggestions now correctly classifies the thus created link between object and tag
• Additional shapes are now available
  • E.g.: round, square
• Object groups can now be filtered by whether they have a superordinate one or not

Dissemination

• 2025-10-08: Presentation at the Autumn Conference of the Working Group Documentation of the German Museum Association: “Interoperabilität schaffen – Geschichten aus 1001 Importen”
  • PDF
  • ODP
• 2025-10-14: Talk on a workshop of the project CiVers (Citation of Versioned Web Pages by PID)
  • PDF
  • ODP
• 2025-10-17: museum-digital Usertagung 2025

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Objects that are linked to a source as referencing the source or being referenced in it are now listed on the source’s page
  • Example: “Novalis Schriften. Die Werke Friedrich von Hardenbergs. Historisch-kritische Ausgabe. Erster Band: Das dichterische Werk. 3. Auflage”
• The object page now displays the new transcription fields for notes, status, and type of the transcription
• New types of classifications for the link between an object and a tag
  • Taxon
  • Topic
  • Mentioned subject (similar to the existing “display subject”)
• Dependencies
  • Updated OpenLayers to Version 10.6

musdb

• Renaming vocabulary entries to blacklisted terms is now impossible
  • Someone managed to create an actor “unknown” by adding a different actor and renaming the new actor entry to “unknown” afterwards.
• Added a sidebar to object group overview pages for additional filtering options
• New types of classifications for the link between an object and a tag
  • Taxon
  • Topic
  • Mentioned subject (similar to the existing “display subject”)
• New APIs for listing all vocabulary entries linked to the objects of a museum
• Dependencies
  • Updated OpenLayers to Version 10.6

Importer

• Core
  • Deaccessions are now covered by the import tool
  • Recipients of deaccessions are linked to the address book / contact list
• Parser
  • CSVXML: The CSVXML parser now covers deaccessions as well
  • ImageByInvno: Added an option to ignore any characters before a given starting combination of characters (if there is a consistent start of the inventory number).

CSVXML

• New fields
  • tag_related_identifier_type
  • tag_related_identifier

Dissemination

• Presentation “Von museum-digital zum eigenen Online-Katalog” for the Working Group Digitization of the State Museum Association of Rhineland-Palatine
  • Slides: PDF
  • Slides: ODP

This content is licensed under a Creative Commons Attribution 4.0 International license.

To that end, there have traditionally been two primary ways for data retrieval. In musdb, one could run batch exports and receive a ZIP with some form of XML files. One per object, with the objects matching the results of any given object search.

On the other hand, there is the public API. Using URL manipulation, one can access the (primary) contents of each page in a machine-readable way. To access the JSON representation of an object’s published metadata, where the object’s ID is 7141 in the Hesse instance of museum-digital (URL: https://hessen.museum-digital.de/object/7141), one simply has to insert json to the path: https://hessen.museum-digital.de/json/object/7141.

Next to the default JSON output, additional APIs are offered wherever suitable for a given data type. For objects, the primary additional output method is a LIDO API.

Thus far, the main limitation of the public API was that it only allowed one object (or institution, collection, etc.) to be queried at a time.

Querying Object Metadata in Batches

After a significant refactoring of the code to load object data for object pages – primarily to improve caching and allow for parallelized requests to the database – we are now finally able to offer APIs for querying object metadata in batch. Thanks to grouped database queries, performance and resource usage scale nicely. Taking simply the currently most recent objects in the Germany-wide instance of museum-digital: Loading all object data of one object and presenting it in JSON takes 0.0087 seconds and loading and generating the JSON for the 100 most recent objects takes 0.197 seconds (or 0.00197 per object). Note that not all queries for all aspects of an object’s metadata are grouped yet, performance may thus get even better over time. This does also not yet account for the overhead of the many HTTPs requests one would previously need to execute to get each object’s metadata one by one – real performance improvements are thus even greater.

Now, how to access object metadata in batches?

The batch access is linked with the search API and reuses its main query parameter (“s”). Say, if one is searching for objects related to Berlin (a.k.a. the place of the ID 61), the URL of the respective search page would be https://global.museum-digital.org/objects?s=place%3A61. The corresponding API for retrieving all of the objects’ published metadata would then be https://global.museum-digital.org/export/json/place:61?limit=24&offset=0. Like the search page itself and its primary API (/json/objects), the full batch export API is paginated with a maximum of currently 100 objects per page being returned.

Currently, the batch export of full object metadata is available for JSON and LIDO (XML) representations of the object data. More can rather easily be added later, should a demand arise.

OAI

Implementing a performant way to export full object metadata in bulk was one of the two main missing components for the long-missing implementation of an OAI-PMH API.

OAI is a standard tailored towards data harvesting. Say, an external service like the German Digital Library or Worldcat wants to do something with external data from diverse sources, e.g. to also display objects or implement a search across the different collections / libraries to find which one has which object / book. To be able to do so, they need to be able to access the respective data in some way. Ideally, using a common standard that describes how to query data, helps to identify any data sets that need to be updated or added (or deleted), and finally presents a uniform way to access the data periodically. That, exactly, is OAI-PMH.

In a nutshell: OAI-PMH allows other services to copy all (published) data from another service in a maschine-readable way and can thus significantly improve reuse in aggregation. Of course this only applies to technical questions; legally, potential re-users need to comply with the metadata license applied by the initial data provider regardless of the (technical) means of access.

Since last week, museum-digital now provides a OAI-PMH API at /oai respective to a given subdomain. E.g.: https://hessen.museum-digital.de/oai. As of now, the OAI-PMH API provides access to the objects’ metadata using LIDO (XML) and the mandatory OAI-DC format.

Note that there are some caveats remaining for now: First, the LIDO representation of object metadata is not (and can by definition not be) as complete and fine-grained as the JSON API. It is also not exactly similar to the LIDO as returned by exports from musdb (one is formed natively in PHP, the other using XSLT, leading to divergent development paths). Also, the LIDO output lists different identifiers from the ones used by the OAI-PMH API and the OAI-DC representations otherwise.

Finally, the OAI-PMH API at museum-digital does not implement OAI-PMH data sets to group collections. Instead, it follows the existing search logic (essentially providing a new endpoint per query). Example searches via OAI might thus look as follows:

• All objects from the Agrargeschichte instance of museum-digital, represented in OAI-DC: https://agrargeschichte.museum-digital.de/oai?verb=ListRecords&metadataPrefix=oai_dc
• All objects linked to Berlin (place #61), in the Berlin instance of museum-digital, represented in LIDO: https://berlin.museum-digital.de/oai/place:61?verb=ListRecords&metadataPrefix=lido
• All objects of the Freies Deutsches Hochstift, Frankfurt am Main, represented in LIDO: https://hessen.museum-digital.de/oai/institution:1?verb=ListRecords&metadataPrefix=lido
• All objects from Baranya with only their identifiers: https://ba.hu.museum-digital.org/oai?verb=ListIdentifiers&metadataPrefix=lido

Credits

Credits where credit is due: the Städel Museum deserves praise for their implementation of OAI-PMH. For me personally, seeing the Städel’s API was the first time I saw a visibly stateless OAI-PMH implementation, enabled by the ingenious idea of using machine-readable, JSON-encoded resumption tokens over UIDs that have to be resolved on the server side. I spent weeks (or months?) making museum-digital’s frontend stateless. By following the Städel’s example, it can remain so even while offering an OAI-PMH API.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Generally, the pace and type of development seems to have changed this year. Rather than doing many small improvements all over the place, there is less but larger and more labor intensive changes and new features. See for example the versioning in musdb (January), the tool for automating imports based on what others call a “hot folder” and the sort option to sort objects by their images’ aesthetics score (both presented here in March), and the new tool for suggesting formulations for object descriptions using large language models (June).

July

Frontend

• Translation of the software to Hindi and Telugu
• Grouping of tags by their relation to a given object
  If an object is linked to more than 10 tags, the tag list of object pages quickly starts looking unorganized and messy. In such cases, the tags will now be displayed grouped by their relationship to the object (thus far: general tag, material, technique, object type, display subject)

musdb

New Features

• Export option for the specific LIDO as expected by the German Digital Library’s “colonial contexts” portal

Improvements & Changes

• The minimum length of fulltext search terms for object search parameters is now visibly enforced in the extended search user interface
  To not overly burden the fulltext search server, any term in a full text search in musdb needs to be a minimum of two characters long. Thus far, shorter full text search parameters were simply ignored. This certainly was confusing at times. Since June, attempting to perform search queries with shorter search parameters is made impossible by a check in the extended search overlay.
• Reception history of objects: Statements of the relevant position within a source can now be 40 characters long
• Transcriptions
  • May now be up to 4000000 long
  • New fields: Notes on the transcription, status, aims

Bugfixes

• Fixed a bug in the batch editing of specific visibility flags for data fields on the addendum tab

Juni

Frontend

• Performance improvements
  • Object search now runs without a connection to the full text search server if no full text search parameter is set
  • If multiple search parameters for an earliest / latest time have been set, they are parsed and combined before being forwarded to the database
• Improvements in the deletion of temporarily created PDF files (PDF export)
• Navigation has been translated to Tamil

musdb

New Features

• Recipient of deaccessed objects can now be linked from within the address book
• New tool for AI-aided formulation o object descriptions (based on existing other metadata)

Improvements

• Object search now runs without a connection to the full text search server if no full text search parameter is set

Importer

• The CSVXML parser has been extended to cover new event types and markings
• Object groups automatically generated to group all objects of an import can now receive a description as set within the import configuration

nodac

The list of selectable languages for the navigation of nodac has now been restricted to those in which there is actually a complete translation

Image credits: “Devanagari_letters.svg” is licensed under Creative Commons Attribution-Share Alike 4.0 International, by बडा काजी, and retrieved via Wikipedia Commons.

This content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license.

New Features

• Tabs are available in the configuration when the site is installed as a Progressive Web App (PWA) (See also)

Improvements

• Duplicate search query parameters for objects associated with times before / after are removed.
  Example: If one searches for “Objects after 1900, whose first recorded associated time is also after 2000”, this is a duplicate query. 2000 is always after 1900, so one of the two parameters can be removed.
  Searches by time begin / end are also the foundation for the timeline. As the timeline is both beloved by web crawlers and resource-intensive to generate, this change significantly reduced server load.
• Any links on timeline pages that do not reference single object pages are marked as rel=nofollow
  Which is to say that bots are told to ignore them.

musdb

New Features

• Tabs are available in the configuration when musdb is installed as a Progressive Web App (PWA) (See also)
• New search option for objects: “Can be published”
  Aggregate search for objects that have not yet been published and which comply with the minimum requirements for publication.
• Separated measurements can now be batch-edited
  Available via “assign results”
• Institution-wide setting to enforce the use of the user-defined object editing interface for all users of the institution
• Exhibitions can now be searched via the API (/exhibition/list)
• A user’s user-agent is checked whenever a page is requested. If it changed, the user will be logged out automatically.
  Protection against session hijacking.

Improvements

• Panorama images for tours through exhibitions / institutions are calculated down to 2400 px height rather than 1400 px height
• The APIs for searching for entries in controlled vocabularies are now available via the main API
  See e.g. /actor/search_linked_to_objects/{term} , /actor/search/{term} etc.

Bugfixes

• Fix: “Visiting scientists” couldn’t open the “location” tab (this erroneously required museum-wide editing permissions)
• Fix: User-defined defaults for descriptions of new objects were ignored when new objects were to be added
• Fix: Thumbnails were displayed as duplicate images linked to exhibitions (tag “images”)
• Fix: Prefixing via the batch editing was broken

Importer

New Features

• New parser for CSV exports / imports from Robotron Daphne

Improvements

• CSVXML Parser
  • New literature-related fields (ISSN, editor, etc.) are now covered
  • References to wikidata are now respected as tags are imported
• The maximum length of any single given tag/keyword is centrally set to 95 characters

Bugfixes

• Some more recently added fields from the “admininstration” tag of musdb’s object pages had been read by the importer but their transfer into the database had not been implemented thus far.

nodac

New Features

• Tabs are available in the configuration when nodac is installed as a Progressive Web App (PWA) (See also)

CSVXML

Improvements

• Added new literature fields: type, editor, edition / issue, ISSN

As always, many thanks to everyone who’s made that possible!

Image credits: “Telugu consonants.svg” by Psiĥedelisto, licensed under CC BY-SA 4.0, via Wikimedia Commons

This content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license.

As always, users with browsers set to using Tamil as their main language will now automatically see the Tamil translation of the site navigation etc. Others can select to navigate the site in Tamil using the language selector at the top right.

Image credits: “Tamil Community – Worldwide.PNG” by Coppercholride, via Wikimedia Commons, licensed under Creative Commons Attribution-ShareAlike 3.0 Unported.

This content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license.

• A Kannada translation of the software is now available
• Museums can now select to display their own citation notes in the menu for such on object pages
  This is especially relevant in case the object itself is to be cited (rather than its record online).
• “Or” search queries can be combined within one search parameter for select attribute search types, e.g. places and tags. The Syntax is as follows: place:61~1
  • For now, this can only be used using the internal query language. As such, there is no corresponding option in the UI for search settings. On the other hand, the search option is thus available using the API.
  • This option is not available when searching for times or full events

musdb

• Institution-wide settings
  • Free text fields that double with similar controlled fields may now be hidden
    • The acquisition of an object may e.g. be directly recorded on the context of a given object or as a separate acquisition process. Recording it as an acquisition process is slightly more labor-intensive, but allows a more fine-grained and accurate documentation. With the new setting, the data fields for recording acquisitions directly in the object context can be hidden from all users of a museum to ensure a uniform use of the preferred functionality.
    • Institution-specific notes on how to cite objects may now be recorded for display on published object pages.
• Search queries for inventory numbers are now character-level searches again (rather than following a fulltext search logic)
• Some event types are incomplete, i.e. they cannot contain a place, or an actor, or a time. This incompleteness was handled differently between musdb, the import tool, and the CSVXML import preparation tool. Now, it is determined by a centralized list and thus similar across all of museum-digital.
• Refactoring of the administrative command line interface
  This mainly concerns auto-correction tools, but also results in that exports for the quick export option are now automatically generated daily.
  
• Separated measurements are now positioned at the very top of the “addendum” tab of object editing pages.
• It is now possible to record web links for object groups

Importer

• The import tool can now be used as a harvester
  • First use case is a harvester for LIDO records delivered via an OAI-PMH interface.
• Externally stored images in formats other than JPG can now be imported
• Significantly extended the LIDO parser to (among others) support the import of multilingual object data

This content is licensed under a Creative Commons Attribution 4.0 International license.

Here, as so often, in a list format:

Frontend

• Bugfix: The detailed description was not reflected in the object API even if set to be so via musdb
• Translation: The frontend is now available in Kannada

musdb

• Bugfix: Fixed error in setting up new two factor authentication via TOTP (app-based 2fa)
• Bugfix: Symbols for image rotation switched (counter-clockwise symbol rotated clockwise and vice-versa)
• Feature: If the generation of a PDF catalogue is triggered via the sidebar of series editing pages, the objects will follow their order within the object group

Importer

• If no explicit name/title for a resource (audio, video, PDF, 3D, externally hosted image) was supplied, there was no fallback. Now, the importer follows the same logic as musdb and will fall back to using the linked object’s object name as a resource title if none is supplied.
• The CSVXML parser can now handle multiple objects per XML file.

Auto uploader

The auto uploader is an entirely new tool. When first started, users will be asked to enter the necessary data for running imports. And they are asked for the path to a folder which will subsequently be monitored for automatic uploads.

Whenever the tool then detects suitable contents for an import in the folder, the data will be uploaded and automatically imported. Intermediate steps like the generation of a settings file for the import are covered by the tool.

The tool can thus be useful for museums that regularly import data formed in a consistent format – say, museums using museum-digital for publishing while mainting a separate collection management system. It should be uninteresting to those who spend much effort on preparing imports (e.g. via CSVXML) or who want to migrate to musdb altogether, as it only becomes useful in the case of regular use.

The code is available here, licensed under GPL v3.
See also the more detailed German-language blog post about the auto uploader.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Today we can gladly announce another large step for making museum-digital more findable and accessible: The frontend is now available in Kannada.

Users with browsers set to using Kannada as their main language will now automatically see the Kannada translation of the site navigation etc. Others can select to navigate the site in Kannada using the language selector at the top right.

This content is licensed under a Creative Commons Attribution 4.0 International license.

AI, Aesthetics and Discrimination

When working with AI it is common to criticize the inherent biases of the models used. Already underrepresented entities (people, viewpoints, etc.) are excluded, because they are not sufficiently represented in the data the model was trained on. In turn, AI repeats what it learned – pre-existing biases in society are replayed and reinforced by AI.

A second, well-founded criticism against AI applications is their impreciseness, their tendency to “hallucinate” entirely wrong results, and the lack of reproducibility of results.

Both criticisms are entirely valid. And both hint at why AI might be a useful tool exactly for generating a sort order by aesthetics. Like AI, aesthetics are imprecise: Ask 10 people to rank some pictures by their aesthetic appeal and you will get 10 different answers. But the general thrust of the answers will likely be more or less the same. There are societal rules – biases – for what pictures are more or less beautiful. But they are fuzzy and interpreted differently by each and every observer. A sense of aesthetics is about learning and reproducing those biases – learned by the inputs we learn in everyday life -, just as AI will be biased based on the data it is trained on.

Sorting then is, in essence, an exercise of discrimination. Sorting entries by ID / age in a database system will favor the newest entries and discrimate against older ones. Sorting entries alphabetically in an ascending order will favor entries whose title starts with “A” while it will discrimate against those whose title starts with “Z”. Sorting based on an AI-generated rank is discriminating. Sorting by beauty or aesthetics is as well. AI allows sorting by beauty.

What is discriminated against?

Now, if everything in sorting is discriminating, it is all the more important to consider what is actually evaluated. In other words, on what basis the discrimination takes place when ranking digitized museum objects by the beauty of their digital reproductions. As stated above, the objects are scored based on the aesthetics score established for their main thumbnails.

Images then have a range of aspects that influence their aesthetics. Image composition, lighting, contrast, the motive, and many more (an art historian could likely list hundreds). For the common critique, it is essentially only the motive that matters: Favoring images of white women over images of Asian men reproduces a range of societal problems that should not be reproduced. Transferred to object photography, the motive may be further differenciated between object type and the actual motive (e.g. of a painting). And such a discrimination is noticable in the results: Paintings seem to be generally slightly better ranked than pictures of tools. A bias based on the displayed subjects of e.g. different paintings is not something anybody from the team would have noticed, but is to be assumed that one will exists.

On the other hand, the influence of motive-focused biases is many times weaker than the actually useful discrimination based on the technical aspects of the images. Objects images taken by a professional photographer with up-to-date equipment are ranked much better than images taken using a digital camera from the 1990s. Images without a timestamp or a watermark are ranked better than ones that do feature one. Images taken with proper lighting and contrast settings are ranked much better than images taken in a dark room, presenting the objects in gray on black. Similarly, one of the most important facets contributing to the aesthetics score the predictor returns seems to be the composition: If an object is centered in the photo taken, it will score way above images featuring multiple objects at different corners of the image (a classic example would be images that show both the actual object and a photo bar).

To reiterate, these technical aspects are visibly the main contributors to the score. And discriminating based on those images is actually useful: It allows museum-digital to present newer users who are just browsing the published collections with objects recorded with a more consistently high visual quality.

What are the alternatives, or who is the audience?

museum-digital suffers the old problem of a lack of a specified target audience. A common user may be a hobbyist looking for other versions of the model train they just bought. They might be a person interested in what art of the 16th century looked like. Or which museum to visit next. Or they might be a specialist, with a much better idea of what they are actually looking for. It is common for users to leave the page after less than a minute. But there is also an astonishing number of users who stay for hours. Accordingly, it is all the more important to offer (sort) options catering to different needs.

A new user who chanced upon the platform and randomly browses it with no further background on museums will benefit from the new sorting method. Sort orders like sorting by ID or title are linear and follow a consistent logic – but that logic may in essence just reflect its own type of randomness. Sorting newer objects over older ones is essentially a random sort order, if the museum does – as is usual – record objects as they are needed in an exhibition, newly enter the museum or are simply the next object in the shelf – all of which follow a very particular, not publicly comprehensible logic. Similarly, sorting objects by their names or titles is linear. In contrast to the object entry’s age in the database, it is also immediately comprehensible to users. But museums are free to determine the object name freely – and often this is necessary. As per the best practices around publication on museum-digital, an object name should be descriptive and usable to distinguish between different objects. As most objects simply are nameless by themselves, this often means that a colleague at the responsible museum invented a name. Seeing a green vase, there may be a rather short list of names that come to mind for most people – “green vase”, “vase, green”, “vase”, “green-ish vase”, “vase in pale green” -, making the selection of the actual name comprehensible. But “green vase” is in a very different position from “vase, green” when sorting objects alphabetically. One might actually be involuntarily be sorting the objects by curator.

Other than an object’s title/name and institution, the one other facet users see when listing objects on an overview or search page is the object’s thumbnail. And as a sense of aesthetics – fuzzy as it certainly is – is roughly shared among most people (with globalization, even actually most people), sorting objects of diverse origins by the aesthetics of their thumbnails suddenly starts to look like a comparatively relatable and reasonable sort order. Nobody will agree 100%, but the rough order instinctively makes sense. That’s more than can be said about the alternatives, unless one actually looks at the sort settings.

Presenting new users with a more consistently appealing and streamlined set of search results (at the first glance at least) on the other hand might encourage them to stay for longer. And if they stay longer, they might end up actually chancing upon more objects – including ones with less well-taken pictures – eventually.

For those users who are researchers and/or specialists, who need a linear, logical sorting, the aesthetics prediction sort option is obviously of much less merit. But it is safe to assume that is this group of users are overrepresented in the above-mentioned group of people who browse the site for hours. And it is also rather safe to assume, that they are generally more used to online databases and the existence of different sort options. Consequently, it can be assumed that they are able to change the sort settings to their needs – or at least have a higher likelihood of being able to do so.

Being very useful for the general public while less so for specialists who can be assumed to be more skilled anyway, it is sensible to make aesthetics-based sorting the default. There are thus three different default sort orders, depending on what objects one lists:

• If a user lists all objects of a given instance, the default sort order remains sorting by the age of the entries
• If a fulltext search was performed, the results are sorted by how well the query string matched the entry by default
• Otherwise – in case of ID-based search queries like a query for all objects created in Berlin – the search results are sorted using the new aesthetics prediction by default

Operationalizing the Prediction

On a side note, operationalizing the prediction proved a challenge in itself. All of museum-digital’s servers are built for traditional web hosting. Hence, they feature quite powerful CPUs, lots of RAM and no GPU. In other words: They are least suitable for AI applications. And they are all the more unsuitable for scoring over a million thumbnails in bulk while remaining otherwise performant.

If the existing servers cannot be used, there are three reasonable alternatives. First, we could have simply rented another server. As a mostly volunteer-run project, this was not an option. Second, we could have used browser-based AI to calculate the score on users’ machines – we might have e.g. let an uploading user’s machine calculate the score of a thumbnail whenever the user uploads an image. But the users’ PCs vary widely, laptops and tablets (usually again without GPUs) have become more and more popular when compared to workstation, and thus such an approach would have made uploading images unbearably slow. It would have also not helped with calculating scores for the already pre-existing million of objects and their main thumbnails. Again, this was not really an option.

Finally, we could use our private machines (specifically: mine). And that is the approach we chose. Based on a new search API parameter aesthetics_score for querying objects whose thumbnails have not been scored yet (aesthetics_score:10001), my PC downloads the yet unevaluated object thumbnails and evaluates a score for each. These scores are then locally stored in one SQLite database per instance of museum-digital and exported into a CSV file. The CSV file is then uploaded to the server and ingested back into the database to set the aesthetics score for the relevant images. The search index is automatically updated with the new scores following the upload.

This structure unfortunately also means that the scoring does not occur in real-time. Depending on when I turn on or shut down my PC, the most recently published objects may remain uncategorized for a while. To be able to fairly represent such objects, they are assigned an impossibly high score that serves to both sort them above the already-scored objects while also marking them as yet unscored. Specifically, the aesthetics predictor returns a score between 0 and 10. As calculating with full integers is generally much more simple than working with floating point numbers, the score is multiplied by a thousand and then rounded, leaving one with a score between 0 and 10000. An object yet unscored will hence be assigned a default score of 10001.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Frontend

• Dates in TEI transcriptions are parsed, irrespective of whether when="" oder when='' was used
• Notes for markings are now publicly displayed
  This was missing thus far and is now implemented similar to how event notes are displayed. If a note exists, a small “[?]” appears behind the marking title line. Upon hovering over it, a tooltip appears with the relevant information.

musdb

• Names and descriptions of exhibitions and object groups can now be translated
• Version control
• Log of “current locations” of an object can be exported as a CSV file
• Uploaded object images can now be hidden or published in one batch operation
• API extended
  • (New functions)
  • Transfer object dimensions
  • List images and resources for an object
  • Image metadata
  • Publish / hide object images

January 2025

Frontend

• Objects can now be sorted by the aesthetics of the thumbnail (A dedicated blog post on this will follow soon)

musdb

• Batch transfer between between free text fields of object data
• Alignment of the maximum field length for notes on opening hours is now consistent between UI and database
• Bug fixed with switching between institutions during consistency checks fixed (relevant only to users with administrative access to multiple museums)
• Literature can now be searched by editors

Importer

• Core
  • Automatic transformation of life dates for actors
    • Year of death “01.01.2012” now becomes “2012”, instead of 01.01 as before
  • “?” and “(?)” are removed from the beginning and end of imported keywords
  • Various types of brackets in keyword names are converted to regular brackets
• Parser
  • Stricter internal implementation of settings, all imports can now implement the start_at setting
    • This is particularly useful for the repeated execution of imports that abort due to new, previously uncovered elements and other debugging.
  • New parsers:
    • MODS (mainly used in library contexts)
    • Parser for Exports from Faust for the Händel-Haus
    • Parser for XML dumps from MuseumPlus Classic (MsSQL > XML export per table > Import)
    • Bugfixes
      • Field “Verwender” in Primus parser was mapped to production events
      • Material / technology are now imported correctly in the parser for BeeCollect exports for the Industrial Museums of Saxony
• „Frontend“
  • CLI now also has options for switching off the import of individual areas
  • Help text for command line tool

nodac

• Splitting of keywords now also recognizes keywords that should be split into places, times, etc.
  • Bsp.: „Helm; Berlin“ > Schlagwort „Helm“ + Ort „Berlin“
  • Example: “helmet; Berlin” > keyword “helmet” + place “Berlin”
• When searching for keywords with ambiguous names, both keywords and generally ambiguous terms are now taken into account
• Times can now be merged with others directly from the time edit page

This content is licensed under a Creative Commons Attribution 4.0 International license.

• Basic re-implementation of separated measurements for objects (tab: “addendum”)
• The type of measurement (width, length, etc.) is now managed using a controlled list, which can easily be extended. This also allows for measurement types of different levels of specificity (“width” vs. “width of socle”)
• The new implementation allows users to identify whether a measurement is exact and add notes for each measurement
• Measured values now need to be entered as a floating point number – consistent search for objects smaller or larger than a given size is thus made possible (previously only available for entries where the system could deduce a numeric value from whatever had been entered)
• Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
• Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
• The list of spaces that can be linked to an object as its current location is now sorted alphabetically

Frontend

• Re-implemented separated Measurements

Import

• The import tool now also uses the new implementation of separated measurements
• Unique naming components of entered tags are automatically parsed into a relation type. German: “Apfel (Motiv)” is automatically split into the tag “Apple” and the relation type “display subject”.
• Users entering a tag with such a naming component that is known to belong to another vocabulary will have their input auto-corrected to reflect both. Entering the tag “Berlin (Motiv)” will be auto-corrected to a link to a “displayed place” “Berlin”.
• Links between two objects are now imported using the dedicated data type for this purpose. They had previously been imported as regular web links.

nodac

• When merging two entries, links between the entries and collections, exhibitions, etc. are now reflected and rewritten
• previously, links to such data types prevented the completion of the merge
• If an entry’s name is marked to always belong to e.g. a tag, the buttons for transferring the entry to the actor, place, or time vocabularies are now hidden.
• If an entry is moved between vocabularies, links between the entry and objects as “display subject” / “displayed place” / “displayed actor” are reflected and translated into new links using the appropriate link type.
• A new context menu on overview pages allows for a quick access to functionalities like the splitting of entries

These are in the form of listings, and this is how it should be here too.

Frontend

Features & Improvements

• Some improvements in background scripts, especially better handling of timeouts when calculating “Similar objects” in very large instances
• Contributors, linked locations and times for an object group are now listed alphabetically by name
• Table headers for event components (who, when, where) are now only displayed in the A4 PDF if there is also content for the row
• New search option for object searches: “Is currently on display”
• Links to the Themator now use the new URL scheme of the Themator
  (https://themator.museum-digital.de/t/690 instead of https://themator.museum-digital.de/ausgabe/showthema.php?m_tid=690&tid=690)

Bugfixes

• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)
• Exactness setting in the “refine search” overlay was not transferred to the actual search query
• Simple embedding of an object (analogous to YouTube videos, for example; accessible via the “Cite” menu of an object page) had various errors / now works again

musdb

Features & Improvements

• In the API documentation of musdb there is now a note that the frontend also has an API
  • Frontend API
    • You do not need to authenticate yourself to use the frontend API
    • The frontend API tends to be faster and easier to use
    • Is read-only
  • musdb API
    • Can do more: Can also see non-public stocks and fields / data types
    • Is much more granular (more queries for the same data, but you likely get exactly the data you are looking for instead of e.g. all data known about a given object)
    • Can be used for writing data
• Suggestion lists when searching for vocabulary terms in the side column of the object search page have been revised
  • Tooltips appear when hovering over
  • Implementation in Vanilla JS, removing jQuery
  • (this means significantly better performance of the search results list in list format, because jQuery no longer needs to be loaded)

• User page / Login
  • Log of logins now also with IP and user agents
  • Login via login persisted in the browser (“Remember me”) is logged and displayed
  • All browsers permanently logged in via cookie are forced to log in again after a password change
  • New option to invalidate all remembered logins on other devices (browser must be logged in again)

• Object
  • More restrictions for the publication of object data records.
  • An object can no longer be published if:
    • … the object name is the same as the object description
    • … the description contains the character string “lorem ipsum”
  • When object entries are unpublished / hidden, the images linked to the image are renamed (thus invalidating links to the images). When publishing the object again, this is reversed so that existing links work again.
  • Spaces in selection lists are now listed alphabetically as the actual location when linking
• Literature
  • Acquisitions can now be linked to literature
    • Previous owners etc. can thus be linked to a literature entry
  • Spaces (actual location) can be linked to literature

Bugfixes

• Overlay for setting searches for objects: Multi-word search terms were converted into multiple searches instead of being searched as a string of words (“red helmet” > “red” AND “helmet” instead of “red helmet”)
• Error when searching for controlled list terms that contained multiple spaces via the “Refine search” overlay (search for license “Public Domain Mark”)

Importer

• Link between literature and spaces (actual location) as well as acquisitions is implemented in the “core” of the import tool
• ImageByInvno parser (assignment of images to objects via inventory numbers contained in the file name) can now be used to import PDF files

files.museum-digital.org

• Added a small script to enhance PDF metadata based on an XML sidecar file. See e.g.: https://files.museum-digital.org/de/Praesentationen/2024-10-18_md-deutschland-eV-stellt-sich-vor_Usertreffen_MA.xml

Main post image generated using illustriousXL_smoothftSPO

This content is licensed under a Creative Commons Attribution 4.0 International license.