museum-digital:blog - Performance

Braving the Waves. Surviving in Times of AI.

2026-10-01T00:00:00+00:00

Now that the preview release of musdb’s new UI has finally been released, there’s time to write about the other sides of museum-digital. More usual struggles. Which is to say: New, larger waves of AI scrapers and bots that once again started to impair our services starting July this year.

We maintain that hypocrisy is not for us. I personally have been involved in discussions about the FAIR principles from 2015 onwards. And in 2015 they were already running under that label for a while. As a formalized document, they’ve been around since 2016, but disregarding the label, the principles and surrounding discussions go back, I guess, at least to the 1990s.

In the cultural sector, with many institutions being primarily tax-payer funded, we want to provide our data and services in a Findable, Accessible, Interoperable, and Reusable manner. Relevant here are primarily accessible, interoperable, and reusable.

Realistically, accessibility refers to two things: Immediate, universal barriers like a requirement for authentication should be avoided where possible. On the other hand, user-specific barriers should be reduced as much as possible as well: Human-readable pages should e.g. be designed with sufficient contrast.

If one sees other services and machines as users, which may be reductionist but is logically much more coherent than doing otherwise, then interoperable refers simply to a special form of accessibility: As data should be provided in a way that as many human users can use as possible, it should also be served in formats that machines can read. Ideally in ways that machines can read without further adjustments - following open standards.

Reusable then builds upon these. As the data is now freely accessible by others, setting appropriate licenses allows for the creative reuse of the data.

Ten years after the FAIR principles were formalized, we finally have someone who reuses our data on a massive scale. We should rejoice. Unfortunately all our previous conceptions of how that reuse might take place turned out to be wrong.

Celebrating AI Scrapers?

AI scrapers flooding web services, and especially larger, well-established infrastructures with requests has been an ongoing story for some two years now. Last year museum-digital was met with a first large wave of AI scrapers. See the previous blog posts (1, 2, 3).

On the one hand, using our data for training AI models is undoubtedly reuse. It is even productive reuse: It’s a small contribution to the training of technology that, at its current quality, sounded like science fiction 10 years ago.

On the other hand, this type of reuse takes place without attribution. AI scrapers work based on scale; Machine-readable APIs are uninteresting to scrapers. They suck up as much data as possible, as quickly as possible. Any page-specific adjustments, such as finding and using the API, would reduce the speed of scraping. The requests are so numerous, that they routinely crash servers. Last but not least, those prominently benefitting from the AI hype are far from sympathetic people.

But principles are principles.

It, again, cannot be denied that what AI scraping is eventually aimed at is reuse. Even though they use HTML rather than the APIs we lovingly carved for them, they interoperate with our services in the way most accessible to them.

The problem is, that they are not playing fair. And if the previously unlikely number of requests leads to servers crashing and shutting down, then all principles were for naught. A service that is providing no data at all cannot provide them FAIRly either.

At museum-digital we have seen the new requests as an opportunity to improve our software. If it can withstand the onslaught of thousands of bots a second, it is confirmedly battle-tested and stable. Improving our publishing software on the other hand also meant cutting out overly resource-hungry functionalities. Last year we dropped most publicly available, server-side PDF generation capabilities, we moved the IIIF APIs to an separate configuration that is specced to be able to fall over without impairing the rest of our services. We improved the parsing of search terms and introduced rate-limiting (how many requests a single IP can perform for a given time).

The new wave of AI scrapers since July is even bigger than last year’s. Those improvements were not sufficient to keep our primary server, which hosts the production databases, stable.

From Each According to Their Ability: Load Shedding

We continued on last year’s course however, analyzing the scrapers’ requests, their influence on server stability (which is to say, which requests were especially resource-intensive) and adjusted our setup accordingly. Note, that we did not scale up: museum-digital runs on exactly the same bare-metal servers today as it did two years ago.

In analysis, we identified two types of especially resource-intensive browsing behaviors that are almost entirely limited to scrapers (as well as some very engaged researches, to whom we express our regret for now regularly banning them):

Scrapers follow links on search pages, especially the facet search, and end up combining more and more search parameters. It is unlikely that a human would perform a search for objects that “Are related to Berlin, and related to Germany, and related to Institution X, but not related to Institution Y, and also related to a time between 1990 and 2000.” For an automated scraper it is normal behavior.
If a published object page has been updated, a snapshot of its current state is automatically saved to provide an archived version of the page at a given time. This is thought mainly for researches to be able to cite a given state of the page. In everyday operation, archive pages should be barely used. They are also de-listed from search machines. But saving snapshots again and again leads to the existence of a large number of links and separate pages scrapers can scrape to no benefit to anybody.

Both functionalities are legitimately useful but scale badly.

We hence introduced load shedding: Whenever a search is performed or an archive page is accessed, the current server load is evaluated. If it is above a certain threshold, the server does not perform complicated search queries (depending on how high load is, it is restricted to a minimum of three search parameters) or does not load the archive page. Instead a warning is presented, that the given action is currently unavailable and a custom HTTP error code is sent. If the same IP causes that error code to be sent twice, the IP is blocked for a while.

As of the time of writing writing, we have thus blocked a total of 131,014,530 IPs in two month, with 5,802,771 being currently blocked.

This approach follows our principles: We try to be fair with everybody. But if users are not fair and do not follow advise, we don’t feel obliged to continue playing fair either.
Unfortunately this approach works on the level of blocking whole IPs. As “residential proxies” - back in the days we called them botnets infecting home routers - have become an ever larger problem, it is not unlikely that actually interested, normal users also unknowingly host an AI scraper at home. I fear I’ve already encountered one such case, were a colleague could not access museum-digital for seemingly no good reason from her home network. Otherwise, it is surprisingly effective - the number of requests has barely been reduced, but our services are essentially stable.

The only crash since happened the day before yesterday and was caused by the combination of bots and an error in our controlled vocabularies - the latter of which is entirely our fault.

Numbers Games: Logging Strategy

After the above-mentioned actions had effectively returned stability to our publicly accessible portals, one issue remained for internal services: File uploads were incredibly slow. It turned out that the large number of requests being permanently logged overburdened the SSD. We have hence disabled the general access logging, restricting ourselves to logging requests that either cause errors or trigger slow responses. This reduces the ability to react well-informed to new waves of requests and actual attacks. Thankfully those also commonly trigger actual errors we can still identify.

Fairness, FAIRness

With these actions we have again averted the need to use more drastic strategies that would limit accessibility to users - human and machine alike - like blanket bans on whole regions of the world or the installation of software such as Anubis. And, again, we have not yet needed to improve our hardware (and pay the price for that), even though that might be in order eventually. It might be wise for other reasons as well.

A holistic picture of one’s setup - software, hardware, systems administration - and the capability to think and act upon these together helps a lot.
But, if anything, hosting in a FAIR, principled way still works in these trying times.

info

AI Use

Post image generated using Anima Base v1.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Summary of the monthly user meetup (April 2023) / New features and improvements

2023-05-07T00:00:00+00:00

We continued the series of monthly user meetups and again discussed the new features and improvements. A summary can be found below.

New Developments

The last month has been an exceptionally slow month in terms of technical development around museum-digital. There are however some newsworthy tidbits.

musdb

Recording external IDs for museums

Museums, like all of us, are present in more and more databases. For linking different databases, it is useful - sometimes necessary - to know the ID of the same entity in both databases. Hungarian law thus mandates collection management systems, which are to be accredited for fully paperless use in museums, to allow storing a museum’s ID with the Hungarian Statistical Office. musdb can now do exactly that: Store external IDs of museums in a given list of external source repositories / databases.

The list of available external databases and the regular expressions to validate IDs in them are available here. For now, the list only contains the Hungarian Statistical Office, but the more options there will be in the future, the better.

More fields covered by “list results” and Excel export

The “list results” page (a.k.a. table view) for object search results offers the option to view the selected objects’ data in a customizable tabular format. Mainly because of that exact tabular format, it is not possible to display everything a full database view of the object offers using the “list results” page. Over the last month, we have nevertheless extended the list of displayable fields in the “list results” table.

Thus, it is now possible to display all translations for the object type, object name, descriptions, etc. of an object. Similarly, it is now possible to list all Weblinks linked to the object as a compiled (comma-separated) field. As the automated report generation and the Excel export tools build upon the same basic code as the “list results” page, these new fields are now also available in those cases.

New field for objects: “Last change of permanent location”

It is now possible to manually record the last date, the permanent location of a given object has been updated. This obviously makes sense for tracking relocations and re-organizations within the museum. Importantly, the field needs to be manually filled out, as the date of the last change of the permanent location may be far in the past or frankly unrelated to the database, even if there is one (say, the museum moves the depot to a different place, but the date is only later updated in the database via a batch editing process).

User interface

After there were some reports of people not seeing the difference between a valid form in musdb and an invalid / incomplete one, we updated the design of submit buttons. Submit buttons in incomplete forms are now blurred out, to add another hint at the incompleteness of the form.

Frontend

Improved performance and decreased bandwidth usage: serving images in webp

Over the last years, there have been a number of new image formats seeking to combine all the features of older and well-established formats like JPG and PNG with an improved compression. The most established format of that newer generation of image formats is webp, which is by now well-supported by all modern browsers and most local image viewers.

To save bandwidth and improve loading speed, we now store and serve full-sized webp versions of newly uploaded object images along with the regular jpg versions. If possible, the webp version is served on object pages. An additional benefit is the aforementioned support for features jpg files do not support, such as transparent image backgrounds.

iFrame embedding now only possible from whitelisted sources

If a museum wants to embed their data from museum-digital into their own website, there are traditionally two ways to do so. The better, but much more complicated option is to use the API to fetch the relevant data and present them in any way one wants. Much easier (and cheaper obviously) is embedding the museum’s data using iframes.

Unfortunately, iframes can also be used for attacks on users’ login data. We have thus now restricted this option to only allow embedding from whitelisted sources. If a museum wants to embed their data this way, this means that they now need to notify their regional administrators beforehand.

Importer

The importer now supports imports for exports from Startext’s HiDa in the configuration of the Saxon State Agency for Museums.

Other News

In other news, the Youtube channel has picked up steam, with some new tutorials both in German and Ukrainian.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.

Summary of the monthly user meetup (March 2023)

2023-04-05T00:00:00+00:00

Yesterday, we held our regular user meetup as scheduled. As promised, below you can find an overview of the new features and updates below some more general points.

General

YouTube channel

There now is a museum-digital YouTube channel. For now, one can find some German-language screencasts on different features in musdb and nodac there.

New sections on project page (www.museum-digital.org)

As already described in the previous post, the project page www.museum-digital.org now features a calendar. Since then, we have also added a small page listing people working on museum-digital - e.g. some of the coordinators and developers - as well as power users. The list will obviously always be more than incomplete, but if you want to be listed, just send a mail.

Development

musdb

Allow batch editing of photographers on image list

Running batch updates on the ownership, license status and photographer of a set of objects can be done either via the institution-wide settings page or via a selection on the image list page. The batch updating functionality of the institution-wide settings page works by replacing the given entries in one of the fields for all objects of the museum - it is thus not suitable if one is to only update the photographer field for a list of objects from a single collection (rather than the whole museum).

Batch updating images’ legal information from the image list (click on an image in the list and drag it to activate selection mode, then update by selection) was only possible in the case of image licenses and rights holders. With a small update, it is now possible to also batch-update the images’ photographers field that way.

The API can now update object event

Thus far, the API could only be used for adding wholly new object events (who did what, when, and where, with the object) or deleting them. With the updates of this month, now possible to directly update events.

This is already actively used in customizing musdb using a Tampermonkey script by some of the norm data editors. They can thus add a custom button to transfer incomplete actor or place information to the object’s description. This button, on the other hand, would not be useful for regular users at all.

UI improvements for the object search interface and new sort option

After a very fruitful discussion and feedback from Hungarian colleagues, we got around to some user interface improvements in the object overview and search interface. A lengthy discussion can be found in a dedicated blog post in German. To sum up: there are now headlines for each subsection of the sidebar and tooltips explain the different buttons and search and sort options upon hovering one’s mouse over them. Additionally, a new sort option was added to sort objects by the number characters within the inventory numbers of the searched objects.

Two bugfixes: Navigating to the previous or next object in a selection

It is a very simple but similarly useful feature: One runs an object search and thus creates a result list of objects. After viewing or editing one of the objects, one can then proceed to the next one in the results list by clicking at the arrows at the top of the sidebar.

Unfortunately, musdb had two bugs preventing this navigation for some of the available sort options. On the one hand, newer sort options required manually adding the search fields to a list specific to the previous/next navigation thus far - and had not been covered yet. Navigating to the next object after sorting by e.g. the length of the searched objects was thus impossible so far. The specific list of sort fields has now been removed in the previous/next navigation. Instead, the main list of sort options from the main object search class is directly used - in effect preventing such issues of “forgotten” values from ever appearing again.

A second issue concerned string-based sort options. For an improved performance, the previous / next navigation works by querying the underlying search index for the next object where e.g. the object ID is lower than the current one (`WHERE id > 20000`). The same search logic is straight-out not possible with string-based sort options, as search queries for values greater than or smaller than a given one cannot be executed with string-based fields.

Fortunately, the issue came up just after we had implemented an option to search and sort objects by the numeric components of their inventory numbers. Thus, the previous / next navigation can fall back to that sort option if a user wants to navigate to the next object after sorting by inventory number. As stated back when we introduced the new search and sort option: It should work for most museums. Unfortunately, there is no such option to mitigate the issues of string-based sorting when sorting objects by their names. The best we could do in this case is displaying a clear error message, stating that the previous / next navigation is not available when sorting by the object’s names.

“Home location” can now be selected as a required field when adding new objects

On the institution-wide settings page, museum directors (or those holding the same user role within musdb) can determine which fields are required when adding new objects. The list of selectable fields automatically covers all free text fields linked to the object. Repeat fields and references to other sections of musdb need to be manually implemented however. As the home or permanent location of an object is a most logical and common field to be required for all object, it was only consequential to prioritize making it available as a required field. We have done so now.

Major performance improvements in “list results” page

The “list results” page provides for a table view of object search results, allows the export of search results to Excel, and is - under the hood - also used for generating custom reports. We managed to achieve a considerable improvement in the page’s performance by bulk loading data and simplifying the code of the HTML page. Both adjustments should not really be noticable when one tries to list or export only some objects. Exporting some thousand objects should work considerably faster now however.

frontend

PDF metadata for main object PDF export

XMP metadata are now written into exported PDF files. Thus, users can more easily identify authorship, titles etc., especially when using specialized software like e-book readers.

Linking from collection descriptions

Collection descriptions are now being parsed for the existence of URLs. If one, starting with https:// (http:// is ignored) exists, the URL is automatically transformed into a clickable link.

nodac

We added some translation variables on the start page.

themator

The “themator” has received a small user interface improvement for mobile devices. The topic-specific navigation or table of contents is now moved below rather than above the main content of a page in the regular “topic” viewing mode on mobile devices.

Importer

Importing GIFs

To simplify the handling of images, museum-digital internally only uses JPG image files (webp versions are generated for a faster loading, but are optional). While they lack some support for features like transparency, JPGs provide for a much better compression with images of three-dimensional things and subjects not featuring solid colors. Newer formats like AVIF and Webp unfortunately still lack the support among locally installed image viewers for us to fully switch over to them.

The importer handles input PNGs by converting them to JPG files. Starting this month, the same can be done with input GIF files.

Improved handling of blacklisted actors, places and times

One of the key issues with imports is what happens after imports. Specifically, museum-digital uses controlled vocabularies for actors, places, times and tags that can be linked to an object directly or via events. When people enter new such entries using musdb, there are a number of safeguards and nudges to ensure productive inputs. Adding a new actor, for example, requires one to enter at least 10 characters of a description for the actor. Input fields for the date of birth and death are also provided. If possible, one can enter a Wikipedia or Wikidata URL (or ID) and directly fetch the relevant information from there.

During imports, there is no space for asking questions to the importing institution. Incomplete or false actor, place, time and tag names (for example, if two actors are linked as one; “John Doe and Jane Doe” is not one single actor!) are given, these are necessarily imported into our controlled vocabularies by default. Depending on the import, this puts a significant burden on our volunteer group of vocabulary editors.

To eventually get to a solution, we have added options for automatically rewriting terms (e.g. “Berlin, Germany” will be autocorrected to “Berlin”) and blacklisting them completely (“Good morning …” is no actor“). Thus far, blacklisted terms were simply ignored by the importer.

“Unknown painter” is not an actor, but the information that it is a painter may still be useful. To allow us to blacklist more and keep the controlled vocabularies “clean” without losing such data, the importer now imports blacklisted terms in event components (“who painted it?”) into the event annotation. If the only known information on the whole event is blacklisted, the event cannot be created - an event always requires at least one linked actor, place or time. The event annotation of “empty events” is hence moved to the object description automatically.

info

License

This content is licensed under a Creative Commons Attribution 4.0 International license.