As such, we have expanded our efforts in rewriting and reformulating key resource-intensive functionalities for increased stability. Different from before, we have also started to fully remove or disable functionalities that are simply not tenable anymore under the current conditions.

PDF Generation

Thus far, there were two basic types of PDFs that were generated (on the server side) in museum-digital’s portals: PDF representations of object pages (“data sheet”) on the one hand and PDFs encapsulating all images of an object in one document for easy printing.

The latter was – simply by nature of its envisioned task – extremely resource-intensive. All image files had to be loaded from disk, embedded into the PDF, compressed and served. The option had thus been available for fewer and fewer objects. Where it was originally available in case of any object with more than three images, it was later limited to objects of less than 40 images. As such, its availability was increasingly hard to communicate clearly, while its usefulness was relatively reduced with the introduction of a new download option for all images of an object. Its natural resource-intensiveness remained a problem however, and as scrapers will click any link they can find, this type of PDF generation continued to be used quite regularly (every few seconds before the recent surge in bot activity). As of last week, the functionality has been entirely removed.

The “data sheet” PDF generation has been further limited as well. As stated in the previous blog post, its usefulness is significantly reduced with the introduction of a print stylesheet (you will get better results simply pressing CTRL + P on an object page and printing the page to PDF). Nevertheless, it remained rather popular and has not been removed entirely. To reduce its impact on server stability, we however further limited its availability: If the server load is any higher than comfortable, the PDF will not be generated and an error message will appear. If the load is high (up from around 70% of comfortable) and the user’s browser language is not the default language of an instance of museum-digital, the same error message will appear.

Failed Search Pages

If a search query for objects fails, users are forwarded to a failed search page, on suggestions for alternative search queries are made. This is essentially the same as Google automatically suggesting corrections when search terms contain typos. Identifying the alternatives and offering previews for each is not free. As it is simply suggestions, the benefit or general accuracy of the suggestions fluctuates from case to case.

Now, looking at the logs, we had a large number of queries for non-existing entities – obviously scrapers who were trying out different IDs after analyzing the URL scheme. Each of those queries was executed and then forwarded to the failed search page, triggering the loading of suggestions and previews and thus further using resources on the server for little benefit (besides getting more links to scrape). We have now introduced a similar logic to the limitations on the data sheet PDF generation. Suggestions and previews are only generated when server load is comparatively low, with non-primary language users being slightly disadvantaged vis-a-vis primary-language users in an instance.

Timelines

Timelines remain popular – and a problem. A very common type of query we would see in our logs would combine timelines with searches by start and end date. This was likely due to another possible loop of endless URL generation for scrapers – specify a timeline until it forwards to search pages for a given timespan, then open the timeline for that timespan. Exactly that behavior has now been made impossible. If a search by a timeline (“start after”, “end before”) has been set, timelines will not be offered in the sidebar anymore. Trying to generate them for such a search using URL manipulation or the API will return an error page.

Search: Cleanup, Image Search & Checking Entity Existence Early

A more messy way of optimizations hit the core of the object search. In around 2021, we introduced a new search logic. Almost all pages relying on the core search logic – search overview pages, maps for objects, timelines, were adjusted to work with the new logic. The only exception from this was the image search. Still, as the new search logic re-used some of the old search logic’s functions, we kept both as separate classes, which grew over time. Simply loading the new search logic took about one ms (without OPCache enabled, measured through PHPBench). This sounds like little, but hints at a lack of modularization of the code and gains relevance with many unpredictable requests with servers automatically spinning up and down.

And indeed, in writing the new search logic, we did not modularize thoroughly HTML generation, query building and database querying. With last weeks updates, there are now separate classes for each of these and functionalities relevant only to the old search functions have been moved to class managing the image search logic. This reduces startup time for only the new / main search logic by about half (ca. 0.6 ms).

Second, we reduced the available search options for image searches. The remaining search parameters are either those actually relevant to the images or those linked to the controlled vocabularies. As a positive side effect, this also solves some issues in communication: Making it legible what the difference between searching images by their own license and by the license of (unrelated) metadata of objects the images are linked to is, is complicated.

Finally, as stated above, the logs revealed a lot of queries for objects linked to e.g. either entirely non-existent places or places that are not linked to any object in the instance of museum-digital altogether. When a place or tag is queried, we hence check whether there exists any public mention of the entity in the current instance of museum-digital during query building. If there is no link at all, it is clear early on that a more detailed (i.e. costly) query combining the search by that entity with other parameters will not return any results.

The Current Situation

All these improvements help, but a look at the current real-world numbers is warranted. On the one hand, the database server now often falls down to half or even less of the expected server load. This is a positive sign for system stability outside of peak times.

On the other hand, there are noticably spikes in the morning (around 10:20 in Germany) and in the afternoon (starting around 5 p.m.). The spike in the morning is likely related to the start of workdays and has led to the server falling over multiple times last week. This can likely be fixed only with a further tuning of the PHP-FPM settings. The spikes in the afternoon and early evening on the other hand remain hard to explain, but are altogether much less critical.

We’re on it.

This content is licensed under a Creative Commons Attribution 4.0 International license.

Generally, the pace and type of development seems to have changed this year. Rather than doing many small improvements all over the place, there is less but larger and more labor intensive changes and new features. See for example the versioning in musdb (January), the tool for automating imports based on what others call a “hot folder” and the sort option to sort objects by their images’ aesthetics score (both presented here in March), and the new tool for suggesting formulations for object descriptions using large language models (June).

July

Frontend

• Translation of the software to Hindi and Telugu
• Grouping of tags by their relation to a given object
  If an object is linked to more than 10 tags, the tag list of object pages quickly starts looking unorganized and messy. In such cases, the tags will now be displayed grouped by their relationship to the object (thus far: general tag, material, technique, object type, display subject)

musdb

New Features

• Export option for the specific LIDO as expected by the German Digital Library’s “colonial contexts” portal

Improvements & Changes

• The minimum length of fulltext search terms for object search parameters is now visibly enforced in the extended search user interface
  To not overly burden the fulltext search server, any term in a full text search in musdb needs to be a minimum of two characters long. Thus far, shorter full text search parameters were simply ignored. This certainly was confusing at times. Since June, attempting to perform search queries with shorter search parameters is made impossible by a check in the extended search overlay.
• Reception history of objects: Statements of the relevant position within a source can now be 40 characters long
• Transcriptions
  • May now be up to 4000000 long
  • New fields: Notes on the transcription, status, aims

Bugfixes

• Fixed a bug in the batch editing of specific visibility flags for data fields on the addendum tab

Juni

Frontend

• Performance improvements
  • Object search now runs without a connection to the full text search server if no full text search parameter is set
  • If multiple search parameters for an earliest / latest time have been set, they are parsed and combined before being forwarded to the database
• Improvements in the deletion of temporarily created PDF files (PDF export)
• Navigation has been translated to Tamil

musdb

New Features

• Recipient of deaccessed objects can now be linked from within the address book
• New tool for AI-aided formulation o object descriptions (based on existing other metadata)

Improvements

• Object search now runs without a connection to the full text search server if no full text search parameter is set

Importer

• The CSVXML parser has been extended to cover new event types and markings
• Object groups automatically generated to group all objects of an import can now receive a description as set within the import configuration

nodac

The list of selectable languages for the navigation of nodac has now been restricted to those in which there is actually a complete translation

• A pixel (px) is – in theory – fixed to the single dot of a screen. It thus depends on the resolution of the screen primarily, at least in grey theory.
• A point (pt) is a single point on a printed sheet of paper. Its roots lie in the DPI setting of the print job.
• em are relative to the font size. If nested elements come with font sizes determined in em, the scaling is appropriate nested (a headline in a generally enlarged section will be further enlarged than a headline in a regular section).
• rem is relative to the primary font size of the document.
• vh and vw are relative to the total screen height and width respectively. 10vw are, e.g., 10 percent of the total width of the screen.
• Some others, like cm or mm exist, but are less used – especially when working outside of print contexts.

Being relative to the font size and nest-able, em values are a great fit for setting sizes of page elements and we thus use almost exclusively use em and rem values for borders, font size, etc.

To make a website responsive, CSS offers a functionality called media queries, with which one can check for characteristics of the browser window. This, obviously, concerns primarily the size of the window, but can also extend to issues such as the activation of a dark mode (if you see this page with white text on a black background, that is the consequence of a media query).

Now, we had also been using em values for probing screen properties using media queries, and the results were far from ideal. Especially phones from different manufacturers provide very different em values even at roughly similarly sized screens. Thus, it becomes hard to determine, if a user really is using a phone, a tablet, or a full sized PC.

Ironically, it is exactly pixel values, that work much more reliably. In theory, a media query asking if a screen is less than 768 pixel wide should definitely be answered negatively in the case of a current flagship phone with roughly 1400 pixel width. In reality, browsers scale down the page and will answer positively, accepting that phones should be presented to CSS as less than 768px wide.

We have now switched to using pixel values for running the media queries to determine the target page layout. And the results are much better, than when we used em values.

See also

W3C: Web Style Sheets – CSS Tips & Tricks: em, px, pt, cm, in…

This content is licensed under a Creative Commons Attribution 4.0 International license.

Either way, we have also worked on some new features: A dedicated area for markings (stamps, watermarks, etc.), one for references and reception (e.g. the publication history of an object), and entering uniform information for the field “object type” has gotten easier. Most importantly, we have added some new event types to better meet the requirements of provenance research.

Markings

Especially for art historians, a precise recording of information on different types of markings, that have been added to an object for indicating authorship and ownership are essential. Be it stamps, watermarks, signatures. Thus far, information on these could only be added using the unspecific field “Inscription” in museum-digital. The new “markings” area (to be found at the bottom of the tab “Addendum” in musdb) aims to tackle this problem. The term “markings” is deliberately chosen to be wide in meaning to also emcompass logos and other types of markings that are more relevant for e.g. technical collections.

If a user of musdb wants to add a new marking, they can scroll down to the bottom of the “Addendum” tab and find an accordion button. A click on this button toggles open the area for adding new markings. First, the tye of the marking and its position on the object need to be selected. A transcription must also be provided to safe. Additional information to add are the creator, a possible mentioned time on the marking, its size and a note.

Creators and times are drawn from the centralized controlled vocabularies for actors and times respectively. Hence, they can be entered by tiping the given term and then selecting the correct value from a drop down menu appearing as one types. If the term does not exist yet, it can be added by clicking on the “+” symbol right before the text field.

Once a marking has been entered, it is public for now, even though we consider adding an option to hide markings even when their object is public.

Reception and References

With “reception”, there is a whole new tab in musdb for linking objects with sources. These sources come from a new controlled vocabulary for sources, which is used together by all institutions and runs on a similar level as actors, times, places, and tags. Reception now describes all instances, in which an object is referenced in a source – e.g. if a painting was printed in a book, or a photo published in a newspaper. Similarly, the discussion of a custom-made car in a car magazine may be recorded here.

The counterpoint to this is the “is reference” area just below reception: This can be used, if the object itself references a publication. E.g., a painting may be drawn after a scene from a book.

Like markings, reception and references are by default public.

Object Types From a List

museum-digital traditonally treats object types as a free text field, while many – if not most – museum databases provide controlled lists of available object types. We use keywords / tags to provide the same functionality, but it surely makes sense to also have some level of uniformity in the field “object type”.

Hence, there is a new aid in adding uniform object types when adding a new object in musdb. As one types in the field, a drop-down list of similarly-phrased keywords appears. Clicking on an entry from that list enters its value into the “object type” field and (if one is adding a new object) automatically links the object with the selected tag.

JSON-LD Metadata for Image Licenses

Another minor improvement is that single image pages now feature JSON-LD markup describing the respective image license. This enables especially Google, to provide users with a more detailed view on how to attain the license to use a given image file.

Importantly, images from museum-digital now also appear in Google Images if one opts to only view Creative Commons-licensed works.

Events: New Event Types and References for Events

The probably most important change are the changes made for entering events in museum-digital. One the one hand, important missing event types have been added. These are:

• Assembled (to be used for objects, that have been assembled from parts. E.g. a car)
• Auctioned
• Bought
• Owned
• Sold
• Restorated
• Damaged
• Destroyed
• Lost
• Edited (for books and publications, that have an editor)

As can be seen from the list, most of the new event types are related to provenances (for all museums that want to use these event types: Please (!!) ensure that you have cleared the necessary rights to publish this information, as events are public). To make events at museum-digital more closely meet the requirements of provenance researchers, a missing piece were also sources for events, which can now also be added.

To do so, the addition of sources for events must first be activated on a museum-level. To do so, a museum director or user with similar permissions must first navigate to the “institution-wide settings” page in musdb, and activate the event sources there.

Once they are activated, an accordion button appears on event editing pages (accessible by clicking on the event type of an already added event). Here, sources can be linked using a drop-down menu similar to the ones used in the features described above.

Finally, with options to add event annotations and sources, there was an increased danger of cluttering the object page in the frontend of museum-digital. Hence, we redesigned the display of event annotations to now appear in a tooltip – if an event has an annotation or a source, an “i”-symbol appears next to the respective element for the event type. Hovering over that element makes sources and annotations visible.

Beitragsbild: Caravaggio: „Der ungläubige Thomas“ (CC BY-NC-SA @ Stiftung Preußische Schlösser und Gärten Berlin-Brandenburg / Murza, Gerhard)

This content is licensed under a Creative Commons Attribution 4.0 International license.

To simplify the management of objects with many images musdb offers the option to upload multiple images at a time. This option is however limited by global server settings like the maximum size of uploads and the maximum number of files that can be uploaded in one go.

To understand the problem, it’s important to know that most tools of museum-digital are written in PHP. Unfortunately, PHP has a tendency to gloss over potential errors and continue without enforcing that the problem to be solved either by the programmer solving a bug or by the user not doing a certain action.

The maximum number of files that can be uploaded simultaneously is a flagrant case of this: Rather than inform the user of their overstepping of the set upload limitations, PHP accepts files until the number of allowed files per upload is reached. Any files after are simply ignored.

Say a user attempts to upload 30 pages of a book in one go. The first 20 will be uploaded correctly, while the last 10 will be skipped. In the worst case, the user doesn’t even notice the missing data. In a better case, they are left confused.

We have thus added a warning that stops the user before submitting the upload form when trying to upload more files than allowed.

This content is licensed under a Creative Commons Attribution 4.0 International license.