IT Security - It's Complicated

Joshua Ramon Enslin
AI generated banner image depicting a key over a synthwave-inspired digital grid.

Today, more or less the same question came up three times. In different contexts and with different aims, the question was posed, what IT security, privacy, and data protection look like at museum-digital.

Now, security is not something has or does not have. It is a collection of perspectives, approaches, best practices etc. There is the security of a server, the security of single processes, the security of users’ data and accounts, the security of users’ computers, the security of the local network the user is on. Et cetera.

In this post I will try to give an introduction to some IT security topics relevant to museum-digital. The list of topics cannot be exhaustive, but should cover the most important ones and provide a reasonable entry point on how to think further about IT security at museum-digital in specific and also IT security in general.

Definitions

Confidentiality, Integrity, Availability

The basic aims of IT security are to - at all times - maintain the confidentiality, integrity, and availability of data and services. Confidentiality means that only exactly those users who have been given permission to view an entry should be allowed to do so. Maintaining integrity means that it can be assured, that data is not wrongly altered, and that changes are transparent and trace-able. Finally, availability means that the data or service should be reliably accessible to those permitted to access them at any point in time.

Anything that impacts confidentiality, integrity or availibility negatively is a relevant issue in terms of IT security. Especially where only availability is impacted, an issue might however not be grave (e.g. a specific site not being accessible for 20 seconds). Or it might be deliberate: If a user tries to try out their password time and again, it is best practice to successively lock down the service’s login to prevent a serious - grave - impact on confidentiality and integrity.

Locking down the service means that the user who tried different passwords gets locked out (negative impact on availability). As such blocks usually (also) happen at the IP level, other users who share the same IP will likely also be blocked at the same time. Availability will definitely be unduly impacted for those.

In the example, availability was sacrificed to ensure confidentiality and integrity. The details of IT security are always a matter of negotiation, interpretation, priorization. What is not negotiable is that they are important.

Perspectives and Basic Approaches

With some aims defined, a second basic question is that of perspectives. The simplest way - and one that pretty much anybody can do - is to think about how, in a given process, confidentiality, integrity, and availability could be disturbed. Once a possible way to break into a system or cause an interruption to its service has been identified, identify or ask somebody how the issue can be resolved / prevented from causing issues.

In almost each case, there will be different ways that will all at least temporarily solve the issue. Imagine you bought an old house, and next to the main door, there was an old, moldy door without a key, which anybody could use to enter the building. You could get your bricklayer of choice to replace the door with a wall. This would be the most effort, but it would undoubtedly secure the house properly (where it comes to the second door at least).

You could take a slab of metal, screw it into the wall behind the door, so that unwanted visitors would be blocked right behind the moldy door. This approach doesn’t fix the issue (moldy door), but isolates it and ensures that even if it is used, nothing bad will happen.

Finally, you could simply build a large fence around the whole building to keep unwanted visitors out. This will usually be effective against most potential evildoers and likely be cheapest, as you planned to build a fence anyway - but realistically, people will always be able to climb over fences.

This again is a gross oversimplification, but it may serve as an illustration to three basic approaches how to fix IT security issues: One may fully prevent them, e.g. by removing a buggy software or fixing the bug oneself. One may isolate problems, to prevent their exploitation (sandboxing), or one may prevent potential attackers from reaching potentially problematic functionality in the first place (e.g. firewalling).

Finally barely program works alone, and “having been owned” can mean different things. A hacker might get administrative access to a server - all would be lost. A hacker might take over a system user account on a server - they would be able to do anything that user account can do. How grave of an issue that is depends on the permissions that system user has. A hacker might find an SQL injection vulnerability - be able to execute arbitrary database queries within an application. Again, how grave of an issue that is depends on the privileges of the database user account. Finally, a hacker might take over a certain user account within an application. They would have access to whatever that user has access to.

IT security thus concerns a number of different levels. Each of these levels of issues can be viewed in isolation, but one breach on a low level of privileges increases the chance of “privilege escalation”: If a hacker compromises an application user of an internal application, it is likely that that user has more permissions than an unauthenticated user (e.g. uploading files). Thus, the “attack surface” - all the options one has to perform an attack - may be significantly increased. If the hacker then finds a bug in code only available to logged-in users, they could potentially use that to gain access of the system user running the application. With that, chance is that they can execute arbitrary code and they likely have an easier time getting administrative access.

Finally, and that should be apparent to anybody who has used computers in the last 20 years: Security can be in conflict with usability. External demands exist and need to be part of the discussion - if not, there won’t be anything to secure.

Principles

Almost in passing, the above also hints at some basic security best practice. On the one hand, complexity is inherently bad from a security perspective. The more code an application has, the more code can have bugs. The more administrative configuration options an application has, the higher the chance to misconfigure something.

Second, access should be given to only those resources and capabilities really necessary in a given context. This is commonly called the “principle of least privilege”. The larger the scope of an application - the more different things it should be capable of - the higher the complexity and the larger the privileges it will need. Splitting an application into multiple smaller, targeted applications allows the reduction of complexity and required privileges.

Finally, one needs to account for all layers of one’s application. Privilege escalation as describes above is much harder, if each application is locked down as far as possible (defense in depth).

A more exhaustive list can be found here.

museum-digital: Scope and Requirements

museum-digital is an initiative that has always been aimed at making digitization in museums and related institutions as simple as possible. For that, software is written and part of simplicity is museum-digital being free of charge. Meanwhile, the scope of the activities around museum-digital has grown significantly: From building platforms for the collaborative publication of museum data, we grew to build a collection management system, a solution for digital exhibitions, a linked open data platform and editing system for controlled vocabularies and more.

From that, the essential constraints can be gleamed:

First, we run some largely internal tools (Gitea, Nextcloud, Jitsi), whose source code we do not control, even though each of those is open source software and we could theoretically fix issues ourselves if the need should ever arise. Otherwise, all services running under the name of museum-digital are controlled and written by ourselves. This is important, because it means that we are in a good position to think about questions of systems administration (hosting and configuration) and software engineering (the actual code) as integrated problems.

Second, museum-digital is running on a shoe string budget. If we can replace financial cost with effort and following best practices, we do.

Third, an explicit aim of all software offered in the context of museum-digital is ease of use. As such, we can harden our internal systems a lot. But we are contrained in enforcing default account security beyond any level users consider “normal”.

Finally, museum-digital has always been a web-based collaborative undertaking.

Hosting

Hosting Scenarios

Again, it makes sense to try to list some categories of how an application might be run or hosted before trying to discuss them.

  1. An application might be run on an air-gapped system without any link to other devices.
  2. It might be run on a PC within a home or institutional network, with no direct way to access it from other devices (as per firewall settings).
  3. It might be run on a local server, accessible by users within the local network, but firewalled away from the global internet. Remote access may be provided using VPNs.
  4. It might be run on a remote server that one manages oneself. Anybody can access the server.
  5. It might be run on remote servers that other people manage, wherein one only rents specific services. Anybody can access the service.

Access in this case is to be understood as being able to connect.

Now, what are the implications of each?

Airgapped local PC

In theory, an airgapped PC is the most secure option. If an attacker cannot even reach the target of an attack in the first case, an attack is unlikely. An with that, a successful attack also becomes much less likely. It does however pose its own challenges in other regards.

As the PC is not connected to other devices, it can only be used by a single user. Doing web research or linking and using external data sources (e.g. external controlled vocabularies) are impossible on the same device. They can be used on an external device, but then any way of automation is precluded (including copy and paste).

Finally, keeping the device truly airgapped is hard. How are updates installed without an internet connection, how are backups done (and how well can the system be reinstalled from a backup)? And if one is talking about museum data: how are object images and other files created by external devices moved onto the PC? Chance is that each of these processes creates a breach in the air gap.

In summary, truly, properly airgapped systems are great for security. Keeping them both updated and actually properly airgapped poses technical and organizational challenges. In terms of usability in an interconnected world, they are the worst option by far. Running them properly is thus very costly , and be it just for the labor costs of frustrated coworkers.

A Single Local PC

Running the application on a single local PC with proper firewall settings shields against direct remote attacks architecturally. In contrast to the airgapped PC, updates, backups and basic automation like copy and paste work well.

If the PC is not exclusively used for the application, the attack surface of the system is however significantly increased by the other programms the computer might be running. If the PC running a collection management system is hacked using an exploit against Outlook, the collection management system is likely compromised, too.

Finally, similar to the airgapped solution, running the application on a single PC without remote access procludes collaboration. Only a single user sitting in front of the PC can use it.

A Local Server

Moving on to server-client architectures, one might host an application on a server accessible only from within the local network. This is generally a great solution, as it means that the application can be properly isolated from the rest of the (ideally few) other things running on the server. It is isolated against remote access, etc. Backups, updates etc. are not curtailed by the architecture.

The downsides lie with the necessary infrastructure. One needs a properly configured local network, a local server, and people to continually keep the systems and their configuration up to date. If one wants to allow limited remote access, one needs to set up and properly configure a VPN - adding complexity and attack surface to the system. All of this costs money and labor.

Finally, there are constraints on collaboration with anybody not regularly part of the organization. Giving them access means configuring not only new user accounts within an application but also configuring specifically tailored access using a VPN.

Maintaining this infrastructure can be outsourced, but outsourcing is comparatively expensive, as service providers likely need to give individual support for the individual setup of the institution. This stands in stark contrasts to the cost saving benefits centralization can offer.

Online Hosting on a Publicly Accessible Server

Hosting an application in a web facing way on servers one controls oneself further increases one’s exposure to potential attackers, as direct connections to the servers can be set up by anybody. Flawless configuration, code, and update regimes thus become all the more important.

On the other hand, online hosting allows joining hands with others (and be it just by buying the tools of the same Software as a Service provider) and thus drastically reducing costs. Also, software vendors and service providers whose service one directly uses can potentially apply updates independently, thus making timely updates as unproblematic as possible.

Given online access, tedious VPN setups for external collaborators are not necessary and collaboration is comparatively simple both within the organization and with external partners.

Online Hosting Using Cloud Providers / Renting Services Instead of Servers

This essentially has the same implications as renting full servers when it comes to the attack surface. The differences between cloud services versus single rented servers lie in technical details and payment structures.

Technically, cloud services rely on distributed computing. Rather than storing data on a single hard drive, it is shared between many servers (potentially with redundancy, though that depends on payment structures and setup). The same goes for other resources such as CPU access. In general, this reduces the risk of a single hard drive failing / a single machine breaking down. Some data may be lost when a server breaks down, but not all.

Also, as cloud hosting always works with distributed computing in mind, scaling up as demands grow is incredibly easy: One simply buys more resources. Growing a service from being hosted on a single server to being run on two at the same time on the other hand is hard.

In terms of payment structures, cloud infrastructures can profit from maximum centralization. Amazon / AWS (cloud) has a better negotiation position than Hetzner (mainly renting out servers) when it comes to hardware prices. Both can buy in bulk and pay much less than a single museum. On the other hand, both want to make profits.

Typical cloud hosting pricing starts very cheap and gets more expensive quickly as one scales up. If one wants to safely store one’s data with no expectation of anybody accessing it, Amazon’s S3 data storage is a great offer. If one’s files are expected to be accessed some millions of times a months, it becomes expensive quickly.

Essentially, cloud hosting is paid dynamically according to demand, rented servers are paid at a set price. If one underutilizes available resources, rented servers are more expensive. If demands are high, chance is that rented servers are cheaper.

Finally, cloud services pose dangers of lock-in. To actually benefit from the distributed computing environment, one needs to use the cloud service’s specific APIs rather than standard components. Leaving a cloud hosting environment may thus require significant changes to the application’s code. A service run on a web-accessible server can with little effort be moved to a similarly set up inhouse server. Moving a cloud-hosted infrastructure fully inhouse is a much larger challenge.

Questions and Hosting Methods

Outsourcing, Bundling Resources or Working Alone

A first relevant question is whether one has the resources and expertise (or the money to pay for expertise) to host things inhouse. If so, managing one’s services inhouse is potentially the most secure option, as network segmentation and firewalls reduce the attack surface significantly.

With proper care, updates, backups etc. can be well done and in theory this is the best option available. On the other hand, this only really applies if one’s inhouse hosting is strictly and always maintained. Does your inhouse admin install critical security updates on Christmas eve?

For most institutions, it makes sense to bundle resources in some way. Using off the shelf software rather than developing one’s own, locally developed services usually results in more secure and more usable software being used.

If one uses a centralized service provider (think Dropbox, Google Docs), the service providers can more efficiently automate things like backups and ideally know better to secure their systems than one does locally. On the other hand, they are a more attractive target for attacks, as they hold much more valuable data than any single institution would.

The same goes for hosting scenarios: The more is bundled, the cheaper it may get, and the more likely a good configuration and architecture becomes. There is no doubt that people at Amazon will update their systems on Christmas eve. On the other hand, again, breaking into centralized services is incredibly attractive for potential attackers and one needs to trust one’s service providers. Hosting one’s applications on Amazon’s infrastructure means that Amazon can read one’s data if it wants (note: I’m using the example of Amazon here as it is by far the largest cloud provider around. The same goes for other large cloud providers).

On-Site or Off-Site

Another question to ask oneself is about physical security: If one runs all one’s applications on a server in the basement and there are no backups stored outside of the premises of an institution, flooding or other natural hazards may result in full data and service access loss, too.

The same issue applies if one hosts one application on rented external servers in a single data center or cloud hosting location, albeit with limitations, as data centers usually have to set up mitigations against natural hazards. Yet, as can be seen with Yandex in Russia these days, physical attacks may still be strong enough to simply obliterate a whole data center or cloud hosting location.

Backups and Redundancy

Here, backups and redundancy come into play. Backups mean that one’s data (including configuration and ideally source code) is stored on a second (third, etc.) device. Redundancy means that one maintains one’s services e.g. in multiple installations on different devices, with the database being regularly synchronized. This way one can simply replace the primary setup with the primary one.

In both cases, keeping a copy or installation in a different place from the primary hosting location is essential to keep data or services available if there is a large-scale outage in the primary location.

Especially with backups, keeping the backup physically as far away from home as possible makes sense. But then, securing one’s backups against access by third parties becomes an issue - especially if the backup rests in different jurisdictions. As such, backups should always be encrypted.

Encryption

Generally, encryption is always an issue when different services communicate (and be that a database-based application with the underlying database). Where data is transferred or stored without encryption in an environment that is not strictly firewalled, it can be sniffed. It can be considered compromised.

This also applies to most local networks. Consider: How hard would it be for an outsider to enter your building and plug their laptop into your local LAN?

Encryption can be used to ensure the confidentiality of data either within in a given context (e.g. while it is transferred from one computer to another) or in general (encrypting a file). In each case, it is to be considered in which context the data needs to be accessed, if it is accessible unencrypted at any point (at least when somebody reads the file) and who has the keys to decrypt it. If an unencrypted file is uploaded using an encrypted connection to a publicly accessible directory on a server, encryption during the update gives no benefit to the confidentiality of the file’s content (though it is still meaningful to protect the credentials used for uploading and it does help integrity). If one has a list of files that need broad, regular access on a server, meaningful encryption is hard. Storing the data right next to a decryption key provides barely any benefit over leaving them unencrypted. On the other hand, it does make searching - especially in-exact search queries - across the data harder

Application Hosting Considerations

Target Audience

Also, again, usability and use must be considered. Here, the target audience and scenarios of use are of utmost importance. Should the service be available to the general public? Should it be available to external collaborators? Or should it only be available to logged in co-workers within the organization?

As such, it makes little sense to host one’s website on a local network. It’s meant for access by the general public.

If one’s organization doesn’t work with external collaborators at all and has a proper VPN setup and a dedicated full-time systems administrator, it does however makes sense to keep one’s file storage solution hosted inhouse. Some things like accounting should likely always be hosted inhouse.

If one has exceptionally high security requirements and is a credible target for targetted attacks by nation state actors (think: if you are a critical infrastructure provider like waterworks or energy systems), airgapping your systems makes sense (and is likely legally required in at least some instances).

Server Location and Jurisdictions

Finally, if one hosts or stored data outside of one’s own premises, locations and jurisdictions should be considered. In most countries, state agencies can legally access the data of foreigners and, given the necessary legal background (e.g. an ongoing criminal investigation), citizens.

Hosting or storing data in a jurisdiction one is not a citizen is thus an almost automatic breach to confidentiality. In some countries, that extends to hosting or storing data with companies with a business seat in the country: Notably, the US CLOUD Act gives US state services access to data stored on the infrastructure of US American companies. In a strict reading, anything stored unencrypted on a US American company’s infrastructure is compromised. That includes Microsoft’s M365.

Hosting Setup at museum-digital

Now, what does hosting look like at museum-digital?

We are broadly following hosting scenario 4 as outlined above. Almost all of the services we provide are build with varying degrees of public access in mind. The frontend platforms, the digital exhibition tool (themator) and md:term are meant for access by the general public. musdb, nodac, etc. are build for access by logged in users at distributed locations, from (in the case of musdb) thousands of institutions.

With the recent rise of AI scrapers, there are millions of web requests to files published on the frontend platforms every day. Hosting at this level on cloud services would likely be much more expensive than hosting on rented servers. Prohibitively so, given the available budget (currently we spend less than 200 Euro / month on hosting).

In the context of museum-digital, we run three servers, all in Germany and rented or donated from German hosting providers or public institutions. Out of these, each fulfills a non-interchangable role: A primary server hosted at Hetzner in Saxony hosts the production systems, including databases, web server, and application code. On this server, we run exclusively software written by ourselves (using a shared, rather minimal underlying software stack; see below).

A second server, hosted at digiS / the Zuse-Institut Berlin, is used as an image server and a bridge into digiS’ backup infrastructure, incl. backups on tape. Recently, this server has also been used as a bridge to the Zuse-Institute’s AI infrastructure for the automatic generation of suggestions for object descriptions. As such, the server runs almost no public-facing application code. It runs nginx as a web server to play back image files and, using the built-in image resizing plugin, dynamically resize images. Forwarding queries for image generation is done using a very small application written in Go.

A third server (hosted at Strato) is used to run external applications we mainly use for coordinating the core team and internal processes: gitea, Nextcloud, a mail server, etc. The server also hosts some public-facing static sites, such as museum-digital’s blog.

Server 2 (digiS) needs privileged access to the primary server (Hetzner) to be able to fetch database backups and image files. The primary server (Hetzner) needs privileged access to server 3 (Strato) for fetching updates to application source code managed in gitea. In each of these cases, access is limited to what’s necessary. For database backups, we run a daily SQL dump of each of museum-digital’s databases. During the export, the data is GPG-encrypted. One path of access from server 2 to server 1 has access to only one directory containing only these encrypted database dumps. The server at digiS thus never gets access to unencrypted database content. Using a separate path of access and thus system user, it has read-only access to the image file storage on the primary server to be able to synchronize image file storage between the two servers. The images are stored unencrypted on both servers, as server 2 is also used for serving them to the public. Remote access is provided using SSH connections (git over SSH, rsync over SSH).

Our backup strategy is good. We do however lack redundancy in directly available infrastructure. Without a mirror server, bringing musdb back at full operationality if the primary server were to fully break down would likely take many hours or even some days (depending on how fast we can upload ca. 4 TB of media files to a new server).

Also, with our choice of software, we are not exposed to non-open source software beyond the hosting providers’ level. We are not directly exposed to any cloud services on any level we can track, though we obviously also rely on their infrastructure and contributions for keeping things updated (just look at the list of companies contributing to Linux).

Application Security: Server-Side

The different hosting scenarios come with their specific architectural security implications. But even in an airgapped system, it is worth considering what damage an attacker can cause if they do get access to the infrastructure. Reducing the potential harm comes down to restricting access to internal technical resources as much as possible. Ideally, an attack can thus be contained at a low level of privilege. Second, to reiterate, any moving part and any complexity comes with an increased attack surface.

This part will be concerned with the security of a server and backend code. Frontend code and account security will be discussed later.

Static Sites

When thinking about a new service, especially online, it’s hence worth considering how far the service can be simplified from a technical perspective. If one’s service is e.g. a blog or a rarely changing website with no interactive components, it may make sense to publish it as a static site. Meaning, that the page is built using whatever backend one likes internally, compiled to folder of static files (HTML, images, etc.) and then published using only those static files. As such, the server needs to run no application code or database - only a web server that parses the requests and responds with th appropriate file.

In general, this is the most maintenance-free, least complex and most secure solution possible. Where there is almost no code being processed, there is almost no opportunity for bugs and security issues. As there is no application code being processed however, dynamically responding to specific user requests (e.g. responding appropriately to a contact form) is impossible in this scenario. Organizationally, it may also pose a challenge, as those who are to update the website need to understand the logic of updating a website in multiple steps (edit, compile, upload).

Components of an Application

Once one needs interactivity of any type, from a contact form used to send mails to storage of input data in a database, one needs some level of application code being executed and exposed on a server. The code itself needs to calculate, but depending on the requirements of one’s system it might also depend on external resources - a database or the file system for storing data, web access to query external data sources automatically, etc.

In many cases (like databases), these are handled by tools external to the application itself and come with their own configuration. On the other hand, an application itself likely consists of the core code of the application and external modules (implementing your own cryptography is a bad solution).

The smaller and more targeted an application, the less external tools, capabilities and resources it needs. But if its scope is expansive, it might need to rely on a range of external tools, services, and capabilities.

Just to illustrate: A typical search query in a web-based collection management system (an inherently expansive system in scope) may thus look as follows:

  1. The user / the browser opens a URL. The URL reflects the current search filter.
  2. (The browser figures out the IP corresponding to the domain, a connection is established and the request is sent to a server)
  3. The server receives the query using a web server such as Apache or Nginx. Based on the queried path, it is decided if the query is one to a static file or one to be forwarded to application code. If a static file was queried, it is directly returned by the web server. Otherwise, the web server forwards the request to the application (it acts as a “reverse proxy”).
  4. The application itself is running either its own web server or a socket. Both are directly accessible only locally. External access is only possible via the main web server application. Based on the queried path, the application identifies that the request was aimed at searching and listing objects. To identify suitable results, the database, a search server or both needs to be queried.
  5. The database runs the query and returns results to the application.
  6. The application takes the result from the database and reformulates or embeds it into the target form (e.g. an HTML page) and returns that to the web server.
  7. The web server forwards the application’s output to the user / browser.
  8. The browser takes the HTML, parses it and renders it into a human-readable page.

This would be a very simple setup without caching. And yet, even just from the server perspective, there are four independent components that need to be kept configured and updated: OS / network stack, web server, application, database. Caching layers etc. are missing in this setup.

Programming Languages

The application needs to be written in a programming language, and the choice of programming language carries implications for typical hosting and execution scenarios. Regardless of the programming language, a rough application-level specification of privileges can be achieved by running applications using dedicated system users. Each programming language however carries its own implication for how privileges can be specified and successively dropped (or not).

Second, as one likely does not only host a single application, it makes sense to consider how one programming language fits in with all the other ones other applications one hosts. If one is already running three Java applications, has staff that have expertise in hardening Java hosting environments, then adding a fourth Java application is simpler than adding one written in PHP. The same goes for libraries or pre-packaged, external code components: It makes sense to use similar libraries for similar purposes across different projects or applications.

All programming languages come with their own implementations of e.g. how a file is written or a web request is performed - be it as part of the language itself or as an interchangeable component / library. Just like the application itself, the underlying programming language version needs to be kept up to date. If one writes or hosts software written in a compiled language, the application needs to be rebuilt do run with the latest patches. Interpreted languages similarly need to updates to the interpreter (the program that actually runs programs written in a given programming language).

Finally, no programming language is inherently secure or insecure. They differ however in how much they protect programmers from writing bugs. It is harder to write code in Rust than in PHP. It’s much simpler to create buggy code in PHP.

Isolating Processes, Users, Use Cases

To summarize, in a web context, there are likely a number of separate but interdependent components that make up a working infrastructure. A large part of securing them comes down to isolating them - also against each other. This can be done by running them on different hardware, virtual machines or such. It can be done (less exhaustively) by running them under different system user accounts with restrictively set permissions. Likely, the applications themselves can be configured to further restrict access and capabilities to whatever is specifically necessary. The application itself can further set itself limitations. Similarly, directly on the application level, one needs to take care to isolate access between different users on a technical level.

museum-digital

All services hosted in the name of museum-digital are either published as static sites or use a similar stack of underlying software:

Use caseComponent
OSUbuntu LTS
Web serverNginx
Backend programming languagePHP (one exception written in Golang); no off-the-shelf framework
Frontend codeVanilla Javascript / Typescript (no off-the-shelf framework)
DatabaseMySQL
Search indexManticore
Cache serverRedis

Which of these services is actually used depends on the requirements of a given application. The less it needs, the less are used.

PHP’s default philosophy is to separate the execution of each request - drop variables, database connections, etc. as soon as a request has been answered / a script has finished execution. Properly configured, this generally helps in not leaking data between requests of different users (on the other hand, it’s generally wasteful and inefficient in terms of resource usage). Different services within a subdomain (e.g.) frontend platforms vs. musdb are on separate PHP-FPM setups using dedicated configurations with restrictively set permissions and capabilities. The choice of PHP as a programming language on the other hand also means that restricting esp. available functions within an application or the PHP setup is hard: PHP comes with a huge number of built-in functions and requires one to blacklist functions one does not want to use rather than whitelist the few one actually needs in a given use case. Limiting access to folders within an application can be done and can be done successively, but does not allow the application to specify if it needs read-only access or read-and-write permissions.

Specific database users are set up for each application, with more precise permission management taking place within specifically musdb and nodac. Say, if a user managed to find a SQL injection vulnerability in the frontend, they could only read data (potentially breaching confidentiality). If they found such a vulnerabilty within musdb, the database user has write permissions to the whole database of that instance of musdb. This would put confidentiality and integrity of the whole region at risk, which is also a downside of the regional structure of museum-digital. The new architecture of musdb will in the near future enable us to more specifically restrict permissions on a per-API basis.

We take care to strictly validate all incoming data and sanitize outputs properly (with a limitation of how transcriptions are handled in the frontend). We use prepared statements and parameterized queries wherever possible to prevent SQL injections. Our passwords are stored hashed, peppered and salted using PHP’s password_hash function and its default hashing algorythm - at this time bcrypt.

Application Security: Client Side

Server side security measures are aimed at preventing attackers from being able to do or see things on the server that are outside the bounds of the application or limiting the damage such a breach can cause. If a hacker can attack a user and potentially get access to their logged in session, they can breach confidentiality and integrity within the permissions of the user’s session.

Again, exhaustive lists are far from the scope of this article and the basic principles from above apply. But just for illustration, here are some potential attack vectors and how to prevent them.

When a cookie is set, one can specify in what contexts the cookie should be available. If it should be generally accessible or only on the same site, if it is to be accessible when a user connects over unencrypted HTTP or if it should only be available over SSL-encrypred connections (where the URL starts with https://) and more.

Attack Vectors Example: Cross Site Scripting (XSS)

To do so, an attacker might first try to execute arbitrary code within the browser. A common would be cross site scripting (XSS). Take for example a search page, hosted at https://example.com/search.php?query=test, that displays the search term (test) somewhere on the page. If the display simply outputs the raw search term, searching for <script>window.alert("test");<script> would execute the JavaScript function window.alert() in the browser, causing a pop-up window to display the text “test”. If that works, an attacker can send a potential user a suitably manipulated link and then execute JavaScript code within the context of the linked site. One thing they could thus do is read the users cookies on that site - including session cookies.

The example above outlines so called “reflected XSS attacks”. Other types exist, e.g. “stored XSS attacks”, wherein the attacker already has write access to the application and uses it to store arbitrary JavaScript code within a page they might legitimately control. If that JavaScript code were to be executed, e.g. a worker of one museum at museum-digital might be able to take over the browser and exfiltrate the session cookie of another worker at another museum using museum-digital.

Preventing XSS attacks can again take multiple forms. First, they can be detected and prevented in the first place. If a user searches for <script>..., the malicious parts can simply be dropped from the search term or the user blocked for attacking the page. This may be done by a specialized Web Application Firewall or as part of the application.

Directly as part of the application, text can (and should) be sanitized for its output context. Rather than printing <script> the server could output the HTML-encoded version (&#60;script&#62;). The search result page would then display the search term as a string of characters rather than executing it as code.

Finally, browsers can be told to restrict the browser’s available capabilities within the context of a web page using “Content Security Policies”. Preventing the execution of inline JavaScript code and JavaScript scripts from outside a list of trusted domains would prevent such a form of XSS altogether. If inline JavaScript is necessary, then the list of connection targets can be restricted (the cookie may be read, but the attacker would fail to send it to a place they can read).

The answer is not to use any of these methods. It is to use all of them to provide defense in depth. Also, again, it is useful to reduce the number of trusted sources etc. If one embeds JavaScript code from an external source and that supposedly trusted external source gets attacked successfully, the attacker can also execute arbitrary code within one’s web page (though there are again ways to prevent that).

Attack Vector Example: iframes

Traditionally, if one wanted to embed one website into another, one could leverage the HTML tag <iframe> built for exactly that purpose. Iframes however also give the embedding site the power to manipulate the embedded site. As such, the embedding site can potentially be used to attack users of the embedded site.

Iframes in principle embed another site as is - including its navigation and design. As such, they pretty much always looked terrible as well. A prettier and more secure way is to embed another site’s data without keeping its design - e.g. by using its API. This requires much more effort (think: money) than embedding using a single line of code, but produces significantly better and more usable results as well.

Protecting against malicious interference by embedding sites using iframes is simple: Prevent your site from being embedded by another. To do so, browsers can be told to refuse the embedding altogether using the CSP frame-ancestors directive and the older X-Frame-Options HTTP header.

museum-digital

At museum-digital, we regularly check new browser security features and implement them. We try to set CSPs cookie settings, etc. as restrictive as possible. We are however curtailed by museum-digital’s own features and age:

  • There are still two or three museums who embed museum-digital into their own site using iframes and have been doing so for more than 10 years by now. Disallowing new ones from doing so is easy, prohibiting existing setups afterwards is much harder to justify. As such, we are permitting iframes from those select sources using specifically restricted frame-ancestors CSP settings. If anybody takes over these two or three embedding sites, the security of users of the brandenburg and saxony-anhalt instance may be at risk.
  • musdb is hosted on the same subdomain as the respective regional frontend portals and features a “preview” button, with which logged-in users can view their unpublished object pages within the respective regional frontend portal. As such, regional frontend portals need access to the users session and read-only access to publishable data of unpublished object records. This both limits how much we can restrict access to sessions - increasing attack surface - and extends the necessary read-only access of the database user the frontend uses - thus increasing the potential damage in terms of confidentiality if the frontend gets breached to provide attackers with flexible DB level access. This could be remedied by a cleaner separation between frontend portals and musdb in terms of hosting architecture.

In terms of XSS attacks, we occasionally fuzz the site using automated tools to try to identify reflected XSS attacks both with automatic cleaning of malicious inputs enabled (as it is in production) and disabled, to ensure we correctly sanitize all output strings even if cleanup and validation fail. We did the same against stored XSS attacks by deliberately entering malicious inputs in the testing setup. In production, we sanitize almost any inputs even before storing them into the database - this is more radical than the general best practice, as it also means that strings containing the character < - e.g. object descriptions or transcriptions - cannot be entered into any text field in musdb (which in itself might be seen as an attack on data integrity - we see it as immediately obvious enough to not be overly problematic however; it does reduce the potential risks in any scenario the data is used and is thus - in our view - worth the risk).

As of now, the new redesign of musdb does not yet implement content security policies. Implementing them should certainly be done this year.

Methods to Identify Security Issues

Writing (comparatively) secure code is in large parts good and careful software engineering: Architecture, planning, and writing legible, understandable and concise code. Creating good software architecture is much about creating a suitable mental model for the requirements of one’s applications and planning on a larger scale what parts or types of parts an application should consist of and how these can - in broad strokes - be expressed in code. Good coding practices are about applying this to actual code.

An important part of both is modularizing and compartmentalizing: Breaking down problems into small units and translating those into software that consisting of functions and classes that each fulfill only one specific aim and have access only to the resources and information necessary to fulfill exactly that. By clearly defining inputs and expected outputs, these small parts can be extensively covered by automatic tests.

Generally, automated evaluation and checking of code can help a lot. This ranges from using language features (e.g. consequentially enabling strict typing in a PHP context) to tests (ensuring that expected inputs lead to expected outputs) to static code analysis and fuzzing (sending random data to the application and ensuring that the outputs are within a list of accepted outputs) and benchmarking to optimize the performance of the code.

museum-digital

Software architecture at museum-digital is a mixed bag. Newer tools, such as the importer (fully re-implemented from the ground up in 2019), were designed with a clear architecture in mind, some are small enough to barely need elaborate planning (e.g. the translation tool), and the oldest have grown and developed naturally over the years with a much less stringent architecture. Refactoring a large existing code base to follow a stringent architecture is a huge effort, often taking months of work. We did that - albeit far from perfect - with the frontend platforms between 2019 and 2021. Starting in 2022, we slowly started to do the same with musdb - the process will be completed once the legacy version has been removed (we are just in the process of moving to a new user interface and architecture right now).

We are using all the automatic code evaluation approaches above to varying degrees. Again, the cleaner the architecture and modularization, the easier it is to test. As such, the core parts of the import tool have a nearly 100% test coverage (how many lines of code are covered by automated testing). In the frontend and musdb we are far from it. Once the legacy PHP code is removed from musdb, bringing its server side components close to that number will be rather easy.

Our focus in using and tailoring these tools to the different functions also depends a lot on its use case. Whereas the core parts of the importer - potentially incredibly security critical, as users might provide any kinds of broken or malicious files for import - are very well tested, the frontend is best equipped in regards to performance benchmarking. Generally, we are much better with unit tests (checking that a single internal component works as intended) than with integration tests (ensuring that e.g. a whole web page works as intended from request to rendered output). Security-focused fuzzing is performed occasionally using Owasp ZAP being run against all web-accessible tools.

Account Security

Finally, even if the code were to be perfectly secure and the hosting architecture as well, confidentiality and integrity could be breached by user behavior. While an institution and its technical staff might control the hosting environment and the software, preventing a user from posting their login credentials on Insta is barely possible. But users might be prevented from having any post-able credentials in the first place. So, how can logins and accounts be attacked or protected?

Brute Force Attacks, Leaked Password Lists

If one offers logins, one likely has some form a login screen. The most straight forward attack on a login workflow is simply trying out many different combinations of usernames and passwords. This would in theory take a lot of time, but as users tend to reuse their credentials on different sites, enter short passwords or try to create passwords they can remember, such attacks can be made much more efficient using lists of already-leaked passwords and dictionaries of the expected language a user speaks.

From a user perspective, using a password manager with different autogenerated, random passwords or using a passwordless authentication method helps. Ideally, chaining those together in the way of two factor authentication helps.

From a service provider’s perspective the problem is huge, as control options are relatively week. Enforcing good - long and complex - passwords is an option, but regularly leads to user frustration. Implementing alternative or mandatory, additional login mechanisms can help, but comes with new onboarding costs for new users and an increase in support requests (with two factor authentication, users can not only forget their password, but also lose their second factor).

Recently, fully doing away with passwords has become popular. A login workflow might e.g. reuse users’ logins at centralized services (Google, Microsoft, Facebook, Github) or simply ask for your username and mail and send you a mail with a specific login link with short validity. The former means relying on large scale infrastures, usually of US American companies. The latter is worse: E-Mails are stored unencrypted and mail accounts are a single point of failure. If an attacker gains access to a user’s mail account, they also gain access to services that rely purely on a mail-based workflow for logins. Add to that, that increasingly users’ mail is hosted by the exactly same centralized US American infrastructures and companies above. And that these mails further normalize opening links in mails.

Brute Force Attacks can also be aided by purely technical bugs. If your login workflow first checks the username and then checks the password second, trying out different combinations with correct username and wrong password will yield a slightly slower response than login attempts that are fully wrong. That way, usernames can be leaked even if the overall attack fails (which can then inform subsequent attacks). Timing attacks such as these can be prevented by adding a random delay to any response to a login attempt, regardless of its success.

Phishing

Even with a good passwords, users personally - in their capacity as humans - can still be attacked to gain access to login data. Phishing attacks are attempts to make a user who has or knows something the attacker does not - e.g. a password - perform an action and/or expose their knowledge to the attacker.

  1. Imagine receiving a call from an unknown phone number. It is your boss, with their voice and common mannerisms, asking you to send them a file related to that evening’s meeting with the mayor. The file might contain confidential data. If enough public video or audio recordings of your boss exist, imitating their voice and mannerisms is relatively low-effort. Finding out about the event might simply take a look into a public event calendar.
  2. Imagine receiving a mail from an unknown sender. The sender introduces themselves as your boss, excuses themselves for using this mail account. They lost access to their business mail account yesterday. And also, they ask you to log into the online collection management system and generate a report on this object. The object is identified by a URL that roughly looks like the URLs your collection management system’s object pages use. You overlook that spelling mistake in the domain name (“musuem” instead of “museum”!), click on the link and enter your password.
  3. Imagine you receive a mail from an unknown sender trying to sell you Viagra for an extraordinarily low price. You can do so using their web shop linked below, where you would be asked to provide your credit card details. Also, you’re a 30 year old woman. You delete the mail.
  4. Finally, imagine that you logged in to a web service you had not used in a while. The login page accepts your password, but requests for a confirmation by way of your mail account. To do so, you should have received a link. You see a link supposedly by that service, click on the confirm button and enter your password again to confirm.

In each of these cases, you are contacted. The sender / caller starts the communication, provides some more or less credible context and asks you to do something. If you complied, you could leak confidential information (either by sending a file or entering data into an online form).

In the base examples, 4) and 1) are embedded within an immediately close context (event in the evening, previous login and confirmation request), 2) and 3) are without a direct prior context. Considering who you are, 1), 2), and 4) somewhat conform with your perceived reality (you are a user of the service, the name, voice, mannerisms of your boss are as expected), while 3) does not (why would you want to buy that?).

The main protection against phishing, from a user perspective is to be overly vigilant.

  • The attack 3) would certainly be identified as a potential attack and ignored or refused (deleting the mail) by a majority of people. It is without a doubt malicious.
  • Enough people would also get suspicious with 2): The sender mail is unknown, and even though your boss’ name is provided and the requests fits into your larger work context (museum / collection management system), the request comes without prior communication and the link - if you look at it closely - is manipulated. Also by using the name and position of your boss, the sender creates a sense of authority and urgency. Some might forward the mail to the actual, regular mail account of their boss. Some might ask for the inventory number or identify the object using the URL schema, search for the object in their collection management system, generate the report might send it (this would breach confidentiality for the single object record). Enough people would click on the link. With the spelling “mistake” contained in the URL, an attacker could have registered a domain looking very closely like the one of your collection management system and faked your login screen. Entering your password means the attacker gets access to your account - potentially breaking both confidentiality and integrity. Likely, they change your account’s password once they gain access - thus also impacting availability.
  • With 1) a potential attacker trying to imitate your boss would need to spend some hours, gather quite a lot of context information, train a voice imitation software. And the damage of that specific file leaking would probably not be too large (though that of course also depends on the content). It could be malicious - the phone number is unknown after all - but as the request is rather credible and the potential damage is rather low, one would likely comply.
    1. directly takes place within a context one has triggered oneself. The likelihood of an attacker knowing that you just tried to log in is minimal. But since you are focused on regaining access to your account, you did not bother to look at the target URL of the confirm button (if your mail client even shows it). You entered your password into whatever page opened upon the press of a button. The likelihood of an attack is low, but if it is successful, the attacker has just gained access to your account (and, if you reuse password, potentially all the other accounts using the same credentials). But exactly because the likelihood is so low, if this had been an attack, it would almost certainly have been successful.

Now, phishing is a type of attack that is essentially non-technical and instead aims to use humans’ behavior against them. From an organizational perspective it can potentially be fought by awareness campaigns and strict adherence to pre-defined workflows. But making people be ever-cautious and strictly adhering to what has been agreed upon is hard. All the more when it does not come to their basic field of expertise or interest. All the more in the cultural sector.

Further technical options are usually linked to a specific communication channel. Enforcing the use of signed and/or encrypted mails (in an institutional setting usually via S/Mime) can help warn users of attacks.

From a service provider’s perspective, fighting phishing is hard, as the attack takes place completely outside of one’s own area of control. One can however enforce the use of authentication methods that are resistant to phishing.

Passwordless Authentication and Two Factor Authentication

The options to break account security on the user side get significantly reduced if a user’s knowledge is not the only factor for authentication. Generally, there are three types of factors that could be used for authentication:

  • “Something one knows” like a username & password combination
  • “Something one has” like a physical security token or the secure enclave of your PC
  • “Something one is” like biometrics

“Something one knows” can be leaked; “something one has” can be lost. “Something one is” is undercomplex: it should be “the knowledge of certain properties one has” (such as a fingerprint). Losing your finger is unlikely. But changing your fingerprint if it leaks is hard, too. Regardless, both “something one has” and “something one is” are barely phish-able.

As of now, the implementations of factors one has are getting better and better - to a level where the major big tech players are pushing for replacing passwords with them. From a security perspective, they are right: Replacing passwords with e.g. a USB stick security token is much more secure (see: “Passkeys”). If you worry about losing those USB sticks - just get two and keep one in a safe.

Security can be further improved by combining two different factors of different types, accessed via two different devices or communication channels.

From an institutional perspective, setting appropriate authentication policies, linking services to centralized institution-level single-sign-on infrastructure wherever possible and promoting the use of passwordless authentication methods makes a lot of sense. Spending some 50 Euro per user and equipping them with physical security tokens does, too.

From a service provider perspective, it is definitely advisable and laudable to offer alternative and secondary authentication options. In how far these can be enforced for all users is a harder question. Common second factors like TOTP are (comparatively) hard to understand and tedious, on-device passkeys and OAuth (authentication using centralized providers) tend to lock users to their device or centralized services. Specific security hardware costs money.

Passwords on the other hand come with workflows users have become accustomed to. Trying to introduce them to other, more secure authentication methods once again comes with frustration and increased demand for support.

museum-digital

To defend against brute force attacks those of museum-digital’s tools that require a login add a delay after each login attempts (if successful or not). The duration of the delay is a combination of a short, random delay (to defend against timing attacks) and one determined by the number of previous login attempts on the level of the session and IP as well as to the whole installation of the current tool. The login screen is thus automatically locked down with an increasing number of login attempts. The threshold for a full-on lockdown is comparatively low - which has in multiple instances negatively impacted availability for legitimate users who tried to log in. We value confidentiality and integrity over availability, if the choice must be made.

musdb offers two factor authentication via mail, TOTP, and WebAuthn / Passkeys. Passkeys can also be used as a passwordless authentication method.

A basic password policy to prevent users from setting overly common or undercomplex passwords is however the only strictly enforced setting. Users can opt to secure their accounts further. But they are not forced to.

Given the very diverse userbase of museum-digital, with very different use cases, levels of expertise, and even more diverse threat levels, we have not seen it as opportune to change that. What musdb lacks as of now is an option to enforce e.g. two factor authentication on an institutional level. Regional admins however, who have access beyond a single museum, are indeed required to enable two factor authentication.

Finally, a common place of attacks and bugs is password reset mechanisms. We have hence not implemented any within any of our tools. Instead, users of the user role “museum director” can create new users and change other users’ passwords, as long as those other users belong to the same museum. Password resets thus become an organizational rather than a technical question. This structure has proven effective for smaller, largely location-specific institutions, as pretty much all museums are when compared to a bank. It does however have a weakness in that the user role is necessary to both perform technically administrative tasks as well as to have the highest level of permissions when working on a museum’s collections. This makes separating and reducing privileges to the least permissions necessary hard on an account level.

Threat Levels, Possible Adversaries and a Conclusion

The above just lists basic levels an attack can take place on, strategies to tackle them and attack vectors to protect against. The portrayal is - again - undercomplex and incredibly incomplete. Much more important than any single one issue however is perspective. Cybersecurity is a mindset.

Do not trust anybody, least of all yourself. Identify what processes take place in your field of work (or live), how they can be subverted in emergency situations, and how a potential attacker might use those processes or their fallback strategies to harm the confidentiality, integrity, and availability of your data and services.

Identify your requirements: Do your users need a graphical user interface? Do your users need to collaborate with others? Do you have sufficient technical expertise and work hours locally?

Identify what components your processes, services, etc. have and how they can be secured independently and in whole. Then identify in how far that impacts your ability to do your actual work and potentially drop some level of security measures, as your actual priorities probably lie with doing your work over being 100% secure.

And finally, identify what level of threat you are under and what an attacker might do with your data. If you are an institution that only publishes select object records via museum-digital, with all internal data kept safely in your inhouse database, then an attacker would simply gain nothing from getting read-only access to your collections in musdb. It contains only published data anyway. If they gained write access, they would be able to alter your data and potentially damage your credibility as a institution involved in research.

If you keep some object records unpublished, read-only access would mean an information leak. If the data only contains publishable data fields, the damage is likely still minimal. If an attacker however gained access to legally protected data (personal data of living or only recently deceased people) or commercially relevant internal data (location of an object in the depot, insurance value), then even read-only access can involve a risk of lawsuits or be used as a preparation for a physical break-in attempts and robbery of your collection. The damage would be huge.

The threat level of course also depends on who you are and what you have: If you are a small local museum that mainly collects regionally common objects of everyday use (that would not be commercially interesting for robbers), the potential threat level is comparatively low. If you work for the National History Museum in Kyiv - an important place of historiography in the middle of a war of that is also fought in terms of historiography - your threat level is likely as high as it gets within museums. If you work for a critical infrastructure provider or are a major politician in the middle of a war zone, it will be yet much higher.

Security is complex. Security needs careful planning and thinking. Security is important. Security is never 100% ideal. Security measures should be appropriate. But more than all, security should be consciously considered as an aspect of everyday (work) life. Always.

P.S.: Data Protection and museum-digital

As the question about cybersecurity is, at least in Germany, often paired with the question of data protection, some notes on that. Data protection generally is concerned with personal data. At museum-digital, we store personal data in four or five places.

  1. User information, e.g. in musdb (e.g. name, password, a log of one’s logins incl. IP used for logging in)
  2. Data from the address book / contact module in musdb (which is e.g. referenced when noting previous owners of an object or the freight company providing for the transport of an object)
  3. Personal names etc. in the shared vocabulary of actors
  4. IP addresses in the logs
  5. Mail addresses of users who have signed up for the news letter or forum within the respective systems

On a non-system level, people who received a support request via mail probably also keep the relevant mail (or at least their response) in their mail folders.

  1. User information can be fully deleted or exported in musdb. Doing so deletes the full account record. It does not however remove the name of the user from the data history logs of database entries the user edited, as we see it in the legitimate interest of the museum to be able to identify who edited the museum’s data in what way and when they did so. Other users of the same museum that have the user role “museum director” and regional administrators can see the user data.
  2. Data from the address book is purely internal and cannot be published via musdb / museum-digital. As such, it is only accessible to user accounts of the museum as well as regional administrators.
  3. The shared vocabularies are fully public and new entries are published under a CC0 license. If a museum enters a new entry, it is thus immediately public. It is important to make sure, that one has the rights to actually record and publish e.g. the name of a creator before adding it to the shared controlled vocabularies. If personality rights speak against a publication, the address book or a free text field should be used. The museum and its workers are responsible for their actions. See the terms of use in Germany for reference.
  4. Logs, including IP addresses are written and stored for a short time to allow e.g. the detection of attacks. See the privacy policy of the frontend for more and up-to-date information.
  5. This is only for users who themselves signed up. They can remove their accounts using the respective functionalities of newsletter or forum.

P.P.S: If you want to read more, Ross Anderson’s Security Engineering and the OWASP website are good starting points.

info
AI Use

Post image generated using Anima Aesthetic v1.1.

info
License