Between 6 a.m. and 9:15 a.m., museum-digital’s main server unfortunately experienced another outage. The immediate outage has been resolved now.
The larger issue remained the same as so often during the rocky months last year:
- With IIIF processes stopping, they keep holding onto RAM, until eventually the server is overloaded
- With no available RAM left, the database (MySQL) falls over
To bring back the server, we needed to:
- Stop the web server
- Restart PHP (freeing up RAM)
- Restart MySQL (bringing it back)
- Restart the web server
This was, to some extend, to be expected: PHP ran without issue for some weeks, and the outage shows in the real world how close we can get. As such, the solution will be to further reduce available resources for the IIIF setup and regularly restart PHP.
For some background, see also:
