A blog on museum-digital and the broader digitization of museum work. Zola 2026-02-24T00:00:00+00:00 https://blog.museum-digital.org/tags/server-outage/atom.xml 2026-02-24T00:00:00+00:00 2026-02-24T00:00:00+00:00 Joshua Ramon Enslin https://blog.museum-digital.org/2026/02/24/mddown-temporarily/ <p>Between 6 a.m. and 9:15 a.m., museum-digital’s main server unfortunately experienced another outage. The immediate outage has been resolved now.</p> <p>The larger issue remained the same as so often during the rocky months last year:</p> <ul> <li>With IIIF processes stopping, they keep holding onto RAM, until eventually the server is overloaded</li> <li>With no available RAM left, the database (MySQL) falls over</li> </ul> <p>To bring back the server, we needed to:</p> <ol> <li>Stop the web server</li> <li>Restart PHP (freeing up RAM)</li> <li>Restart MySQL (bringing it back)</li> <li>Restart the web server</li> </ol> <p>This was, to some extend, to be expected: PHP ran without issue for some weeks, and the outage shows in the real world how close we can get. As such, the solution will be to further reduce available resources for the IIIF setup and regularly restart PHP.</p> <hr /> <p>For some background, see also:</p> <ul> <li><a href="https://blog.museum-digital.org/2025/12/29/trimming/">Trimming.</a></li> <li><a href="https://blog.museum-digital.org/2025/12/22/cleaning-out-our-closet/">Cleaning Out Our Closet</a></li> <li><a href="https://blog.museum-digital.org/2025/12/09/updates-ai-scrapers-and-resilience/">Updates, AI scrapers, and Resilience</a></li> </ul> <div class="my-4 flex flex-col rounded-lg bg-(--admonition-bg)" style="--admonition-bg: var(--admonition-info);" > <div class="flex items-center rounded-t-lg bg-(--admonition-bg) p-1"> <div class="mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert" style="--url: url(icons/info.svg);" > info </div> <span><strong>License</strong></span> </div> <div class="pl-4"><p>This content is licensed under a <a rel="external" href="https://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International license</a>.</p> </div> </div> https://blog.museum-digital.org/2026/02/24/mddown-temporarily/IMG_0438-scaled.png 2025-06-11T00:00:00+00:00 2025-06-11T00:00:00+00:00 Joshua Ramon Enslin https://blog.museum-digital.org/2025/06/11/maintenance-outage-search-database-not-available-for-today/ <p>Some minutes ago, our search server crashed. To bring everything back to normal, we will have to re-index, which will take a few hours. As that is being done, most of the frontend will remain operational without issue. musdb will, depending on the instance, be unavailable into the night.</p> 2024-03-06T00:00:00+00:00 2024-03-06T00:00:00+00:00 Joshua Ramon Enslin https://blog.museum-digital.org/2024/03/06/server-outage-switch-fault/ <p>Between March 6th, 2024, 9:54 a.m. and 10:34 a.m. museum-digital’s main server was unavailable due to a switch fault at the data center at our hosting provider, Hetzner. Details can be seen here on <a rel="external" href="https://status.hetzner.com/incident/f9758dfb-0bc2-440e-8743-0c322194b682">Hetzner</a>’s website.</p> <p>By 10:34 the error was fixed by the technicians in the data center and all systems at museum-digital are operational again.</p> <p>We are sorry for the downtime and looking into ways to better mitigate similar issues in the future.</p>