Resolved -
A stuck query from a manual backup dump did not terminate and caused higher DB load than usual due to a glitch. The average p99 latency was up at ~200ms between 7:00 and 10:00 AM as opposed to usual levels around 100ms. All systems are back to normal levels now.
Oct 21, 11:02 CEST
Resolved -
This incident has been resolved.
Oct 9, 14:00 CEST
Monitoring -
Services are back up at normal scale. We continue to monitor operations.
Oct 9, 13:56 CEST
Update -
Database has recovered, we're scaling out again.
Oct 9, 13:53 CEST
Update -
A data migration unexpectedly caused excessive load on a backend database causing it to lock up. We're redirecting traffic away from it to let it recover.
Oct 9, 13:50 CEST
Identified -
The issue has been identified and a fix is being implemented.
Oct 9, 13:44 CEST