Partial/Temporary Outage

Due to a large scale roll-out of gridBoxes, the gateways/…/scans endpoint was called significantly more often than expected. The implementation of this endpoint was suboptimal, so we could not keep up with the load, which lead to timeouts and subsequent retries, further exacerbating the problem.

Resolution

• Scale in replicas serving the endpoint in question to relieve pressure from the database.
• Disable gateway.../scans endpoint
• Scale out again
• Analyze and optimize the sub-optimal query used by the endpoint in question.
• Re-enable gateway.../scans endpoint

Action Items

Reduce polling interval of the gateways/…/scan endpoint in Phoenix

Optimize query backing the endpoint in question and re-enable

• Reconsider how scans are handled and stored
• Audit existing queries on large tables to identify potential future bottlenecks

‌

See also https://status.gridx.de/incidents/9nd1bqyb6bdy