XENON and API is down

Incident Report for gridX GmbH

Postmortem

On 2026-01-02, our API gateway stopped routing traffic around 08:04 UTC. This caused a 68‑minute outage (08:05–09:13 UTC) with 100% request failure for all customer-facing services like:

The issue was identified through monitoring and resolved by working with the vendor to restore normal gateway operation. All affected endpoints are reachable again.

Root Cause

A third-party vendor misconfiguration incorrectly invalidated our API gateway license, causing gateway instances to stop serving traffic for all services routed through it.

Resolution

  • Investigated failed requests and gateway behavior using existing monitoring and logs.
  • Engaged the vendor to restore normal operation of the gateway.
  • Verified recovery via monitoring.

Action Items

  • We are working with the vendor to get better visibility into licensing status.
  • We are looking into adding more ingress redundancy to reduce possible single points of failure.
Posted Jan 13, 2026 - 09:21 CET

Resolved

This incident has been resolved.
Posted Jan 02, 2026 - 10:18 CET

Update

The issue seems to have affected only the HTTP API, not gRPC. This means most, if not all, communication between gridBox and cloud was NOT affected.
Posted Jan 02, 2026 - 10:18 CET

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 02, 2026 - 10:12 CET

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 02, 2026 - 09:58 CET

Update

Seeing issues with Ingress
Posted Jan 02, 2026 - 09:25 CET

Investigating

We are currently investigating this issue.
Posted Jan 02, 2026 - 09:22 CET
This incident affected: Frontend and Public API.