Newly registered Assets not available in Asset list and Search

Incident Report for gridX GmbH

Postmortem

A change deployed to production caused events to not be sent after appliances were registered in our system, affecting the indexing of our Xenon Search feature and making 180 assets unsearchable from their creation time (after 11:57 CET) until the incident resolution (17:25 CET).

Root Cause
The problematic change modified event emission in the appliance creation handler to occur synchronously. However, the context used for sending events had already been cancelled by a transactor at this stage, resulting in events not being emitted. Unit tests, although written to ensure correct event sending, used a mock transactor implementation that didn't accurately reflect the cancelling context behavior. Additionally, E2E tests for this handler didn't cover event sending.

Resolution
A fix was deployed to utilize the correct context, and a backfill of missing events was performed to rectify the issue.

Action Items

Improve observability of event streams with alerting for faster identification of issues.
Shore up test suite to thoroughly cover the event emission around assets, preventing these issues in the future.

Posted Nov 27, 2024 - 15:05 CET

Resolved

Notifications are again being processed as expected.
Search has been backfilled ensuring all assets can be found.

Posted Nov 21, 2024 - 17:30 CET

Monitoring

The fix has been deployed and we have confirmed we have returned to normal operation. We're monitoring the changes, and working on backfilling the missing assets in Search and Notifications.

Posted Nov 21, 2024 - 16:43 CET

Update

Update on the impact of this incident: Asset List is not impacted. Xenon Search and Notifications are impacted, and will have to be backfilled after the issue is solved.
Fix is being deployed.

Posted Nov 21, 2024 - 16:12 CET

Update

We are continuing to work on a fix for this issue.

Posted Nov 21, 2024 - 15:17 CET

Identified

We have identified an issue of missing asset create events, resulting in new asset information to be missing on the Asset List and Xenon Search. We have identified the issue and are working on a solution. No data has been lost.

Posted Nov 21, 2024 - 15:16 CET

This incident affected: Frontend and Platform Components (beta) (Asset inventory management).