Incident Title: Complete Platform Outage Due to Infrastructure Update

Start Time: July 2, 2025 06:30 IDT

Resolved Time: July 2, 2025 07:31 IDT

Impact: Complete platform outage

Incident Summary

An infrastructure update performed during off-hours on July 1st caused a disruption to backend services. The update unintentionally removed and recreated a key compute resource with incorrect configuration, preventing new servers from launching. This led to a temporary outage of backend functionality between July 2, 2025 06:30 IDT and 07:31 IDT.

Note: The infrastructure change was performed during off-hours to avoid disrupting customers workflows. Unfortunately, the resulting issue impacted early-morning procedures before mitigation could be completed.

Root Cause

A misconfiguration introduced during the infrastructure update prevented automatic provisioning of new resources. This resulted in an inability to handle backend workloads until the issue was manually mitigated.

Affected Customers

Only customers who had live procedures during the downtime window experienced impact. All other services remained unaffected.

Resolution

The engineering team intervened to restore service by manually adjusting the configuration and adding resources. Service was fully restored by 07:31 IDT.

Follow-up Actions

• Strengthen monitoring and alerting for provisioning failures
  
• Improve escalation and on-call response procedures
  
• Ensure infrastructure configuration updates are verified post-deployment
  
• Establish clear protocols following critical infrastructure changes
  

---

We sincerely apologize for the disruption. We are taking steps to prevent similar issues and improve platform resilience.

Status: ✅ Resolved

API went down.