Skip to main content
CRON CADENCE: */5 * * * * UTCSOURCE: OFFICIAL STATUSPAGE V2 JSON FEEDS
Agent Outages
MODULE: VERIFIED OUTAGE POST-MORTEM REGISTRYVERIFICATION: PRIMARY VENDOR STATUS & BLOG DISCLOSURES ONLY

Sourced AI Infrastructure Outage Post-Mortems

Every incident write-up on agentoutages.com links directly to the provider's own published status report or engineering post-mortem and extracts concrete architectural lessons for multi-step agent builders.

ANALYSED EVENTS4 Benchmarks
PRIMARY CODES429 / 503 / 529
RETRY HAZARDUnjittered Storms
WRITE DEFENSEIdempotency Keys
Fiber-optic patch cables plugged into a core network switch inside an availability zone
CORE FIBER PATCH BAYAZ FAILOVER

OpenAI (API, ChatGPT & Sora)

INC-01 · K8S DNS & CONTROL PLANE

Incident window: 11 December 2024 · Observed duration: Approx. 4 hours 30 minutes

OpenAI Incident Report: Telemetry Service & Kubernetes Control-Plane Overload

Published Provider Root-Cause Summary

According to OpenAI’s published post-mortem, a configuration rollout of a new telemetry service triggered high-volume cluster-wide Kubernetes API calls across clusters, overwhelming the Kubernetes control plane and degrading internal DNS service discovery.

Agent Pipeline Blast Radius & Mitigation

Unhedged agent orchestrators without bounded retry budgets entered synchronized retry storms, exhausting worker pools while waiting on hanging TLS/HTTP connections. Teams running secondary-provider circuit breakers failed over stateless tool-routing prompts within 15 seconds.

Cloudflare (Workers KV, Control Plane & Analytics APIs)

INC-02 · KV & CONTROL PLANE SPLIT

Incident window: 2 November 2023 (Referenced Control-Plane Benchmark) · Observed duration: Multi-hour control-plane degradation (edge proxy remained up)

Cloudflare Engineering Post-Mortem on Control Plane and Analytics Outage

Published Provider Root-Cause Summary

Cloudflare documented how simultaneous regional data-center power failures in Oregon exposed hard control-plane dependencies for APIs, Workers KV writes, and log pipelines even while cached global edge routing continued serving traffic.

Agent Pipeline Blast Radius & Mitigation

Demonstrates the mandatory split between data-plane and control-plane state: agent session routers that wrote state synchronously to control-plane KV stores on every turn stalled, whereas local durable queues degraded gracefully.

Anthropic (Claude Messages API HTTP 529 Events)

INC-03 · HTTP 529 CAPACITY SHED

Incident window: 2024-2025 Peak Traffic Windows · Observed duration: Intermittent 10 to 45 minute regional capacity windows

Anthropic API Status & HTTP 529 Overloaded Error Documentation

Published Provider Root-Cause Summary

During sustained compute saturation, Anthropic’s inference gateway returns HTTP 529 (Overloaded) distinct from per-organisation HTTP 429 (Rate Limit) responses.

Agent Pipeline Blast Radius & Mitigation

Naive SDK wrappers that treat HTTP 529 identically to HTTP 400 abort long-running agent runs mid-task. Resilient orchestrators classify 529 as an immediate trigger for exponential backoff with full jitter or cross-provider model fallback.

GitHub (Actions, Webhooks & Git Operations)

INC-04 · WEBHOOK REPLAY STORM

Incident window: 14 August 2024 · Observed duration: Approx. 1 hour 40 minutes

GitHub Status Incident History (August 2024 Database Configuration Change)

Published Provider Root-Cause Summary

An unintended configuration change on an internal database cluster disrupted core Git operations, Pull Request webhooks, and Actions runner dispatch.

Agent Pipeline Blast Radius & Mitigation

Autonomous software-engineering agents triggered duplicate PR comments and duplicate CI commits when webhook deliveries replayed after recovery, underscoring why every agent write tool must check deterministic idempotency keys.

INSIGHTS, TOPIC TAGS & GUIDES INDEX · LAST SYNCED 2026-11-06
RSS 2.0 Feed

AI Infrastructure Outage Post-Mortems, Reliability Tags & SRE Guides

Filter sourced provider incident post-mortems, compound multi-step SLA models, and production failover patterns across model APIs, vector databases, and edge runtimes.

Cloudflare Cron Worker polls 6 upstream Statuspage v2 feeds every 5 minutes; incident post-mortem briefings are verified against official vendor engineering disclosures.

Filter by Topic Tag

Reference Guides & Interactive Workbenches

4 matching

Dated Primary-Source Briefings & Standards Signals

3 entries
#HTTP-529#Failover#SLA-Math

Handling HTTP 529 Overloaded Distinctly From Per-Tenant HTTP 429 Rate Limits

Anthropic API documentation distinguishes regional compute saturation (HTTP 529) from workspace token bucket exhaustion (HTTP 429).

Actionable takeaway: Trigger immediate cross-region or cross-model fallback on HTTP 529 rather than burning 60 seconds in local retry sleep.

Primary source: Anthropic API Error Reference
#Failover#Retry-Storms#Status-Telemetry

Control-Plane Telemetry Rollout & Kubernetes DNS Cascade Lesson

OpenAI documented how cluster-wide control-plane load degraded internal DNS discovery for roughly 4.5 hours, stalling unhedged synchronous agent loops.

Actionable takeaway: Enforce strict per-turn connect/read timeouts (under 12s for routing turns) and fail over to a warm secondary provider.

Primary source: OpenAI Public Status History
#Idempotency#Webhook-Replay

Webhook Delivery Backlogs & Deterministic Agent Idempotency Keys

Delayed webhook queues replaying after upstream database recovery caused stateless coding agents without idempotency ledgers to execute duplicate write actions.

Actionable takeaway: Hash (event_id + tool_name + canonical_args) in durable storage before executing any external mutation tool.

Primary source: GitHub Status Incident Archive

Reviewed by Nadia Lindqvist, Lead Reliability & Agent Systems Editor

Verifies provider incident timelines and failure-mode mitigations.

Last reviewed:

Reviewed every 3 months

AI-assistance & editorial check: Research outlining and formatting are assisted by AI tools in an offline draft workflow. Nothing on this site publishes automatically without verification. A named human editor opens and verifies every primary source link, command snippet, and factual statement prior to publication. See our Editorial Policy and Corrections Policy.

Verified Primary Sources

Get Future Incident Post-Mortems by Email

Agent Uptime & Failover Post-Mortems

Store your email for engineering breakdowns whenever a major model, vector, or edge compute incident teaches a new resilience pattern. No emails are sent yet.

Protected by Cloudflare Turnstile · Isolated per-site D1 (agentoutages-leads).