01High-traffic consumer platform · critical incident
When production breaks, the first decision is what not to do.
Critical incidents on a high-volume platform test architecture and leadership at the same time. The signal shows up at the edge; the cause almost never lives there.
- Decision
- Separate mitigating from fixing: reduce the impact first, root cause second.
- Trade-off
- Roll back fast or fix for real — decided by what each minute costs.
- Outcome
- Every critical incident becomes a resilience and governance standard.
Context
A platform peaking at millions of requests per minute on AWS and Kubernetes, with several layers of services and dependencies on external integrations. A production problem rarely stays in one place.
Challenge
The symptom reaches the end user, but the cause sits a few layers below — sometimes outside your own system. Meanwhile the impact is multiplied by volume, every minute.
Decision
Separate mitigating from fixing. Reduce the impact first; find the root cause with a clear head. Resist the rushed fix that silences the symptom and hides the problem.
Architecture
Trace the request from the channel to its origin: edge, BFF, services, integrations. Logs and observability are what turn a hypothesis into evidence.
Trade-offs
Rolling back is fast, but may not be an option when the origin belongs to another team or vendor. Fixing forward solves it for real, but demands certainty. The choice depends on what each minute of impact costs.
Leadership
In an incident, clarity is leadership: who investigates what, who communicates, when the next update lands. Deciding under pressure without turning the moment into a search for someone to blame.
Outcome
Every critical incident feeds back into resilience standards and technical governance — the work doesn’t end when the graph goes back to normal.
Reflection
Incidents are audits nobody scheduled. They show you exactly which architectural decisions were implicit.