Scaling microservices to one million users without losing your mind

Scaling is rarely a fireworks show. It's a slow, methodical reduction of variance — at the network layer, at the data layer, and most importantly at the human layer. The interesting failures don't come from load. They come from the moments your team's mental model of the system stops matching the system itself.

Contracts at the edges

Every service boundary is a contract whether you wrote it down or not. I've watched a 'simple field rename' on a JSON payload take down four downstream services because the contract lived only in tribal knowledge.

• Generate clients from a single source of truth (OpenAPI, Buf, or GraphQL schema).
• Reject unknown fields by default in critical paths; tolerate them in eventually-consistent ones.
• Run consumer-driven contract tests in CI, not as an aspirational quarterly initiative.

Idempotency is a feature, not a vibe

Past about ten thousand requests per minute, retries become a load test you didn't sign up for. Every mutating endpoint should accept an idempotency key, and every event handler should be safe to replay.

ts
// pattern: idempotency table guards the side effect
async function chargeCustomer(key: string, payload: ChargePayload) {
  return db.transaction(async (tx) => {
    const seen = await tx.idempotency.find({ key });
    if (seen) return seen.result;

    const result = await stripe.charges.create(payload);
    await tx.idempotency.insert({ key, result });
    return result;
  });
}

Observability is debugging from the future

If you're not investing in tracing the same week you ship a new service, you'll regret it the week after. The cost of attaching a trace ID at the edge and propagating it religiously is trivial. The cost of forensically reconstructing a distributed bug at 2am is the rest of your night.

“Observability isn't a tool you buy. It's a property of the code you write.”