Keeping a high-concurrency transaction platform standing 50k → 400k+ users

I drove the re-architectures that kept the platform up as daily users climbed to 400k+ — replacing the ceilings one live single-point at a time.

50k → 400k+, live collision-proof cutover sub-second pushes 1 day → 30-day queries

Situation

Over four years the member management platform grew from 50k to 400k+, and growth kept finding the limits of the old mechanisms — a counter document in the database serialized every ID; notification fan-out stalled under load. More than once, the failure happened in production first.

Constraint

No rewrite budget, no downtime allowance — every fix had to land inside a weekly release train, live, with users on.

Approach

Every target was a live global single-point, so I optimized for reversibility over speed — small, individually revertible PRs shipped in series rather than one big-bang cutover. I championed bringing a distributed cache into the platform, then root-caused each bottleneck to its mechanism and replaced the mechanism, not the symptom: ID generation moved from a database counter to atomic Redis INCR — cut over with a +1-billion offset so old and new ranges could never collide; notification fan-out moved to Redis pub/sub (shipped, reverted on a production surprise, re-landed hardened); blocking full-set scans replaced with cursor-based SSCAN; a leaking in-memory config cache rebuilt with a 15-minute refresh cycle.

Outcome

The platform sustains 400k+ daily active users without the old failure modes. The patterns introduced (cache-aside, atomic counters, pub/sub fan-out) became house standards for modules built after.

The design, in code

Representative snippet of the production design — rewritten for publication; not company source.

id-cutover.ts 
// Live cutover with users online: the new range starts one billion above
// the old counter, so both generators can run at once and never collide.
const OFFSET = 1_000_000_000;

const legacy = await db.counters.findOne({ _id: 'memberId' }); // old ceiling
await redis.set('seq:memberId', legacy.seq + OFFSET, { nx: true }); // seed once

const nextId = await redis.incr('seq:memberId'); // atomic — no document lock
// Rollback is boring on purpose: point back at the old counter.
push-fanout.ts 
// The bottleneck was never the sockets — it was one process fanning out
// to every connection. Publish once; each instance delivers to its own.
export function broadcast(userId: string, event: Event) {
  return redis.publish(`push:${userId}`, JSON.stringify(event));
}

// Every instance subscribes and only touches sockets it actually holds.
redis.psubscribe('push:*', (channel, payload) => {
  const userId = channel.slice('push:'.length);
  localSockets.get(userId)?.send(payload); // sub-second, N instances
});
config-cache.ts 
// The old in-memory config cache grew without bound and leaked. Rebuilt
// cache-aside: read local, fall through to Redis, refresh on a fixed cycle.
async function getConfig(key: string): Promise<Config> {
  const hit = local.get(key);
  if (hit && hit.expires > Date.now()) return hit.value;

  const value = await redis.get(key).then(JSON.parse);
  local.set(key, { value, expires: Date.now() + 15 * 60_000 });
  return value; // bounded, self-refreshing — config edits go live in minutes
}

War stories

The tuning months

Before this platform earned its Redis layer, it had to earn its database. The reporting layer predated any business-intelligence team — and one stretch of growth broke it visibly: operational reports stopped coming back at all, and on the busiest collection a query couldn't return even a single day's records without timing out.

The months that followed were unglamorous: reading the slow-query log like a newspaper, matching covering indexes to the query shapes people actually ran (and deleting the indexes nobody did — on write-heavy collections an unused index is pure tax), rebuilding aggregation pipelines stage by stage, and moving text search off regex scans onto a proper search index.

Thirty-day report windows came back comfortably on collections where one day used to time out. That work never made a release announcement — but every feature shipped since stands on it.

Measure first, then argue

Push notifications were arriving late, and the room's instinct was to tune the WebSocket layer — it was the visible component. Before touching anything I measured where the time actually went, and the cost wasn't in the sockets at all: it was the fan-out loop behind them, delivering to every connection from one place. We moved fan-out to pub/sub so each instance delivers to its own connections, and pushes went sub-second. The loudest component is rarely the bottleneck; the profile decides, not the intuition.

The revert that came back stronger

The fan-out migration shipped — and production surprised us, so we reverted. The second landing looked different: instead of one big change, a series of small, individually revertible steps, each one hardened by what the first attempt had taught us. Nothing about the design was wrong; the delivery was. On a live transaction platform the unit of change has to equal the unit of rollback — that rule came out of this war and never left.

A billion-ID head start

ID generation had to move from a database counter to atomic increments in Redis — with users online and no maintenance window. Instead of choreographing a precise cutover between two live generators, we made collision impossible: the new range started one billion above the old counter, so both could run at once and never meet. Rollback was equally boring — point back at the old counter. When transactions are on the line, the clever solution is the one with nothing left to go wrong.