The first production service on a brand-new stack — from zero to clean handoff

The pilot the whole stack migration was judged on — I built it solo, and every pattern I set became the template later services copied.

first on the new stack buy the base, build the moat solo → team → clean handoff became the template others copied

Situation

The company needed AI-assisted customer conversations with human takeover — and had just decided on a new microservices stack no one had shipped to production yet. I was handed both problems at once.

Constraint

Solo start; NestJS, gRPC, Temporal, Pulsar and an off-the-shelf messaging backbone — all new to me at once; the outcome would decide whether the company's migration was viable.

Approach

Self-hosted an off-the-shelf messaging backbone as the message transport (the framework was the CTO's call; the data model and flows were mine), then built only the differentiators it doesn't have: an agent-routing queue encoding status > load > wait-time into a single Redis sorted-set score — banded so status always outranks load, and load outranks waiting time; assignment just reads the head of the queue; a mini back-office management system so agents see a member's account records, top-up / withdrawal history and permissions in one screen instead of switching systems; AI answers first — designed to absorb the repetitive ~70% of inquiries — with a 10-second race guard that hands the chat to a human whenever the model stalls or errors. Concurrency was the war — the score race and the duplicate-invite storm both have their own war stories below.

Outcome

First production service on the new stack — the pattern reference (config via Vault, contract-first gRPC, structured logging, graceful shutdown) that later services copied. Grew the team to two, handed the service to a maintenance team, later to the AI team. It now runs as a standalone multi-channel middleware.

The design, in code

Representative snippet of the production design — rewritten for publication; not company source.

routing-score.ts

// One number encodes the whole routing policy — banded so each term
// owns an order of magnitude and can never outvote the one above it.
const score =
  statusLevel * 1e18 + // status ▸ away sinks below every online agent
  chatCount * 1e15 +   // load ▸ fewer active chats ranks first
  timestampMs;           // waited-longest ▸ idle-since breaks the tie

// Every state change is one atomic increment — no read-modify-write race.
await redis.zincrby(QUEUE, +1e18, agentId); // agent steps away
await redis.zincrby(QUEUE, -1e18, agentId); // agent comes back online

// Assignment is O(1): the best agent is simply the head of the set.
const [next] = await redis.zrange(QUEUE, 0, 0);

agent-invite.ts

// Two messages in the same room, same millisecond, both want a human —
// without a fence, two agents get invited into one chat. First caller wins.
async function inviteAgent(roomId: string) {
  const locked = await redis.set(
    `agentRoomLock:${roomId}`, '1', { nx: true, px: 10_000 });
  if (!locked) return; // someone else already took the room — walk away

  const agent = await assignFromQueue(roomId);
  await notify(agent, roomId); // TTL aligned to op: a crash can't lock forever
}

ai-race-guard.ts

// AI answering first means AI failing first. The model is a flaky
// dependency: race it against a timer, and on stall or error fall
// straight through to a human — the user never gets a spinner.
try {
  const reply = await Promise.race([
    askModel(message),
    rejectAfter(10_000), // ten seconds, then it's a person's job
  ]);
  return send(reply);
} catch {
  return forceToAgent(message); // timeout OR error → human takeover
}

War stories

The day the cluster said no

Queue positions lived as scores, and the original update was read-modify-write: read the score, compute in memory, write it back. Under concurrency two updates interleaved and one silently overwrote the other. I rebuilt the updates as atomic increments wrapped in a transaction — and then production revealed our Redis cluster wouldn't accept the transaction at all. Because the score design was banded — status, load and waiting time each own their own order of magnitude — every state change was already expressible as a single atomic increment, so we degraded to pure atomic ops the same day. A fix isn't done until it survives the real topology — and data designed for plan B is why the fallback was a same-day change, not a redesign.

Two agents, one conversation

Two messages arriving in the same conversation at the same moment each decided a human agent was needed — and two agents got invited into one chat. The fence is a per-room lock: the first caller takes it and performs the invite, everyone else finds it held and walks away, and the lock's time-to-live is aligned to the operation so a crash can't leave the room locked forever. Anything triggered by "a message arrived" will eventually be triggered twice in the same millisecond — design for the second caller before it exists.

The model is a dependency you can't trust

AI answering first means AI failing first. Every model call races a ten-second timer, and on stall, error or timeout the conversation is handed straight to a human — the user always ends up with a person, never with a spinner. The model gets treated like any flaky third-party dependency: timeout, fallback, and never in the critical path without an exit. That guard fired in production more than once, and nobody noticed — which was the point.