The transaction backbone — from a legacy monolith to a contract-first service

Every financial transaction on the platform passes through it — millions a day, and it can never process one twice.

monolith → microservices millions of transactions/day idempotent by construction 2-min self-serve SDK 200+ types · one base workflow

Situation

Every financial transaction on the platform — withdrawal, top-up, loyalty reward, configuration change — must pass one audited create → review → approve → execute lifecycle. The legacy implementation was tightly coupled inside a monolith; every consumer re-implemented its own wiring.

Constraint

This is the platform's single point of trust: production pushes millions of transactions a day through it — into the tens of millions at peak — and a duplicate execution means a transaction is processed twice.

Approach

Rebuilt it as a gRPC microservice orchestrated by Temporal: one generic durable workflow runs every request type, with business extension points as child-workflow slots. Idempotency is structural — the workflow ID doubles as the creation key, so a retried step returns the existing record instead of creating twice. Integration became a published SDK: any service registers an executor with one import; adding a new request type is a self-service CI form, not a queue on my desk. 100+ SDK releases with disciplined semantic versioning.

Outcome

Millions of transactions a day in production. Duplicate execution is prevented by construction rather than caught after the fact. New-type integration went from a 30-to-60-minute, person-dependent process to two-minute self-service.

The design, in code

Representative snippet of the production design — rewritten for publication; not company source.

create-record.workflow.ts 
// Idempotency by construction — the workflow ID doubles as the creation key.
async function createRecord(req: CreateRequest): Promise<TxRecord> {
  const workflowId = `tx-${req.type}-${req.requestId}`;

  // Look up before write: a retried step finds the record its
  // first attempt made — and returns it. Never a duplicate.
  const existing = await repo.findByCreationKey(workflowId);
  if (existing) return existing;

  return repo.insert({ creationKey: workflowId, ...toRecord(req) });
}
retry-policy.ts 
// Retries are for infrastructure, never for business rejections.
const retryPolicy = {
  retryableErrors: ['UNAVAILABLE'], // a business "no" must stay a no
  backoff: { initial: '500ms', factor: 2, maxAttempts: 3 },
};
base.workflow.ts 
// One generic workflow runs every request type. A third-party callback
// can arrive before our own state has landed — so signals are buffered
// until the record they refer to exists, and behaviour changes ship
// behind a version marker so in-flight histories replay deterministically.
export async function proposalBaseWorkflow(req: Request) {
  const buffered: Signal[] = [];
  setHandler(decisionSignal, (s) => buffered.push(s)); // never dropped

  const record = await createRecord(req);         // idempotent create
  await condition(() => buffered.length > 0);      // drain once state exists

  if (patched('settle-v2')) return settleV2(record, buffered);
  return settleV1(record, buffered);       // old runs keep old logic
}

War stories

The redemption that didn't charge

Customer support found it before any dashboard did: a reward-redemption flow was handing out value without deducting the balance. A pre-charge step in the workflow was being skipped — silently.

Rollback first, debug second. The trail led to a shared contract file that had quietly fallen behind on one branch; a routine package upgrade activated the stale copy. Tests never caught it, because the change came from a different kind of project — the classic blind spot where two individually safe changes combine into one unsafe system.

The fixes were systemic, not local: the workflow fails loudly if the pre-charge step doesn't run; every release regression-tests every transaction path, no exceptions; and the incident report went up and out — the whole team knows this failure shape now.

One more thing grew out of it: the first version of our risk-alerting pipeline — every redemption event published onto the event bus, alerting the risk team in chat. Detection first, automation later. Incidents are cheapest when they compile into systems.

The signal that arrived too early

Our workflows accept human decisions and third-party confirmations as signals — and one day a callback landed before our own state had finished persisting. The workflow received a signal for a record that, from its point of view, didn't exist yet; worse, on replay the interleaving wasn't deterministic. The fix was structural: signals are buffered until the state they refer to has landed, and every behavioural change to the workflow code ships behind a versioned patch marker — so workflows already running mid-history replay the exact logic they started with, while new ones take the new path. In durable workflows, a code deploy is an event your running history has to survive.

The retry that had to return the same record

The engine retries failed steps — that's the point of it. But on a transaction path, a retried create is a potential duplicate record, and a duplicate record is a transaction processed twice. So creation isn't guarded by a check bolted on afterwards; it's guarded by construction: the workflow ID doubles as the record's creation key, and every create looks up before it writes — a retry finds the record its first attempt made and returns it. Updates carry status guards, and illegal state transitions throw instead of writing. When the failure mode is financial, idempotency can't be a best effort; it has to be impossible to get wrong.

Four environments, one engine

Several environments shared one workflow-engine cluster, and workers began stealing each other's tasks — a developer's local worker could quietly pick up and fail a task that belonged to a shared test lane. The fix was a naming discipline, not more hardware: task-queue names resolve per environment — production gets a fixed name, local runs are suffixed with the developer's identity, test lanes with the environment. Boring, but it ended a whole class of "works on my machine, breaks your test run" ghosts. Shared infrastructure needs namespacing before it needs capacity.