The exactly-once reliability layer for a cross-stack rewards pipeline
Four services, three languages, a reward path where a double-pay is real loss — I designed the layer that makes payout happen exactly once, with zero changes to the legacy system.
Situation
Marketing wanted automatic, time-boxed missions for users close to their next loyalty tier: detect them in real time, run a 7-day window, grant the upgrade instantly on success.
Constraint
The reward path is financially sensitive: a duplicate grant is a real loss, and events arrive at least-once from both the stream and the workflow engine — two independent sources of duplicates.
Approach
Built the chain end-to-end across four services in three languages — the boundaries followed the platform's existing seams (the streaming platform, the workflow engine, the reward facade), composing proven engines instead of building new ones: a Java/Flink job filters the event stream and computes entry-anchored goals from a lookup table; a Rust + Temporal engine runs one durable workflow per user; a NestJS facade grants through the existing upgrade path — zero changes to legacy systems. Exactly-once is layered: an atomic claims ledger (one chance per user, tier, month) gates enrollment, and a human-readable idempotency key (`mission:user:tier:month`) gates the grant — debuggable at a glance in production. Thresholds live in exactly one place — producers broadcast unconditionally, so two repositories can never disagree about the same number.
Outcome
Verified end-to-end against real production-like traffic in the pre-production lane — honest scope: it runs in pre-production (QAT), not production — with funnel metrics at every stage and a dry-run-by-default rollout I put through an outside red-team before calling it safe. The service decomposition wasn't mine to claim; my design was the reliability layer on top — exactly-once by an owned ledger and key. That layer, plus detect / orchestrate / grant with single-owner config, is the template for the next mission types.
The design, in code
Representative snippet of the production design — rewritten for publication; not company source.
// Two independent sources of duplicates — message re-delivery AND workflow
// replay. One atomic ledger claim per user·tier·month neutralizes both.
async function grantOnce(userId: string, tier: number, month: string) {
const key = `mission:${userId}:${tier}:${month}`; // human-readable, debuggable
// Insert-if-absent is the whole fence: the second caller collides and stops.
const claim = await ledger.insertUnique({ _id: key, at: Date.now() });
if (!claim.inserted) return; // already granted — a replay, not a new reward
await legacyUpgradePath.grant(userId, tier); // reuse old system, 0 changes
}// Every threshold has exactly one owner — a lookup table. Producers
// broadcast unconditionally, so two repos can never disagree on a number.
const GOALS: Record<number, number> = { 1: 500, 2: 2_000, 3: 10_000 };
// Entry-anchored: the goal is fixed at enrollment, immune to later edits.
function goalFor(tier: number): number {
const goal = GOALS[tier];
if (goal === undefined) throw new Error(`no goal for tier ${tier}`);
return goal;
}