Hand-building the platform's highest-volume data path — over ten million records a day

Every balance change a user sees, pulled from 10+ heterogeneous systems most of which my team didn't own — aligned record by record, always adding up.

over ten million records/day 10+ heterogeneous sources cross-team alignment hot / cold split

Situation

The platform's most demanding data path — a superset of the transaction backbone. Every balance change a user sees must be complete, ordered, and fast to query, but records originate from 10+ heterogeneous sources that disagree on shape, timing, and delivery guarantees: millions/day through the audited workflow path, the rest streaming straight from providers and internal systems most of which my team didn't own.

Approach

Hand-built the ingestion pipeline — composing a stream processor with a durable-workflow engine — and chose an analytics-oriented store deliberately, so long-window analytics volume wouldn't contend with production reads and writes for resources: hot recent history on the operational store with covering indexes; long-window analytics offloaded to a columnar layer fed by the pipeline; source-specific shapes normalized at the boundary, with per-source caching and channel-aware routing. The hard part was cross-team alignment, not code: I scanned every transaction path across two generations of the system, drove the direction for the owning teams, and reconciled historical mismatches record by record where amounts didn't line up — and shipped it while learning the real-time analytics database and stream stack on the fly, in two weeks.

Outcome

Over ten million records a day (climbing) ingested and queryable; recent-history reads stay fast while analytical queries stopped competing with production traffic.

War stories

Most of the work was comparing numbers

Nobody tells you this about building an ingestion pipeline: the code is the quick part. For a long stretch of development, one record subtype ran as a permanent probe — continuously reconciled against its sources while everything else was still moving.

Then the cross-checks widened: daily totals per record type compared against the analytics team's independent numbers, and per-user daily histories diffed record by record. When a figure disagreed, that was the day's work — trace which source dropped, duplicated or reshaped it, fix the boundary, backfill, compare again.

It's manual, it's unglamorous, and it's the only reason the number on a user's screen can be trusted. Correctness isn't a property you declare; it's one you keep re-earning.

The design, in code

Representative snippet of the production design — rewritten for publication; not company source.

ingest-boundary.ts

// 10+ sources disagree on shape, timing and delivery. Normalize at the
// boundary, then split by read pattern: hot recent rows stay operational,
// long-window volume goes to the columnar store so neither starves.
async function onRecord(raw: SourceRecord, source: Channel) {
  const rec = normalize(raw, source); // per-source shape → one canonical row

  await Promise.all([
    hotStore.upsert(rec),        // recent history — covering-indexed reads
    columnar.append(rec),         // analytics — off the production path
  ]);
}

reconcile.ts

// Correctness is re-earned daily: diff each source's totals against the
// analytics store. A mismatch is the day's work — trace, fix, backfill.
async function reconcile(day: string) {
  for (const type of recordTypes) {
    const mine = await hotStore.countByType(type, day);
    const theirs = await columnar.countByType(type, day);
    if (mine !== theirs) {
      logger.warn(`drift ${type} ${day}: ${mine} vs ${theirs}`);
      await flagForBackfill(type, day); // never silently paper over it
    }
  }
}