Short, practical lessons from operating production systems.
A personal side project — three repos, news → signals → offline validation, with the model kept on a tight leash: it narrates the numbers, deterministic code computes them. The honest problem-and-solution log, gaps included, lives on the Hermes page.
Instead of coordinating a live cutover, make collisions impossible. Boring beats clever when transactions are on the line — the full story is in the scaling case.
A fix isn't done until it survives the real topology. The war story — a race fixed with MULTI, a cluster that refused it — lives in the support case.
A threshold that lives in two systems will eventually disagree with itself, silently. We deleted the producer-side filter and left the number in exactly one place. Debugging is a graph search — fewer edges wins. The full chain is in the mission-pipeline case.
On transaction paths we assume every step will be retried, because someday it will be. Design for the retry and the 2am page never comes — the mechanics are in the backbone case.
The skill that matters is discernment — catching what the model got wrong before it merges. The full discipline, gates and failure file live on the AI page.
If a system can't survive your absence, it isn't finished. Documentation, runbooks and self-service forms aren't chores; they're how you leave.
A service once shipped without signal handling — it died mid-request on every deploy. Graceful shutdown is now a readiness gate: any change touching a service entry point without the drain checklist gets blocked.
Every morning before standup: message-queue backlog depth, workflow-retry counts, the slow-query log, CPU and memory on the hot services — and the shape of a leak before it pages anyone. Dashboards in Grafana; log-based alerting and incident forensics in Aliyun SLS; OpenTelemetry traces and Prometheus metrics underneath. Not because watching is comforting — because every one of those curves has a story, and I want to hear it before the users do.
When transactions or data need backfilling, someone writes a "one-off" script at midnight. Ours aren't one-offs: batched, resumable, idempotent, dry-run first — because a repair script that fails halfway is a second incident. I've written more of these than I can count; the discipline is what keeps them boring.
Before the platform earned its Redis layer, months went into the database itself: covering indexes matched to real query shapes, aggregation pipelines rebuilt stage by stage, full-text search where regex scans used to crawl. The fastest cache is the query that no longer needs one.