Hermes — building an LLM-in-the-loop trading research system

Why three repos instead of one Hermes monorepo?

The problem: the three concerns have completely different lifecycles. Backtesting is throwaway research — dozens of one-off scripts I run once and never touch again. The live assistant has to stay clean and always-on. The news thing is basically a static-site generator.

What I tried: early on it was one workspace. Mixing 40 experimental backtest scripts next to the always-on trading server made the trading repo look terrifying and made it hard to reason about "what's production."

How I solved it: split by lifecycle and by data source. Backtest is a research sandbox with results gitignored; trading is the disciplined production repo with one FastAPI process and a strict build contract; news is a self-contained generator that deploys to Vercel. Each has its own README, its own skill file, and its own changelog GitHub Action.

LLMs return "JSON" that isn't valid JSON. How do you parse it reliably?

The problem: the model wraps output in code fences, adds prose before the array, puts raw newlines inside string values, and — worst — gets truncated at max_tokens mid-object. json.loads fails on all of these and the whole day's render dies.

What I tried: plain json.loads, then a regex to strip code fences. Regex alone still died on truncation and embedded newlines.

How I solved it: a hand-written extractor that repairs instead of just parsing — strip fences, seek to the first bracket, escape stray newlines that appear inside strings, then walk the brace depth and auto-close any unclosed brackets when the response was cut off, before finally validating. If it still won't parse, return nothing and skip rather than crash.

# Repair, don't just parse: LLM JSON is often truncated mid-object.
def json_extract(raw):
    s = strip_fences(raw)              # drop ```json … ``` and prose
    s = s[s.find("["):]             # seek to the first bracket
    s = escape_newlines_in_strings(s) # raw \n inside values → \\n
    s += "]" * unclosed_depth(s)     # auto-close if cut off
    try:
        return json.loads(s)
    except ValueError:
        return None                    # skip the day, never crash it

The trading LLM sometimes returns no text at all. Why, and how do you handle it?

The problem: an Anthropic-compatible model can return a thinking/reasoning block first, so naively grabbing the first content block gives you the reasoning, not the answer — or an empty string.

What I tried: reading the first content block. Wrong block type gives garbage or nothing.

How I solved it: iterate the content blocks and return the first one whose type is text; log the block types seen when there's no text block, so I can debug it. And every caller degrades gracefully — if the call returns nothing, the pipeline falls back to a raw, non-LLM summary instead of failing the run.

# The model may emit a thinking block before the text block.
def first_text(blocks):
    for b in blocks:
        if b["type"] == "text":
            return b["text"]
    log.warning("no text block; saw %s", [b["type"] for b in blocks])
    return None   # caller falls back to a raw, non-LLM summary

How do you stop an LLM from inventing prices and numbers in the commentary?

The problem: if you let a model write "market narrative," it will confidently state a price or a % move that never happened. In a trading context that's dangerous.

What I tried: nothing structural at first — just trusting the prose.

How I solved it: the structural rule is that the LLM only narrates pre-computed numbers, never computes them. All screening, levels and scores are deterministic code; the model receives the finished numbers as input at low temperature and is instructed to cite only supplied data. The sentiment score, for example, is computed in code (SPY return + VIX + % above the 200-day average) — the model only writes the words around it.

Honest limitation: today this is a prompt-plus-low-temperature guardrail, not an automated verifier. I don't yet cross-check every number the model echoes against the input programmatically — that post-hoc echo-verification is a known gap I'd close next. It's tolerable for now because the model is handed finished numbers and asked only to narrate, so the blast radius of a slip is prose, not a traded level.

Yahoo Finance rate-limits bulk downloads. How did you get a full universe of prices?

The problem: pulling 1y daily bars for ~550 S&P 500 + Nasdaq 100 names gets throttled or returns empties from yfinance.

What I tried: per-ticker loops (slow, hits limits) and plain bulk download (gets blocked).

How I solved it: one batched download over a curl_cffi session with impersonate="chrome" so the requests look like a real browser — that session is the key fix for the rate-limits. Finnhub fundamentals are throttled separately at 1.1s/call (≤60/min) with a dedicated retry wrapper for HTTP 429.

Backtests lie. How do you keep a strategy honest before trusting it?

The problem: it's trivially easy to write an intraday backtest that shows big profit and is pure fiction — because it ignored commissions and slippage, or leaked future data into the entry.

What I tried: early backtests with no cost model. Numbers looked great and were meaningless.

How I solved it — two guards:

1. Stage 0 "Honest Engine" runs before any strategy logic: it picks a few random trades, prints the full cost breakdown, and asserts that net P&L actually equals gross minus cost — the run raises if the cost model isn't wired in. No strategy gets evaluated until costs are provably deducted.
2. No look-ahead: entry is decided from the first two composite bars only; the exit loop scans only bars strictly after entry. Plus a safety check that take-profit distance ≥ stop distance, which raises if violated.

# A backtest that hasn't proven it deducts cost isn't allowed to run.
def stage0_self_check(trade):
    cost = trade.shares * slippage_per_share() * 2 + commission()
    assert isclose(trade.net, trade.gross - cost), \
        "cost model not wired in — refusing to evaluate strategy"

Backtest P&L looked good but you still didn't go live. What was the check?

The problem: a strategy that "wins" on paper can still be one you shouldn't trade — too few trades, one lucky month, a single symbol carrying it.

How I solved it: a separate review agent is a required pre-live gate — every strategy must pass it before it's allowed to run live. It ingests the results and trades CSVs and generates a "challenge report" that deliberately hunts for weaknesses (concentration, drawdown, schema quirks) instead of celebrating the headline return. It even normalizes two different result-CSV schemas so old and new runs can be compared. A strategy doesn't graduate on a good return alone; it graduates by surviving the challenge report.

The daily news pulls the same story from five sources. How do you dedupe?

The problem: WSJ, MarketWatch and Google News all carry the same headline with different URLs — the digest fills up with duplicates.

What I tried: dedupe by URL. Doesn't work — same story, different URLs.

How I solved it: two-layer dedup — exact URL hash plus fuzzy title matching (ratio ≥ 85), so near-identical headlines from different outlets collapse to one. Collection itself is concurrent with a per-day cache so re-runs are cheap, and each source failure is caught and logged instead of killing the batch.

Timezones: you're in Malaysia trading US markets. What broke?

The problem: "yesterday's session," "earnings in N days," and "last completed session" are all wrong if you compute them in local time — the US market runs overnight my time and shifts with US daylight saving, and there are US holidays.

How I solved it: all trading-day logic computes in US/Eastern using an NYSE calendar (pandas_market_calendars), never hardcoded offsets; "last completed session" is the most recent closed NYSE session, which handles weekends and holidays. Only the cron schedule is expressed in local time. Fail-safe rule: an unknown earnings date pauses swing trading on that name.

Config-vs-secret discipline: what belongs in the code and what doesn't?

The problem: early on, two things that should have been configuration were baked into source — a personal notification target, and an API-required contact identifier (SEC EDGAR mandates a contact string in the request's User-Agent).

How I solved it: moved both to environment variables. The notification target now returns nothing and safely no-ops if unset; the SEC contact reads from an env var with a neutral placeholder default (the API requires a string, so it can't be blank). Every repo's HEAD is clean — all secrets and personal identifiers load from env or a gitignored .env.

The lesson: the distinction that matters isn't "secret vs public" — it's "does this identify a person or a machine?" An API-required field like a contact email still needs to be config, not a hardcoded default, because the default leaks you. And a removal commit is not a fix: once something is in version-control history, only a history rewrite plus credential rotation actually removes it.