Zero fabricated data — except the ones I wrote myself

I built an equity research pipeline so a language model could never invent a number. It worked. Then I found three fabricated numbers in it. All of them were mine.

01 — What I was building

The NoCashFlow Stock Analyzer takes a ticker and returns a seven-page bilingual equity research report: structural trends, quarterly momentum, financial health, valuation against peers, price action and ownership, risks, and a synthesised verdict. Six specialist agents run in parallel; an editor agent synthesises their drafts.

The architecture has one rule, and everything else follows from it. I call it API-First: every number in the report is computed in Python from the data API. The models never see raw statements and never do arithmetic. They receive a JSON block of finished figures and are told, in the system prompt, what they may do with it:

This is enforced twice: once in the prompt, once in the architecture. No agent has a calculator. The single place a model influences the numbers is peer selection — it returns a list of ticker symbols, and even then the multiples themselves are fetched and median'd in Python.

I was confident about this design. I still am. It does exactly what it claims.

It just doesn't cover what I thought it covered.

02 — The three numbers

Not one of these came from a language model. The guardrail held perfectly. Every figure the models touched was correct, sourced, and reproducible. The fabrications were in the Python.

03 — Why they looked believable

The footer was true when I wrote it. That is the whole mechanism. A provider swap changes imports, environment variables, model identifiers — and every one of those changes announces itself, because the code stops running if you get it wrong. Prose doesn't. A sentence at the bottom of a generated report is read by the author exactly once, on the day it is written, and never again. It produced no error, no failed test, no degraded output. The one thing it did produce was a false claim about provenance, printed on every page, by a system whose entire purpose is provenance.

The scenario valuation was a tautology wearing a method name. Here is the line, unedited:

# mult is the scenario multiplier: 0.75, 1.00, 1.25.
# The company never enters the calculation:
"upside_downside_pct": round(((mult - 1) * 100), 1)
# → bear −25.0, base 0.0, bull +25.0. For every ticker, forever.

The reason this survived is more interesting than the bug: the base multiple was the company's own forward P/E, and the fundamental was its own forward EPS. Multiply those together and you get the current share price by construction. The base case could not have been anything other than zero. Once the base is pinned to the present, the bear and bull cases are just the numbers I picked.

And it looked like analysis. It had a method name (scenario_valuation), a structured output, three named scenarios, and — decisively — it was computed in Python. It inherited the credibility of the API-First rule without doing any of the work.

The score never met its own scale. Seven dimensions, each capped at 60–70 points, averaged. The maximum reachable value was 67. The threshold for the top verdict was 75. "Strongly Constructive" was unreachable code. So was "Negative" at the other end — the floor was 35. Every company the system had ever analysed landed in the middle three regimes, and that was not a finding about companies.

One dimension is worth its own sentence. The valuation score read:

"valuation": 60 if v.get("pe_ttm") is not None else 45,
# Measures whether a P/E exists. Not whether it is high.

A valuation dimension that scores data availability is a second data-quality metric wearing a different label.

04 — How I caught them

Not through monitoring — there wasn't any. Not through a failing test — there weren't any covering this. Not from a wrong output, because none of these produced one that looked wrong.

I caught them because I was writing a public build log entry for the project, and the card had a field labelled Result. To fill it honestly I had to read my own code closely enough to describe what it does, and describing it is a different act from writing it. Within an hour I had three defects that had been there since before the repository had a git history to date them by.

The lesson is uncomfortable and I think it generalises. Nothing surfaces a false assumption faster than having to state it in front of people who could check.

05 — Root cause

One sentence: API-First governs where numbers come from. It says nothing about whether they mean anything.

The rule was designed against a specific threat — a model asserting a figure that does not exist in any source. It defeats that threat completely. But it silently created a second category, and I never named it: figures that are deterministically computed and still empty of information. These are more dangerous than hallucinations, for three reasons. They are stable, so they never look anomalous. They are reproducible, so they survive review. And they are protected by the very rule that was supposed to make the report trustworthy — the moment a number is computed in Python, I stopped interrogating it.

Underneath that, three concrete causes. Provenance stored as prose: the provider name appeared in several places, none of them derived from anything the code executes. A valuation that used its answer as its input: anchoring on the company's own multiple guarantees the base case is the status quo. A composite score whose scale was never checked against its own thresholds: nobody had asked whether the top band was achievable.

06 — What I actually fixed

Two of the three are rebuilt. One is not, and I am publishing that rather than quietly closing it first, because which one survived is the point.

The scenario valuation now produces prices. Every scenario is fundamental × multiple, and the inputs are rounded before they are multiplied, so a reader who multiplies the two printed numbers lands on the printed fair value. Upside is always fair_value / price − 1. Both identities are asserted before anything renders; a valuation that disagrees with its own arithmetic raises and fails the stage instead of reaching the reader.

The tautology is not gone — it is disclosed. The base case is still, by construction, roughly today's price, because the base multiple is still the company's own. What changed is that the report now says so, labels it the market-anchored case, and puts the peer-anchored number beside it: one House Fair Value blending the scenario base and a peer-median relative value in fixed halves. When the peer set is too weak to trust, the relative leg is dropped outright rather than quietly down-weighted, and the report names the exclusion.

The engine also learned to refuse. No positive per-share fundamental with a matching multiple returns unavailable with a stated reason instead of a decorative number. An earnings multiple above 100× is a statement about a near-zero denominator, not about what the market pays for profits — the basis is skipped and the skip is reported.

Fixing it surfaced a fourth problem I hadn't seen. The valuation was computed at fetch time, before peer selection runs — so anything peer-anchored would have been computed against an empty peer set on every run, forever. The whole block is now rebuilt after the peers land, and re-checked for consistency there.

The score has a real scale — most of one. Dimensions now span 25–85 instead of 25–70, and the valuation dimension measures premium or discount to the peer median rather than whether a P/E exists. With the current weights the reachable range is 26.8 to 83.1, so the top verdict is now achievable. The bottom one still is not: "Negative" needs a score below 25 and the floor is 26.8. Two other things I described to myself as done are not: a missing dimension is still filled with 50 rather than excluded from the mean, and nothing reports how many of the seven were actually scored.

The provenance footer is still a written sentence. Half of that defect is closed: the message shown without a key now names the environment variable the loader actually reads, so the instruction is followable. The footer is not. It still credits the wrong provider, in both languages, in a literal string — the exact defect this article opens with, still live in the file while I write about it. The fix is designed and unwritten: read the provider from the client object that ran, collect model identifiers from the crew that ran, and fill the sentence from that state, so changing models changes the footer and there is no longer a string to forget.

The test suite is real, and it is not the twelve I claimed. 127 tests pass, 32 of them on the valuation engine, each written to fail against the old code rather than merely pass against the new. The one I care about most is test_scenario_output_is_never_a_ratio_around_one. Next to it: test_no_positive_fundamental_yields_an_unavailable_scenario, test_weak_peer_set_drops_the_relative_leg_rather_than_hiding_it, and test_inconsistent_valuation_fails_the_stage. What no test asserts yet is the one thing this section is about: that every regime on the verdict scale is reachable.

07 — What changed structurally

Two rules came out of this, and they are broader than the three bugs.

Any user-facing claim about the system must be derived from runtime state, not written as a literal. If the report says something about how it was made, that sentence has to be computed. Prose about a system is a cache with no invalidation. This is a rule I have stated and not yet enforced — see the footer.

Any deterministic figure must be able to say "unavailable." A calculation that always returns something will eventually return something meaningless, and a meaningless number that never admits it is worse than a gap. The valuation engine now has that path on every block. The verdict score does not: its missing dimensions still resolve to 50.

One structural fix did land in full, and it was found by instrumenting rather than by reading. A single specialist agent raising an exception propagated out of the thread pool and killed the entire run, discarding five completed sections. One missing section is a warning. A lost run is an outage. They were the same event. Each specialist is now isolated: its failure is recorded, its section is left empty, and the gap is surfaced to the editor by the contradiction detector.

What is still missing is the instrumentation itself. The pipeline cannot answer basic questions about its own runs — how long one takes, how many model calls it makes, how many figures came back empty, which steps were skipped and why. Progress is still human-readable strings passed to a callback and discarded. Until that is a structured log with a status per step and per-phase durations, every performance claim I make about this system is a memory, not a measurement.

08 — What I actually learned

Deterministic is not a synonym for correct. Moving a calculation out of the model and into Python removes one failure mode and introduces another: the unexamined formula. The model's output at least gets read — that is the whole point of a human review step. Python's output gets trusted, precisely because it came from Python.

The failure modes that matter in financial AI are not the loud ones. A model that produces an obviously wrong number is a solved problem; you notice, you fix it. A pipeline that produces a stable, plausible, reproducible, structurally meaningless number will pass every review you have, indefinitely, because nothing about it looks like a failure.

09 — What I still don't trust

The data source. Everything comes from one unofficial, rate-limited API. Retries and caching handle absence. Nothing in the pipeline handles wrongness — if a plausible but incorrect figure comes back, every downstream control passes it through, and every agent will faithfully interpret it. "The model didn't fabricate this" and "this is true" are different statements. I can currently only guarantee the first.

The peer set. A model chooses the comparables. The multiples are Python's, but the comparison set is not — and it now anchors half of the House Fair Value, which raises the stakes on that choice. It is disclosed in the footer. It is not solved.

The composite score. It is mathematically consistent now, and still has one regime nobody can reach. The thresholds are my judgement, and calling something "Constructive" because a number crossed a line I chose is exactly the kind of false precision this article is about. I have not decided whether it belongs in the report or should stay an internal ordering tool.

The provenance line. Named above, still open, printed on every page.

Narrative causality. Agents are instructed to label conclusions Observed, Inferred, or Speculative, and never to assert causality without supplied evidence. That is an instruction, not an enforcement. I have no mechanism that verifies compliance.

Any report nobody read. Not one of these should leave the system without a human reading it. That is not a temporary limitation pending better models.

The pipeline was built to stop a language model from fabricating data. It does that. What it could not do was stop me — and I had spent so long designing the guardrail that I never thought to check the side of the wall I was standing on.

Stock Analyzer is a live build. Finance Engineering carries the current architecture, stack and open issues.

This is a financial/technical educational note, not investment advice. Code references describe this project at the time of writing; the repository moves.