Method lesson

The holdout and pre-registration

Freezing 2023 onward, never looking at it during exploration, and opening that door ONCE per candidate.

The question: How do you know whether a finding is real or the noise of the window it was found in?

What was found

There's no way to know from inside that window. However sophisticated the test: if the same data was used to CHOOSE the candidate and to JUDGE it, the judgement is contaminated. The only defence that works is boring: set aside a slice of data before starting, don't touch it, and use it once at the end. The project froze from 1 January 2023 — around 2.7 years and two distinct regimes — with a guard in the code that raises an exception if any script tries to read from there during exploration. And the rule that makes it work: the final gate opens ONCE per candidate. If it fails, the candidate dies; it isn't refitted and run through again.

Try it yourself

Why the gate opens only once. If you refit and retry on the holdout, each attempt is another roll of the dice.

Probability of passing by chance % = (1 − (1 − alpha)^attempts) x 100
Probability of passing with nothing real

Attempts until passing by chance is more likely than not:

With a single attempt the probability is the promised 5%. With fourteen it's already more likely than not that a candidate with no edge whatsoever clears the final gate. Nobody decides to cheat: they decide to "try one more variant", fourteen times.

Illustrative example numbers for practice — not real data.

How it was tested, step by step

  1. Before running anything a pre-registration is written: hypothesis, the economic mechanism by which it SHOULD work, data, exact test, computed minimum detectable effect, and decision rule. If the minimum detectable effect is larger than the plausible effect, the experiment isn't run.
  2. All exploration lives in pre-2023 data. There you can test, tune, discard and retry as many times as needed.
  3. When a candidate clears exploration, the holdout is opened ONCE, and that opening is recorded in the ledger with its date. Recording it is what stops it being repeated quietly.
  4. The result of applying this to the 28 candidates the previous criterion had killed: 7 survive exploration with p between 0.000 and 0.013, and NONE replicates in the holdout.
What this does NOT say. A holdout isn't magic and the project documents its own limitations. For the re-judged candidates the holdout was CONTAMINATED — they had been selected using the full history — which would bias toward passing; failing it is therefore strong evidence, but passing it would not have been. And for two of the new sources — ETF flows, top-account positioning — the global holdout is simply useless because the data starts inside it; there an internal split was used, declared and flagged as such, which is weaker and is said to be. Finally: a holdout is consumed. Every time it's opened it's worth a little less, and it doesn't regenerate.

Tracking

This finding has no live number: it was measured on archived data, and putting a chart here would imply continuous tracking that doesn't exist. The figures are above, with the script that produced them.

← All research