Method lesson

What it's compared against: the null

A signal isn't compared against zero, but against what chance would have given in the same era and with the same duration. Getting the era wrong multiplies the bar by seven.

The question: My signal made 4% per trade on average. Is that a lot?

What was found

Entirely on when and for how long. A random 20-day window of Bitcoin has returned +8.63% on average over the full history, +3.86% post-2017 and +1.22% over the last two years. The same 4% trade is a failure in one context and a success in another. That's why the null has to be matched by DURATION and by ERA: random windows of the same length and from the same time slice as the trades being judged are drawn, and the signal has to beat that, not zero. The project had a real bug right here: its null function always drew from the FULL price array, even when scoring a recent subset, so a candidate from the last two years was compared against a bar seven times richer.

0.0%5.8%11.5%17.3%23.1%1.8%1.0%0.7%0.3%5d8.6%3.9%2.6%1.2%20d20.6%8.2%5.8%2.1%40dFull historyPost-2017Post-2020Last 2 years
Average return of a RANDOM Bitcoin window, by duration and by era. This is the bar to beat, and it changes by a factor of seven depending on where you draw from.docs/research-postmortem-and-plan-2026-09-07.md §2.4

Try it yourself

Set your signal's average result and chance's, in the same era and duration. The subtraction is the only thing that counts.

Excess over chance = Signal return − Return of a comparable random entry
Excess over chance

How much of the signal's result was just the market (%):

With the defaults — a 4% signal against the 3.86% chance gave post-2017 at 20 days — the excess is 0.14 points. 96% of the result was the market. Try changing the null to 8.63 (full history) or 1.22 (last two years) and watch the verdict change without touching the signal.

Illustrative example numbers for practice — not real data.

How it was tested, step by step

  1. Each of the signal's trades is taken and its exact duration and the time slice it happened in are recorded.
  2. Thousands of random windows with that SAME duration are drawn from within that SAME slice. That cloud of results is what entering at random would have given.
  3. The signal's score is its percentile within that cloud. A percentile of 0.52 means "indistinguishable from chance"; one of 0.95, "better than 95% of comparable random entries".
  4. The correction the project adopted: the range to draw from is now a MANDATORY argument, and the function raises an exception if that range doesn't contain the trades being evaluated. Plus a unit test that fails if anyone unmatches it again.
What this does NOT say. The interesting part is what happened when it was fixed: re-scoring the 13 rejected candidates with an era-matched null, ZERO change verdict. The bug was real, it had to be fixed, and it resurrected nobody. That's worth saying because it's the right way to close a bug: fix it, re-run what it touched, and publish that nothing changed — rather than leaving the doubt hanging or using it as an excuse to reopen dead candidates. A second defect in the same toolkit, also documented: the function called "block bootstrap" wasn't one, but an independent resampling of trades, which ignores temporal clustering and understates uncertainty when signals fire in clusters.

Tracking

This finding has no live number: it was measured on archived data, and putting a chart here would imply continuous tracking that doesn't exist. The figures are above, with the script that produced them.

← All research