Method lesson

The test that wasn't measuring skill

For two weeks, the project's "mandatory" acceptance criterion scored a signal with a real +40% per trade exactly the same as one with no edge at all.

The question: If a test rejects everything put in front of it, is there nothing there, or is the test broken?

What was found

On 4 September 2026 a test comparing the signal quarter by quarter against buy-and-hold was declared "gold standard, mandatory before calling anything verified". From then on EVERYTHING failed. The diagnosis was done the right way: injecting a genuine edge of known size into the real Bitcoin series and measuring with the project's actual engine. The test's win rate was FLAT in alpha: 42.1% with zero edge and 42.1% with a 40% per-trade edge. The test was measuring market exposure, not skill. A signal that is in the market 35% of days loses most quarters against a 100%-invested buy-and-hold, by construction, in an asset with positive drift.

35.253.171.089.0106.90%5%10%20%40%real edge injected per tradePlacebo test (the one that works)Quarterly Wilcoxon (the retired one)
An edge of known size is injected into the real series and measured with both tests. The good one rises with the edge; the retired one is a flat line — it scores a real +40% the same as a zero.analysis/scripts/diagnose_methodology_2026-09-07.py

Try it yourself

Why the test was doomed: a signal that is only in the market part of the time is compared against one that is always in, in an asset that rises. Drag the exposure.

Difference against buy-and-hold = Drift x Exposure − Drift
Annual difference, with ZERO skill

Edge per unit of exposure just to break even (%):

At 35% exposure and 45% annual drift, the signal starts almost 30 points behind before getting a single call right. That's the hole two weeks of rejections fell through.

Illustrative example numbers for practice — not real data.

How it was tested, step by step

  1. The right way to diagnose a test is to inject a signal whose edge you know, because you put it there, and check whether it sees it. A test that can't see a real edge is worthless, however elegant its statistics.
  2. The project's placebo test DID behave properly: it went from 0.52 with no edge to 0.95 with a 10% one. When the two tests disagreed, the broken one was adopted. The whiplash of those days — a signal recovered, rejected and recovered again within 24 hours — was the symptom.
  3. The new criterion has three levels, and none is skipped: information content (does the signal know anything?, with a matched null and an information coefficient over the full panel), economic value (does it improve Sharpe at MATCHED EXPOSURE, with costs?) and robustness (holdout, parameter sensitivity, and a placebo of the hypothesis itself).
  4. The retired test is kept, but demoted: it's still computed as a descriptive measure of "how much beta does this give up?", and it REFUSES to issue a verdict.
What this does NOT say. Full honesty requires telling the other half: re-judging with the new criterion the 28 candidates the retired test had killed, 7 survive exploration and ZERO survive the holdout. That is, the earlier rejections were made with the wrong test, and still arrived at the same place. What changes isn't the survivor list: it's that we now know which were null and which simply weren't judgeable. And there was a second error of the same kind, caught midway inside the new criterion itself: the first version compared the tilt against a constant 1.0 baseline, which REWARDS beta — the mirror image of the bias that sank the retired test. With that version 7 candidates "survived". Corrected to a matched-exposure baseline, none did.

Tracking

This finding has no live number: it was measured on archived data, and putting a chart here would imply continuous tracking that doesn't exist. The figures are above, with the script that produced them.

← All research