Power: what the experiment can see
The thirty-second calculation that would have saved weeks of work: before running a test, knowing the smallest effect it could possibly detect.
The question: When a test says "there is no signal", is it saying that, or is it saying "I could not have seen one"?
What was found
It's failure mode number two of the eight the project documented, and probably the most expensive. Entire families were judged on between 5 and 63 trades: classic patterns with n between 5 and 63, Wyckoff with 17, the Nadaraya-Watson envelope with 8 to 11, exit timing with 5 or 6. At those observation counts you need an edge of around 10% PER TRADE to detect it reliably, and real edges in liquid markets are 1 to 3%. The consequence is that most "null results" from those weeks aren't null: they're NON-RESULTS. Power was never computed before running a test, so there was no way to tell "there is no signal" from "the experiment couldn't see one".
Try it yourself
The calculation to do BEFORE. Set the number of trades, the edge you expect, and how much a typical trade varies.
Minimum detectable effect at that trade count (%):
Try n = 17 (Wyckoff's event count) with a 2% edge: the minimum detectable effect comes out above 7%, three times larger than what was being looked for. That experiment could not have returned anything else.
Illustrative example numbers for practice — not real data.
How it was tested, step by step
- The intuition is simple: an edge is detectable when it stands out from the noise. The noise of an average over n trades isn't the dispersion of one trade, but that dispersion divided by the square root of n — which is why more observations help, and why they help less and less.
- The resulting signal-to-noise is t = edge x sqrt(n) / dispersion. By convention a t of around 1.64 is needed to declare one-tailed significance.
- Turning that formula around gives the minimum detectable effect: MDE = 1.64 x dispersion / sqrt(n). That number can be computed BEFORE having a single result, and it's what decides whether the experiment is worth running.
- The rule the project adopted: every run prints its MDE, and a null result without an MDE isn't filed as null, it's labeled an inconclusive experiment. And in the pre-registration, if the MDE is larger than the plausible effect, the experiment is simply NOT RUN.
| Trades | 1% edge | 3% edge | 5% edge | 8% edge | 12% edge |
|---|---|---|---|---|---|
| 20 | 20% | 20% | 30% | 70% | 90% |
| 35 | 10% | 30% | 60% | 90% | 100% |
| 60 | 10% | 20% | 40% | 90% | 100% |
Probability of detecting a real edge of that size, with the good test, at the number of trades on which almost every candidate in the project was judged. Read the 1-3% column, which is where real edges live.