Method lesson

Multiplicity: the best of many tries

With 432,053 hypotheses tested, the best result from pure chance already sits at 4.6 standard deviations. Anything below that is not a finding.

The question: If I test many things and keep the best one, what does it have to score for it to mean anything?

What was found

A p-value of 0.01 means "this would happen by chance one time in a hundred". If you test a hundred thousand things, it will happen a thousand times by chance. Put that way the logic is obvious, and it still sinks more quantitative research than anything else: the project's threshold scanner alone enumerated 431,256 hypotheses, ranked moreover by cumulative return — exactly the wrong metric, dominated by a few multi-year holds from 2010-2013. Adding the other families, the two-week total is on the order of 10^5 to 10^6 tests, with no global correction and no held-out slice of data. The two survivors were chosen after the fact from that same set: their nominal significance isn't interpretable.

Try it yourself

Drag how many hypotheses are tested and see where the bar lands. The scale is logarithmic: each step multiplies the number of tests by ten.

Threshold |z| ~ sqrt(2·ln N) − (ln ln N + ln 4pi) / (2·sqrt(2·ln N))
Best result expected from pure chance (|z|)

Number of hypotheses that assumes:

The default (10^5.64, about 437,000) sits just above the 432,053 real hypotheses the project's ledger has accumulated, and gives the same documented 4.6. Compare it against what a single well-posed test needs: 1.64.

Illustrative example numbers for practice — not real data.

How it was tested, step by step

  1. If you draw N random numbers from a normal distribution, the largest of them grows with N in a known and fairly slow way: roughly the square root of 2 x ln(N), with a small correction.
  2. At 432,053 tests that expected maximum from pure noise sits at 4.58 standard deviations. Any scanner candidate below that is, literally, what you would expect with no signal at all.
  3. The project's solution wasn't a finer statistical adjustment but a LEDGER: a file accumulating every hypothesis tested, with its family and date, seeded with the real history. The correction is applied against that global counter, not against today's session's test count.
  4. And the strategic conclusion, which is the opposite of the usual intuition: the more cells you test, the higher the threshold rises, and the threshold rises faster than any real signal. Testing more combinations on the same data doesn't get you closer to the answer: it moves you further away.
What this does NOT say. Counting hypotheses has its own nuance, and the project states it: treating three horizons of the SAME signal as three independent tests is excessively conservative, because they aren't. That's why the adopted criterion doesn't rest on the p-value alone: it rests on a consistent sign across horizons, the same sign out of sample, low correlation with what's already validated, and a mechanism written down first. And there's a prior check that stops the counter being inflated with smoke: measuring overlap between candidates BEFORE counting them as separate findings. The project found five scanner "findings" overlapping each other between 40% and 100% — the same signal counted five times.

Tracking

This finding has no live number: it was measured on archived data, and putting a chart here would imply continuous tracking that doesn't exist. The figures are above, with the script that produced them.

← All research