The scores get tested in public

Validation report, published 2026-07-15

Nine quarterly cutoffs, 2024-04-01 through 2026-04-01. At each cutoff T the full scoring pipeline was refit on data resolved by T only, the board was ranked with production gates, and forward performance was measured on events resolving in the following 90 days against the as-of-T baseline. The design and the exact language rules for describing outcomes were committed before any number existed (specification 2026-07-10, reading guide 2026-07-13).

Tested strictly on data the rankings had never seen, the picture is mixed, and we publish it as it came out. Alert-eligible wallets, the slice the paid alerts draw from, went on to beat the prices they paid in six of the nine windows, by about +0.7 to +1.4 cents per share with confidence intervals clear of zero (largest window n = 2,236,857 wallet-events). In the weakest window (2025-04-01) the same slice did not demonstrate forward edge beyond price at this sample size, and the top decile of the board went on to negative forward excess (-0.54 c/share, 95% CI [-0.92, -0.14], n = 73,450 wallet-events). Both halves of that sentence carry equal weight.

Before publication, three independent layers had to agree: a timestamp-integrity audit of the resolution dating (run as a hard gate before any cutoff), a structural invariant check on the run’s output tables, and an independent recompute of both headline populations from the raw scored fills in separately written SQL. All 18 cutoff-population pairs agreed within 0.001 cents per share.

The gate slice: alert-eligible wallets

Alerts are the paid promise, so this table leads. Forward price-relative excess for wallets that were alert-eligible at T, with event-clustered bootstrap intervals (events are the independence unit; correlated legs in one event family count once).

In the 2025-04-01, 2024-04-01, and 2025-01-01 windows, alert-eligible wallets did not demonstrate forward edge beyond price at this sample size; 2025-04-01 is the weakest of the three, with a negative point estimate. In the other six windows, alert-eligible wallets went on to beat the prices they paid, by about +0.7 to +1.4 cents per share, with intervals clear of zero in each.

Decile tables

Forward excess by as-of-T rank decile. Under the pre-registered reading guide, the only licensed pairwise comparison is decile 1 against decile 10, and only where their intervals do not overlap; where the top decile’s interval includes zero, the mandatory sentence appears verbatim. Patterns the guide has no language for stay in the tables.

Rank correlation

Spearman between as-of-T rank and mean forward excess, wallets with at least 10 forward events. Point estimates only; no interval is computed for rho, so none is implied. At every cutoff the direction was the same: higher-ranked wallets went on to higher forward excess.

Persistence

Share of top-300 wallets at T still top-300 at the next cutoff, against the base rate of being ranked top-300 at all. Descriptive only; no interval accompanies this table, so no comparison adjectives are used. The first transition is degenerate: the early board ranked fewer wallets than the top-300 floor.

Riders

Estimation-population rider: dropped, pre-registered. Recorded 2026-07-11, before any cutoff ran: excluded market-maker and bot wallets shift no price-baseline bucket by more than 0.01 cents and no category prior moment by more than 0.03 cents, so the with-and-without grid was dropped in advance, and that decision was published on this page before the run.

Resolution-timestamp robustness rider: kept. Timestamp error matters to this design only where it could move a fill across a window edge. The rider recomputes both headline populations with every fill resolving within 24 hours of either window edge removed. Across all 18 pairs it moves the mean by at most 0.33 cents per share (the grid’s smallest-n negative window, where the rider moves it toward zero), and by at most 0.06 in every window with more than 100,000 wallet-events. No sign that survives an interval in the main grid changes under the rider.

Limitations

• Resolution dating. The windows slice on each market’s recorded close moment. An audit of 760,617 resolved markets with trades found 1.67% dated more than 10 minutes after the best available truth proxy and 0.21% more than 24 hours after it (median lateness 0.59 hours, p90 30.4 hours, dominated by sports). What matters to the design is boundary crossings: 271 events (0.036%) span a window edge closely enough for this error to move them between windows; the rider above bounds the effect. A further 2,107 resolved markets carry a scheduled-close fallback date in the future; they are excluded from every window by construction, which loses sample but cannot leak the future into a ranking.
• Survivorship. Forward windows only contain wallets that kept trading. A wallet that stopped after T contributes nothing forward, and stopping is not random.
• Category mix shift. The venue’s category composition moved across the nine windows; a rank built on one mix is tested against another. The two windows overlapping major sports cycles are the grid’s largest by an order of magnitude.
• Fees. Returns are price-relative and fee-exclusive throughout.
• Retagging churn. The venue re-tags markets; category-level slices inherit that churn.
• High-frequency series families are excluded from scoring and from this validation, per the standing methodology.

Already public

• The methodology: price-relative scoring, empirical-Bayes shrinkage, independence rules, and a limitations section that states what the scores cannot yet claim.
• Per-wallet calibration diagrams on every wallet page, computed from the same scored trades the rankings use.
• The receipts ledger: every featured alert, recorded at detection and graded on resolution.
• The statistics page: aggregate calibration, alert grading, and trader-skill numbers from the same scored data, refreshed with the board.