| Reference integrity, pre-registered (RIC-PREREG-1) | 8.x | 200 PubPeer-flagged vs 199 matched controls; confirmation on 200 new vs 196 | Faulty references 32.6% vs 1.14% (28.6×). The rule fixed in advance (≥ 10% faulty) held on fresh data: 93.9% [89.4–96.6] vs 4.0% [1.9–8.0] | Needs DOIs in the reference list. A flag is evidence for review, not a verdict |
|---|
| Reference integrity on full text, pre-specified (RIC-PREREG-3) | 8.9 | 205 PubPeer-flagged vs 410 same-journal controls, open-access PDFs | Same ≥ 10% rule, applied to DOIs read from the PDF text: 91.3% [86.6–94.5] vs 4.9% [3.1–7.6]; 0.16% of correct references wrongly faulted | Only with the 8.9 parser and matcher. Registration committed after the data. An earlier, smaller test (RIC-PREREG-2, 100 controls) was not confirmed |
|---|
| Reference integrity vs RefChecker (open source) | 8.6 | 46 flagged + 46 control papers | DNI flags 93.5% vs 6.5%; RefChecker 88% vs 28% | PubPeer flags are not all citation faults |
|---|
| Metascience Novelty Indicators Challenge: special mention | Challenge entry | 100,000 papers; replication package on 1,000 | Special mention (Challenge Works, with Nesta, RAND and the University of Sussex) | Recognition of the entry, not an accuracy measurement |
|---|
| RINoBench novelty ranking (ML research ideas, expert labels 1–5) | 7.0 | 277 test items, pre-registered | Spearman ρ 0.171 [0.05, 0.29]; gpt-5 0.140, o3 0.113. No paired difference with gpt-5, o3 or gpt-oss excludes zero | Ideas, not manuscripts. Every system tested is weak (ρ ≤ 0.17) |
|---|
| Repeatability | 7.0 | 20 items, run twice | SD 0.008; ρ between runs 0.977 | Stable is not the same as correct |
|---|
| Prior-art search recall | 8.x | 100 items, mean 27 known related works | Claim-level search: 0.19 hits per item (was 0.04) | Recall under 1.5%: the main weakness |
|---|
| Real manuscripts | 8.x | 15 beta-tester PDFs | 15/15 complete; judge spread ≤ 0.05; about $0.05 per paper | No expert comparison yet |
|---|
| Drift check (serving) | 8.9 | 20 items, October vs November repeats | November repeats agree with each other (ρ 0.95) but differ from October more than October's own repeats | A change in the model serving, not in DNI's code; the serving model is now logged per call |
|---|
| More prior art for the judges (exploratory) | 8.10 | 23 manuscripts, 4 settings, 2 repeats each | Cited works compared: 4.4 → 8.9 per paper; uncited related works 1.9 → 3.1; same cost; scores move little beyond re-run noise | No expert ratings on these papers: a change, not a proven improvement |
|---|
| UIUC Axiomatic Benchmark | v4.0 core | Published benchmark | 73.0% overall; 55.1% on Axiom 3 | Measured on the v4.0 formula, not the live reader |
|---|