About · DNI v7.0-dev · build 8.10 · formula DNI-32.0-FT

What DNI is, and what we have learned

DNI returns two separate results for a paper: a reference-integrity check of every DOI in its bibliography, and a novelty score from AI judges who read it against retrieved prior art. Each comes with its evidence and with everything it could not verify. It is a research instrument, not yet a validated product.

The two results at a glance

ResultWhat it answersHowStatus
Reference integrity (DOI input)Do the DOIs in the publisher-deposited reference list exist and point to the works printed?Deterministic lookups in Crossref, DataCite, OpenAlex and DBLP. No AI.Validated: pre-registered test passed (RIC-PREREG-1)
Reference integrity (full text)The same check on DOIs read from the manuscript itself.PDF text parsed, then the same lookups. No AI.Passed a pre-specified test (RIC-PREREG-3), registration committed after the data
Novelty scoreHow far the paper's contribution goes beyond the prior art it is compared with.Three AI judges (Gemini) read the paper against retrieved prior art.Not validated on manuscripts; on research ideas, ranks like frontier LLMs (all weak)
Text screeningInstruction-like text aimed at an AI reviewer.Pattern detector; quoted text, titles and reference lists are ignored.Advisory only; never changes a score

How a run works

StepWhat happensUses AIIn the novelty score
ReadText from the PDF or pasted manuscript; the same normalisation every time.No—
ExtractUp to 5 claims and up to 10 cited works the paper builds on.YesNo
RetrieveCited works resolved; uncited related works and a per-claim search, limited to work published before the paper.NoNo
Check referencesEvery DOI in the bibliography (up to 120) checked against the DOI system and scholarly indexes.NoNo: a separate result
JudgeThree judges score Uniqueness, Tension and Synthesis; a second panel joins if they disagree; Tension is re-asked under another framing.YesYes: 0.6 U + 0.3 T + 0.1 S
GateFixed rules decide what is asserted, flagged or withheld (next table).NoDecides what is shown

What DNI withholds instead of guessing

WhenWhat DNI does
A judge failsExcluded, never imputed. Fewer than 3 valid judges: provisional. None: no score.
No abstract on a DOI submissionNo score: the judges would be reading keyword tags, not the paper.
Judges disagree, or Tension changes with framingThe spread and the alternative reading are shown next to the score.
Score near a class boundaryNoted: a repeat run could change the label.
Fewer than 5 checkable referencesNo 10% verdict on reference integrity.
A reference cannot be checked by DOI (ACM IDs, retitled preprints)Marked unverifiable with a note, not counted as a fault.
Instruction-like text in the manuscriptShown with its location; the score is not adjusted.

Evidence

TestVersionDataResultCaveat
Reference integrity, pre-registered (RIC-PREREG-1)8.x200 PubPeer-flagged vs 199 matched controls; confirmation on 200 new vs 196Faulty references 32.6% vs 1.14% (28.6×). The rule fixed in advance (≥ 10% faulty) held on fresh data: 93.9% [89.4–96.6] vs 4.0% [1.9–8.0]Needs DOIs in the reference list. A flag is evidence for review, not a verdict
Reference integrity on full text, pre-specified (RIC-PREREG-3)8.9205 PubPeer-flagged vs 410 same-journal controls, open-access PDFsSame ≥ 10% rule, applied to DOIs read from the PDF text: 91.3% [86.6–94.5] vs 4.9% [3.1–7.6]; 0.16% of correct references wrongly faultedOnly with the 8.9 parser and matcher. Registration committed after the data. An earlier, smaller test (RIC-PREREG-2, 100 controls) was not confirmed
Reference integrity vs RefChecker (open source)8.646 flagged + 46 control papersDNI flags 93.5% vs 6.5%; RefChecker 88% vs 28%PubPeer flags are not all citation faults
Metascience Novelty Indicators Challenge: special mentionChallenge entry100,000 papers; replication package on 1,000Special mention (Challenge Works, with Nesta, RAND and the University of Sussex)Recognition of the entry, not an accuracy measurement
RINoBench novelty ranking (ML research ideas, expert labels 1–5)7.0277 test items, pre-registeredSpearman ρ 0.171 [0.05, 0.29]; gpt-5 0.140, o3 0.113. No paired difference with gpt-5, o3 or gpt-oss excludes zeroIdeas, not manuscripts. Every system tested is weak (ρ ≤ 0.17)
Repeatability7.020 items, run twiceSD 0.008; ρ between runs 0.977Stable is not the same as correct
Prior-art search recall8.x100 items, mean 27 known related worksClaim-level search: 0.19 hits per item (was 0.04)Recall under 1.5%: the main weakness
Real manuscripts8.x15 beta-tester PDFs15/15 complete; judge spread ≤ 0.05; about $0.05 per paperNo expert comparison yet
Drift check (serving)8.920 items, October vs November repeatsNovember repeats agree with each other (ρ 0.95) but differ from October more than October's own repeatsA change in the model serving, not in DNI's code; the serving model is now logged per call
More prior art for the judges (exploratory)8.1023 manuscripts, 4 settings, 2 repeats eachCited works compared: 4.4 → 8.9 per paper; uncited related works 1.9 → 3.1; same cost; scores move little beyond re-run noiseNo expert ratings on these papers: a change, not a proven improvement
UIUC Axiomatic Benchmarkv4.0 corePublished benchmark73.0% overall; 55.1% on Axiom 3Measured on the v4.0 formula, not the live reader

What can and cannot be claimed today

We can sayWe cannot say yet
Flag papers whose references are often faulty: 93.9% of PubPeer-listed vs 4.0% of controls on deposited listsThat a flag means misconduct: it means faulty references
Do the same from the manuscript text: 91.3% vs 4.9%, almost never faulting a correct reference (0.16%)That this holds for scanned or paywalled PDFs, which were not tested
Show every score with its prior art, judge spread, coverage and refusalsThat the novelty score is accurate, or fit to rank papers alone
Rank research ideas about as well as frontier models (ρ 0.171)That retrieval finds the relevant prior art (recall under 1.5% of known related work)

Agreement of the novelty score with expert ratings of full manuscripts has not been measured.

What did not work

Idea testedResult
Scoring novelty claim by claimRanking accuracy fell twice (Δρ −0.17, −0.19); kept as a display only
A probabilistic scoring model (TypeSafe Jev) as the Uniqueness judgeWorse than the current judges (ρ 0.217 vs 0.318); its confidence did not track its errors
Warning the judges about injected instructionsChanged scores on clean papers with no benefit; retired
Full-text reference test on 100 controls (RIC-PREREG-2)Not confirmed (upper bound 13% > 10%); a larger test then passed

What we have learned

  • Checking references is more reliable than judging novelty. A deterministic, no-AI check separated flagged from control papers on a pre-registered test; the novelty score is better than chance but weak, like every model tested.
  • A judge only weighs what it is shown. Retrieval finds under 1.5% of known related work, so missed prior art, not judging, limits the novelty score.
  • Some ideas made things worse and were dropped. Scoring claim by claim lowered ranking accuracy (Δρ −0.19); it is kept as a display, never scored.
  • Text inside a manuscript can steer AI judges (one hidden instruction moved a score from 0.53 to 0.86). DNI screens for it and says so; detection is not protection.
  • Reading references from a PDF needs care: broken line breaks, a paper's own DOI and ACM-style IDs caused every false alarm in a first audit. Fixed parsing brought the full-text check close to the deposited-list check (4.9% vs 4.0% of controls).
  • Withholding is part of the product: failed judges are excluded, never imputed; no abstract means no score; every advisory is shown.

Data: OpenAlex, Crossref, DataCite, Semantic Scholar, DBLP (open). Judges: Gemini (proprietary). PubPeer-flagged dataset: Michal Wójcik. Flagged papers are never named.


Enter a DOI or paste a manuscript to run an assessment.