← Use Cases

AI Math Citations - Do They Check Out

AI Math Citations - Do They Check Out

What this research found

AI Math Citations - Do They Check Out

Research question

Does a bibliography entry identify a real work, and does that work support the surrounding claim?

Audits a sample of bibliography entries from AI-written mathematics papers and checks a smaller set of nearby claims. The final revision separates direct publication-record evidence from later user-supplied notes.

Data and comparison design

The sample contains five references from each of 20 papers, selected with a recorded seed. Bibliographic identities are first reconciled with publication records. The smaller claim-support exercise distinguishes metadata, abstract-level evidence and inspected theorem text. Later operator-supplied notes are retained as a separate evidence layer rather than silently promoted to direct source verification.

Recorded findings

Reference classificationDirect publication-record passFinal pass including supplied notes
Match8088
Minor bibliographic difference87
Demonstrated genuine mismatch00
Unresolved125
Final reference classificationCount
Match88
Minor bibliographic difference7
Demonstrated genuine mismatch0
Unresolved5

Seven identity upgrades rely on supplied notes; Popa’s historical date is also resolved through them. Unverified dates, editions and pagination remain explicit.

For the ten claims: 3 supported narrow attributions, 3 qualified/version-limited, and 4 unverifiable. All six favorable assessments rely on supplied operator notes—not independently inspected theorem text. The unresolved claims remain Stark, Kasteleyn, Osterwalder–Schrader and Eskin.

What the comparison shows

An online-first versus print-year difference is not automatically a false citation. Equally, a matching DOI does not establish that the cited source proves the mathematical statement.

Assumptions and limits

Some identity upgrades and favorable claim assessments rely on supplied notes rather than independently inspected theorem text. Unresolved references and version-dependent support remain explicit.

Sources and research materials

The original .science export and matching poster are preserved byte-for-byte from the upstream case. The package contains the conversation, session metadata, and the resources inventoried in its internal manifest. This introduction summarizes the archived record; the research, external sources, and model evaluations were not rerun for this import.

How this research was produced

AIPOCH planned and ran this investigation end to end — searching the literature, producing the figures, and drafting the report. The full session transcript is available to inspect.

Share:

Run this kind of analysis on your own question

Start a session and see how an AI co-scientist accelerates your research. Pay-as-you-go, nosubscription required.