Callimachus is an open-source tool that checks the citations in a manuscript: whether each source exists and is the one cited, is formatted correctly, and actually supports the claim. It hands a language model the one call only a reader can make, weighing a claim against a source, and nothing else. The model never fetches a source, rewrites a claim, or names a verdict; the pipeline around it stays deterministic and on the record.
$ python run.py --input manuscript.pdf --accuracy standard
Sample findingAttention Is All You Need
The claim in the manuscript
“RNN sequence-to-sequence models have not been able to attain state-of-the-art results in small-data regimes[37].”
↓ cites [37] Vinyals et al., 2015, fetched and read in full
What that source actually says
“… when training on [the small training set] only, a baseline LSTM does not achieve any reasonable score, even with dropout and early stopping.”
The verdict
Partially supported
The source shows one model struggling on limited data, not that these models can’t. Its own attention variant scores competitively, so the broad claim isn’t established.
Most checkers answer one question, whether the reference exists or is formatted right, and hand you a number. Callimachus refuses to collapse identity, form, and support into a single score. A reference can be real but misidentified, correctly formatted but irrelevant, or on-topic yet unsupported by the text. Those stay separate, and every answer keeps its evidence.
01
Identity
Does the source exist, and is it the cited source?Resolved before any semantic support is credited.
02
Citation form
Does the bibliography entry pass the deterministic checks for its style?Style findings stay independent of identity and support.
03
Claim support
What does this source establish about this exact claim?Supported, partial, contradiction, related, off-topic, or non-decidable.
[ 02 ] How it works
The model proposes. The system controls the record.
A deterministic Python driver owns every phase: it reads the manuscript, decides which sources exist, and fetches their text before any judgement happens. A language model runs in exactly one phase, Verify, on one claim and one source at a time. It cannot admit a source, bypass grounding, write a terminal state, or compose the report. When required proof or integrity is missing, the run fails closed instead of guessing.
1.0
Parse
Read the manuscript (DOCX, LaTeX, PDF, Markdown, plain text, or HTML) into claims and references. The claims are extracted here, so the model is never free to invent one.
Deterministic
2.0
Resolve
Establish whether each source exists and is the one actually cited. A source is either found or it is not; a miss is recorded as unresolved, never invented.
Deterministic
3.0
Fetch
Retrieve and normalise the source text; identity and content hashes are retained as evidence.
Deterministic
4.0
Verify
The one place a model runs. It answers a chain of yes/no questions about a single claim and source; code composes the verdict, then a second judge re-checks the evidence.
Guarded jury
5.0
Report
Compose the deterministic report and audit ledger from admitted results only.
Deterministic
parse → resolve → fetch → verify → report
[ 03 ] Inside Verify
The model answers. The system decides.
This is the only step where a language model runs. It is handed one exact claim from the manuscript and the passages of the source Callimachus already fetched, and it is never asked to grade the citation in a single shot. It answers a chain of narrow yes/no questions, and deterministic code turns those answers into the verdict.
Q1 Does the source give usable support for what it is cited for?
yes → Q2no → Q3
Q2 Does it support the claim fully, at the same scope and certainty?
yes · Supportedno · Partial
Q3 Does the source instead show the opposite?
yes · Contradictionno → Q4
Q4 Is it in the same or an adjacent field at all?
yes · Relatedno · Off-topic
[ 04 ] Runs on any model
Model-agnostic by design.
Because the model does such a narrow job, weighing one claim against one source through bounded yes/no questions, you are not tied to a single provider. Point Callimachus at a hosted API, at a model on your own machine through Ollama or FreeToken, or at any OpenAI-compatible endpoint.
Anthropic
OpenAI
Gemini
Mistral
OpenRouter
GLM
FreeToken local
Ollama local
Any OpenAI-compatible endpoint
Small model? Add the RAG.
Large-context models read the whole source. For smaller local models, an optional dependency (requirements-rag.txt) enables extractive RAG: a deterministic in-document ranking (BM25) that feeds the model only the most relevant passages, within a fixed character budget. If that dependency is missing, Verify stops instead of silently truncating the evidence.
example · C7 / jury1_guard
[ 05 ] No silent recovery
When no admissible result survives, the report says so.
Here the Jury 1 proposal failed the verification guard. The pair stayed operationally complete but non-crediting: no plausible replacement verdict was invented to fill the gap. Missing evidence is recorded as missing, not converted into success.
Semantic outcomeNone
Resolutionjury1_guard
Creditingfalse
Evidence scopefulltext_complete
[ 06 ] What you get
The result is a record, not a paragraph.
Read the report first, then follow any finding down to the retained evidence, the source identity, the model's attempt, and the deterministic finalisation that produced it. The run ledger is an authoritative, auditable record, with a report history and integrity seal.
report.htmlSelf-contained, reader-first companion
report.mdDeterministic final report
run.sqliteAuthoritative audit ledger
sources/Normalised evidence with identity and hashes
journal + sealReport history and signature status
On the name & the mark
Callimachus of Cyrene compiled the Pínakes at the Library of Alexandria, the catalogue that first answered which scroll was truly which. This tool asks the same of every source a manuscript cites, and keeps the evidence attached to the answer.
The mark writes it as [C]: at once a citation reference and a harness: the brackets that let the model weigh a claim but never let it write the record.
[ C ] Inspect before you trust
Citation verification with the evidence still attached.
