Native PDF
The provider receives the native PDF and the benchmark prompt, with no added context.
How a compile is measured.
Every result here names the population it was measured over and links the receipt it came from. The recovery study tests our own pipeline with and without its recovery step.
TAVONEL research · same-pipeline comparison
Recovery makes a measurable difference on difficult pages.
80.6with recovery
53.7without recovery
olmOCR-Bench score across 1,403 documents and 8,413 checks, measured 2026-08-08. Only the recovery step changed between the two runs. This is a research measurement, not a live service rate or a competitor comparison.
Method and limits
On olmOCR-Bench, scored by the benchmark's own evaluator at revision cfa88c1e, the same pipeline scored 80.6 with the recovery step and 53.7 with only that step switched off — a gap of 26.9 points, 95% confidence intervals 79.62–81.57 and 52.62–54.93, which do not overlap. Model, evaluator revision, corpus, source manifest, test set and settings were identical; the only difference was whether the documents recovery delivered carried their content. Measured 2026-08-08 over 1,403 documents and 8,413 checks. One category, headers and footers, scores higher without recovery, because a check that a phrase is absent passes trivially on an empty page — the no-recovery figure is generous rather than harsh.
The qualification contract defines what a knowledge-compilation result has to carry: a frozen configuration, corpus and output digests, a named denominator, the complete per-category result, and reproducible scoring material.
GDP.pdf evaluation design
GDP.pdf is a public set of 100 held-out tasks across ten professional domains. We have published the PDF-alone and PDF-plus-compiled-context arms with their actual conditions and deviations. Fixed retrieval and adaptive routing remain unrun; the four-arm design is not a completed benchmark or a leaderboard score.
Review the public dataset and reference harness.
Compiled context
The same native PDF and prompt are paired with TAVONEL's sealed compiled context. This is the primary comparison with Native PDF.
Fixed retrieval
The same compiled corpus is queried through a fixed embedder and reranker, then cited evidence is passed to the same target model.
Adaptive routing
The same corpus and query pass through a sealed eligible routing policy. The receipt records the chosen candidate and, where available, its control counterfactual or shadow run.
Published supporting research
These findings remain scoped to the population and question named in each downloadable receipt.
Recovery changes the outcome
What this means for you: on a page that is hard to read, the recovery step is the difference between getting the document's content and getting an empty page.
Receipt R-01 · recovery counterfactual on olmOCR-Bench · Download the receipt · Read the finding in full
Results by document type
What this means for you: results depend on the kind of document, so the per-type breakdown in the receipt is a better guide than the overall average.
Receipt R-03 · official olmOCR-Bench evaluation, by document type · Download the receipt · Read the finding in full
Compilation refuses more than it emits, sometimes
What this means for you: when a document points at something that was not supplied with it, the compiler refuses that document rather than emitting a world with a dead link in it.
Receipt R-02 · knowledge-compilation properties on a thousand-document corpus · Download the receipt · Read the finding in full
The eight metric families
Document reading
What the reader recovered from the page, and what it invented or dropped: text, layout, tables, formulas, reading order, and the coordinates every later stage binds to.
- text fidelity to the source
- layout
- table
- formula
- reading order
- region placement
- hallucinated and omitted content
- latency
- GPU memory
- cost
Evidence
Whether each compiled statement is bound to the source region that actually supports it, and how much of the world is bound at all.
- evidence binding precision
- evidence coverage
- source citation exactness
- region correctness
Identity
Whether two mentions of one thing became one object, and at what cost in wrong merges, wrong splits, and cases left open for a person.
- entity resolution precision
- false merge
- false split
- unresolved rate
Knowledge
Whether the objects, claims and relations built on top of the read are correct, valid against the schema, and honest about the places sources disagree.
- claim correctness
- relation correctness
- ontology validity
- conflict detection
Temporal
Whether the world knows which revision it is holding: what has been superseded, what has gone stale, and what a past answer stood on at the time.
- stale knowledge rate
- supersession correctness
- point-in-time correctness
Recompilation
Whether a source change was traced to exactly the parts it invalidated — no wider, and crucially no narrower — and what that saved against rebuilding everything.
- affected set precision
- affected set recall
- work avoided
- cost avoided
- equivalence with a full rebuild
- publish refusal correctness
Ask
Whether an answer stands on cited source, points at the right region, and declines when the world cannot support it.
- grounded answer precision
- citation precision
- abstention correctness
- stale answer prevention
Operations
What it costs and how long it takes for a change in a source to reach a world an agent is allowed to read.
- median and 95th-percentile time from source change to Active World
- cost per 1,000 pages
- cost per changed semantic unit
What a result has to carry
Every figure published here binds to a record, and five things have to be true of it before the figure may be read as a result.
A configuration nobody could edit mid-run
The model, its revision, the input mode, the prompt and the run configuration are recorded as digests taken before the run started.
A denominator
The record names the population the run covered, and so does every individual metric on it.
The raw predictions
The corpus and the run's own output are bound by digest, so the scoring can be repeated by someone who does not trust ours.
A price snapshot
Cost per page means nothing without the prices it was computed at, on the date it was computed.
The complete result
Every category the run covered is in its receipt, not only the figure quoted on a page.
The complete receipt schema
A record missing a digest, or missing the population a rate was measured over, is refused by the build rather than rendered with a blank cell. These are the fields the validator checks.
- Dataset
- Which corpus was read.
- Dataset version
- Which cut of it, because corpora are edited.
- Corpus digestsha256
- sha256 over the corpus, so “the same dataset” is a checkable statement rather than a claim.
- Denominator
- The population every rate in the record was measured over, and how many members it has.
- Arm
- Which arm of the preregistered comparison produced the predictions, by the name the preregistration gives it.
- System under test
- tavonel_live for the product path, tavonel_research for a research pipeline, vendor for a third-party service we ran under the same conditions.
- Model id
- The exact model, never the vendor or the family name.
- Model revision
- The revision served on the day. Two runs of “the same model” are not the same run.
- Input mode
- What the model was actually given: the page image, extracted text, or both.
- Prompt digestsha256
- sha256 of the prompt, frozen before the run started.
- Config digestsha256
- sha256 of the run configuration: schema, decoding, retries, thresholds.
- Compiler version
- The TAVONEL build that compiled the world the metrics were read from, or “notApplicable” for a vendor arm, which compiles none.
- Hardware
- The machine. Latency and cost are properties of a machine, not of a model.
- GPU
- The accelerator, or “none” for a CPU run. The field is answered, never left blank.
- Runtime
- The serving runtime and its version.
- Price snapshot
- The prices in force on the day, because every cost figure expires.
- Vendor terms digestsha256
- sha256 of the vendor's service terms as captured for the run, or “notApplicable” for an arm that is not a vendor's.
- Raw result digestsha256
- sha256 of the unaggregated predictions, so an average can be recomputed by someone else.
- World digestsha256
- sha256 of the compiled world the metrics were read from, or “notApplicable” for a vendor arm, which compiles none.
- Run receipt digestsha256
- sha256 of the receipt that ties all of the above to one execution.
- Date
- The day the run happened, as YYYY-MM-DD.
- Metrics
- Every figure, with its family, its unit, and its own denominator.
- Comparison basis
- same_condition for a run we executed; quoted for someone else’s published score, which stays theirs.
- Published failures
- What the run got wrong, recorded in its downloadable receipt.
Qualification rules
Freeze the configuration
Model id, revision, input mode, prompt and run configuration are pinned before the run and recorded as digests. A run whose prompt was edited while it was running is one run of two configurations, and is comparable to neither.
Publish the denominator
The record names the population it was measured over, and so does every individual metric. The same percentage over a whole corpus and over the subset that failed are two different findings that look identical without it.
Keep the complete result in the receipt
Every run ships with a downloadable receipt that holds its complete per-category result, so any figure on this site can be checked against the whole run it came from.
Reproduce before comparing
Someone else's published score is recorded as theirs, with the source, and is never restated as something we measured. A comparison waits for a run executed here under the frozen configuration above.
North Star metric
Definition
Verified Fresh Knowledge Coverage
The share of knowledge in an Active World that carries source evidence, passes validation, and agrees with the latest revision of the source it came from.
Supporting metrics
- Evidence Coverage
- Stale Knowledge Rate
- Recompile Avoidance
- Equivalence Pass Rate
- Identity Review Rate
- Conflict Rate
- 95th-percentile time from source change to Active World
- Grounded Ask Precision
- Correct Abstention Rate
Also in the trust case: Research notes · Evidence · Reproducibility · Research · Trust
Run this on your own files
Tell us about your sources and we will scope an evaluation with you.