How a compile is measured.

GLM-OCR

Reference row: on the board for context, not ranked with the settled rows.

text Edit 0.0444 · table TEDS 0.4941 · reading-order Edit 0.1796 · over 1,599 pages · tag box_success_only_after_failed_inject

REF; 52 OmniDoc FAILED-injected; metrics pre-inject (47 empties); optional SUCCESS-only rescore

glm_ocr OmniDoc SUCCESS-only rescore (2026-09-06 20:27 KST)

- **Kept REF** (founder policy / `_build_comparison.py` REF set); board REF row updated.
- BEFORE (pre-inject, 47 empties in 1651 GT match): text Edit **0.0846**, formula **0.2960**, table Edit **0.5014**, TEDS **0.4874**, reading **0.2127**, pages **1651**, tag `settled_full_1651_parallel`.
- AFTER (box CPU workers=2, $0): text Edit **0.0444**, formula **0.2708**, table Edit **0.4914**, TEDS **0.4941**, reading **0.1796**, pages **1599**, tag `box_success_only_after_failed_inject`, elapsed **612.9s**, returncode=0.
- Prepare: 1599 nonempty SUCCESS-only; 52 FAILED excluded (47 empty+5 missing).
- GT: filtered OmniDocBench to 1599 SUCCESS-matched pages (full bench source; missing FAILED would otherwise score as empty under full 1651 GT).
- Note: initial zip extract via system unzip corrupted CJK filenames (90 orphans); fixed via Python UTF-8 extract before final score.
- Soft-skip installed: `omnidoc_raw/glm_ocr/` and `C:/Users/yspow/box_omnidoc_results/`.
- Prior REF metrics backed up under `omnidoc_raw/glm_ocr/BACKUP_pre_success_only/`.

Claude Opus 5 - Claude Code subscription surface

Reference row: on the board for context, not ranked with the settled rows.

text Edit 0.0896 · table TEDS 0.8284 · reading-order Edit 0.1675 · over 1,651 pages · tag settled_full_1651_parallel

REF; 21 prepare-skipped under full GT; speed missing from speed board

opus5_subscription ParseBench table ≈ 0

- Cause: GFM pipe tables vs scorer requiring HTML `<table>`.
- Fix path: pipe→HTML re-prepare + table-only rescore (no re-inference).

opus5_subscription olmOCR timing

- **overall 0.740372 is valid.**
- Recorded score_min ~0.2 min is anomalous (likely resume/cache) — **do not use for speed ranking.**

Infinity-Parser2-Pro

FOUNDER_EXCLUDED 2026-09-04 over GPU spend (H100x2, about $7/h per pod). Speed is reference-only; no quality reading was completed, so this model has no quality row.

Does compiled context change what a model gets right?

A held-out task set answered twice by the same model — once from the PDF alone, once with TAVONEL’s compiled context beside it. The measured delta, the conditions it was measured under and the deviations from the sealed protocol are on its own page.

Read the GDP.pdf run

Which reader should the router send a page to?

The board above. It answers what a page costs to read and how closely each reader reproduced it, which is the evidence a routing policy is learned from.

Native PDF

The provider receives the native PDF and the benchmark prompt, with no added context.

Compiled context

The same native PDF and prompt are paired with TAVONEL's sealed compiled context. This is the primary comparison with Native PDF.

Fixed retrieval

The same compiled corpus is queried through a fixed embedder and reranker, then cited evidence is passed to the same target model.

Adaptive routing

The same corpus and query pass through a sealed eligible routing policy. The receipt records the chosen candidate and, where available, its control counterfactual or shadow run.

Recovery changes the outcome

What this means for you: on a page that is hard to read, the recovery step is the difference between getting the document's content and getting an empty page.

Receipt R-01 · recovery counterfactual on olmOCR-Bench · Download the receipt · Read the finding in full

Compilation refuses more than it emits, sometimes

What this means for you: when a document points at something that was not supplied with it, the compiler refuses that document rather than emitting a world with a dead link in it.

Receipt R-02 · knowledge-compilation properties on a thousand-document corpus · Download the receipt · Read the finding in full

A configuration nobody could edit mid-run

The model, its revision, the input mode, the prompt and the run configuration are recorded as digests taken before the run started.

A denominator

The record names the population the run covered, and so does every individual metric on it.

The raw predictions

The corpus and the run's own output are bound by digest, so the scoring can be repeated by someone who does not trust ours.

A price snapshot

Cost per page means nothing without the prices it was computed at, on the date it was computed.

The failures the run produced

Published with the run, not summarised out of it. A table that survives only because its worst row was left out is worth less than no table.

Freeze the configuration

Model id, revision, input mode, prompt and run configuration are pinned before the run and recorded as digests. A run whose prompt was edited while it was running is one run of two configurations, and is comparable to neither.

Publish the denominator

The record names the population it was measured over, and so does every individual metric. The same percentage over a whole corpus and over the subset that failed are two different findings that look identical without it.

Publish what failed

The weaknesses ship with the run. A hypothesis that did not hold is a finding, and a table that only survives because its worst row was left out is worth less than no table.

Reproduce before comparing

Someone else's published score is recorded as theirs, with the source, and is never restated as something we measured. A comparison waits for a run executed here under the frozen configuration above.