Datalab releases OmniParseBench, an open benchmark for document parsing

Datalab has published OmniParseBench, an open benchmark for OCR and document parsing that the company says is designed around 16,288 tests. According to the PyPI project page and Datalab’s blog post, the benchmark evaluates single pages through binary checks rather than edit distance against one reference output. In practice, each test asks whether a given element is present in a provider’s output, in the right order, in a specific table cell, or inside a layout block of a certain kind. Datalab presents this as a way to focus on whether systems extracted the right content, not whether they matched one preferred output style.

A benchmark built as unit tests

The protocol described on PyPI is centered on page-level, yes-or-no assertions. A test can check one or more arguments, each corresponding to content at one place on a page. The benchmark groups results into three families: text, tables and layout. The headline score is the mean of those three family scores.

This design matters because, in Datalab’s telling, older benchmarks have become easier to optimize through formatting adjustments or benchmark-specific post-processing. In its blog post, the company argues that widely used alternatives can reward compliance with expected output conventions more than underlying model quality. Those criticisms are Datalab’s claims, not independent findings in the supplied sources, but they explain why the company says it built OmniParseBench and why it plans to use it for its own release reporting.

On coverage, PyPI states that each test is tagged across several dimensions. These include structure tags such as table, math, form and multi_column; rendering tags such as handwriting, tiny, rotated and degraded; role tags such as heading, caption, footnote, list, code, figure and header_footer; script tags based on ISO 15924; and language tags based on ISO 639-3. The tags do not decide whether a test passes. Instead, they let users slice the benchmark and score subsets of the data. Because one test can carry several tags, tag-level scores overlap and do not average back to the headline number.

How scoring and runs work

The benchmark can be run on Datalab’s dataset hosted on Hugging Face or on a user’s own dataset, according to the PyPI description. It also exposes lower-level scoring and prediction primitives so teams can plug it into their own evaluation pipelines. A run is resumable, and each provider setting is treated as its own run. The project says each run writes a scores.jsonl file with one line per test, while summary results can be read from summary.json.

Scoring does not reduce every outcome to pass or fail. PyPI lists four verdicts:

• pass
• fail
• unsupported, when a provider cannot produce the requested output
• error, when grading itself raises an issue rather than the provider failing

A notable methodological choice is that unsupported and error verdicts are excluded from every score rather than counted as failures. PyPI says this is intended to avoid penalizing vendors for outputs they cannot produce or for bugs in the scorer. At the same time, that means cross-system comparisons may depend partly on which output types a provider supports. A system that cannot emit a certain representation can have those tests left out of its score instead of receiving zeros on them.

The project also says provider adapters use the same timeout per call and, by default, each provider’s most accurate settings. Datalab’s tooling can call vendor APIs, but it asks for confirmation before doing so because those calls cost money. The package requires Python 3.11 or later and is distributed under Apache 2.0 for the code, while data and tests carry mixed licenses depending on source.

What is public, and what is not

From the supplied sources, the public release clearly includes code, documentation and a packaged benchmark on PyPI. PyPI also says the repository contains a way to run the benchmark using Datalab’s dataset on Hugging Face or a user’s own data. The same page points users to documentation covering the API, benchmark design, test construction, providers and tagging. It also states that if users find a wrong test, they can open an issue, which suggests the benchmark is meant to be inspectable and amendable.

However, the available sources do not provide a full breakdown of the 16,288 tests by family, language, script or document type. They also do not include aggregate leaderboard results for OmniParseBench itself. The angle’s figure of 16,288 tests is therefore reported here as part of the benchmark’s framing, but the supplied texts do not detail the exact split behind that total. Likewise, there are no benchmark-wide performance numbers in the source material showing which providers lead on text, tables or layout.

Results claimed around other benchmarks

Instead of publishing OmniParseBench leaderboard results in the supplied materials, Datalab’s blog focuses on why it believes existing benchmarks are insufficient. The company says olmOCR is saturated and cites examples where post-processing can change reported scores by around one or two points. It also argues that ParseBench can be strongly influenced by prompt wording, formatting conventions and specialized pipelines, especially for charts and visual grounding. These are substantive allegations made by Datalab, but they should be read as the company’s account. No independent validation is included in the provided sources.

Still, those claims illuminate what OmniParseBench is trying to optimize for. The benchmark’s yes-or-no checks are explicitly presented as harder to game than reference-matching approaches. Datalab argues that this structure should produce clearer error signals, because a failed test corresponds to a specific mistake on a specific page. That could make it easier for model developers and enterprise buyers to see whether a system struggles with, for example, rotated text, forms, handwriting or table structure instead of only seeing one blended score.

Limits visible in the release itself

OmniParseBench’s own design also creates limits. First, the supplied sources do not show complete benchmark-level outcomes, so outside readers cannot yet assess whether the benchmark produces rankings that differ materially from older alternatives. Second, because unsupported outputs are omitted from scoring, the benchmark may reward quality within a provider’s supported output range rather than breadth of output types. Third, tag slices overlap by design, which is useful for diagnosis but can make summary interpretation less intuitive.

There are also practical constraints. Running vendors requires API keys and paid calls. The benchmark can be resumed and compared across runs, but reproducibility may still vary with provider settings because each setting is its own run. Finally, the dataset inherits mixed source licenses, and Datalab notes that upstream datasets should also be cited. That does not weaken the benchmark’s value, but it does mean adoption in production evaluation workflows may require legal and operational review.

For English-speaking enterprises, the concrete change is that OmniParseBench offers a more inspectable way to test document AI systems page by page, with slices for tables, layout, scripts and hard rendering conditions rather than one opaque score. In practice, that could help teams compare vendors or internal models on the document features they actually care about, but they should also check which outputs are marked unsupported, review the benchmark’s licensing terms, and avoid treating any single headline score as a complete measure of document-understanding quality.