Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation基于 LLM 的语义评估对表格提取进行基准测试的 PDF 解析器

Existing benchmarks for table extraction fall into two broad categories: table recognition datasets that operate on cropped table images, and document-level benchmarks that evaluate table extraction in the context of full pages.现有的表格提取基准可分为两大类：在裁剪的表格图像上进行的表格识别数据集，以及在完整页面上下文中评估表格提取的文档级基准。

Table recognition datasets have driven progress in table structure recognition from isolated images. Large-scale datasets such as PubTabNet [46] (568K images from PubMed Central), FinTabNet [45] (113K tables from financial reports), TableBank [15] (417K tables via weak supervision), PubTables-1M [36] (nearly one million scientific tables), and SynthTabNet [26] (600K synthetic tables with controlled structure and style variation) provide extensive training and evaluation resources, while SciTSR [4] contributes 15K tables with structure labels derived from LaTeX sources. These datasets also gave rise to the dominant evaluation metrics: PubTabNet introduced TEDS, and GriTS [37] later proposed grid-level evaluation as an alternative. The ICDAR 2021 competition [11] complemented these efforts by targeting table image to LaTeX conversion, a task recently advanced by reinforcement learning over multimodal language models [20]. While instrumental for advancing table recognition, these datasets provide cropped table images rather than full documents, making them unsuitable for benchmarking end-to-end PDF parsing pipelines where tables must first be detected within a page of mixed content.表格识别数据集推动了对孤立图像中表格结构识别的进展。 大规模数据集如 PubTabNet[46]（来自 PubMed Central 的 568K 图像）、FinTabNet[45]（来自金融报告的 113K 表格）、TableBank[15]（通过弱监督获得的 417K 表格）、PubTables-1M[36]（近一百万科学表格）以及 SynthTabNet[26]（600K 合成表格，具备受控结构和样式变化）提供了丰富的训练和评估资源，而 SciTSR[4] 则贡献了 15K 带有结构标签的表格，来源于 LaTeX 源码。 这些数据集也催生了主流评估指标：PubTabNet 引入了 TEDS，随后 GriTS[37] 提出了网格级评估作为替代。 ICDAR 2021 竞赛[11] 通过针对表格图像到 LaTeX 的转换进一步补充了这些工作，近期通过多模态语言模型的强化学习[20] 推动了该任务的进展。 虽然这些数据集对表格识别的推动作用显著，但它们提供的是裁剪的表格图像而非完整文档，不适用于评估需要先在混合内容页面中检测表格的端到端 PDF 解析流水线。

Document-level benchmarks evaluate table extraction from complete pages where tables appear alongside text and figures. The OmniAI OCR Benchmark [27] evaluates overall document extraction accuracy but lacks table-specific metrics, while olmOCR-Bench [31] includes 1,020 table-specific unit tests across 1,402 PDFs but focuses on cell-level pass/fail verification rather than holistic table quality assessment. OmniDocBench [28] (1,355 pages) and READoc [19] (3,576 documents, 15 parsing systems) go further by including explicit table evaluation, yet both rely on TEDS and edit distance metrics that capture structural similarity without assessing semantic equivalence. PubTables-v2 [35] provides the first large-scale benchmark for full-page and multi-page table extraction (467K single pages with 548K tables and 9,172 multi-page documents), extending PubTables-1M from cropped images to document-level evaluation using GriTS. The ICDAR table competitions [7, 6] established early document-level benchmarks on small collections using cell adjacency relations, and SCORE [16] more recently addresses evaluation methodology by proposing interpretation-agnostic metrics that handle legitimate structural ambiguity, though still operating at the structural rather than semantic level. Soric et al. [38] benchmark nine extraction methods across three document collections totaling approximately 37K pages, relying on TEDS and GriTS for evaluation.文档级基准评估完整页面中的表格提取，表格与文本、图形并存。 OmniAI OCR Benchmark[27] 评估整体文档提取准确性，但缺乏表格专用指标；olmOCR-Bench[31] 包含 1,020 项表格单元测试，覆盖 1,402 份 PDF，但侧重于单元格级的通过/失败验证，而非整体表格质量评估。 OmniDocBench[28]（1,355 页）和 READoc[19]（3,576 份文档，15 种解析系统）进一步加入了显式的表格评估，但仍依赖 TEDS 和编辑距离指标，仅捕捉结构相似性而未评估语义等价性。 PubTables-v2[35] 首次提供大规模的完整页和多页表格提取基准（467K 单页含 548K 表格，9,172 份多页文档），将 PubTables-1M 从裁剪图像扩展到文档级评估，并使用 GriTS。 ICDAR 表格竞赛[7, 6] 在小规模集合上建立了早期的文档级基准，采用单元格邻接关系；SCORE[16] 最近通过提出解释无关的指标来处理合法的结构歧义，尽管仍在结构层面而非语义层面工作。 Soric 等[38] 在三个文档集合（约 37K 页）上基准了九种提取方法，仍使用 TEDS 和 GriTS 进行评估。

Evaluating extracted tables against ground truth requires metrics that jointly assess structural fidelity and content correctness. Directed adjacency relations (DAR) [7], introduced in the ICDAR 2013 Table Competition as the first metric designed specifically for table structure evaluation, captured local cell neighborhoods but could not represent global table structure, motivating the tree-level and grid-level approaches that followed.对提取的表格进行评估需要同时衡量结构保真度和内容正确性。 有向邻接关系（DAR）[7] 在 ICDAR 2013 表格竞赛中首次作为专门针对表格结构的指标出现，捕捉局部单元格邻域，但无法表示全局表格结构，促使后续的树级和网格级方法的出现。

Tree Edit Distance-based Similarity (TEDS). TEDS [46], introduced alongside PubTabNet, has become the de facto standard for table recognition evaluation. Both predicted and ground-truth tables are represented as HTML trees whose leaf nodes (<td>) carry colspan, rowspan, and character-level tokenized content. The tree edit distance is computed with a unit cost for structural mismatches and normalized Levenshtein distance for cell content, yielding $\mathrm{TEDS}=1-d\,/\,\max(|T_{\mathrm{pred}}|,|T_{\mathrm{gt}}|)$, where $d$ is the edit distance and $|T|$ the number of nodes in each tree. Since both structure and content are compared at the character level, the score is sensitive to markup choices (e.g., <thead> vs. <tbody>, or <th> vs. <td>) and surface-level string differences.基于树编辑距离的相似度（TEDS）。 TEDS[46] 与 PubTabNet 同时提出，已成为表格识别评估的事实标准。 预测表和真实表均以 HTML 树表示，叶节点（<td>）携带 colspan、rowspan 和字符级标记内容。 树编辑距离采用结构不匹配的单位代价，单元格内容使用归一化的 Levenshtein 距离，得到 TEDS=1−d/ max(|T_pred|,|T_gt|)，其中 d 为编辑距离，|T| 为树中节点数。 由于结构和内容均在字符层面比较，分数对标记选择（如 <thead> 与 <tbody>，<th> 与 <td>）以及表层字符串差异敏感。

Grid Table Similarity (GriTS). GriTS [37] addresses the HTML sensitivity of TEDS by operating directly on the table’s 2D grid representation, using factored row and column alignment via dynamic programming. It defines separate metrics for structural topology ($\mathrm{GriTS}_{\mathrm{Top}}$, via intersection-over-union on relative span grids) and content ($\mathrm{GriTS}_{\mathrm{Con}}$, via longest common subsequence similarity), each yielding precision, recall, and F-score. By avoiding the tree representation, GriTS treats rows and columns symmetrically and is robust to markup variations, though content comparison remains string-based.网格表相似度（GriTS）。 GriTS[37] 通过直接在表格的二维网格表示上操作，使用动态规划进行行列对齐，解决了 TEDS 对 HTML 的敏感性。 它为结构拓扑（GriTS_Top）定义了基于相对跨度网格的交并比，为内容（GriTS_Con）定义了最长公共子序列相似度，两者均给出精确率、召回率和 F 分数。 通过避免树表示，GriTS 对行列对称处理，对标记差异更具鲁棒性，然而内容比较仍基于字符串。

SCORE. Although SCORE [16] presents itself as a “semantic evaluation framework” addressing the format rigidity of TEDS and GriTS, it normalizes tables into format-agnostic cell tuples and evaluates index accuracy by checking whether cells occupy correct grid positions and content accuracy via edit distance on cell text, with tolerance for small row/column offsets. This avoids penalizing markup differences across output formats, though the structural tolerance may also mask genuine errors, and the underlying cell comparison remains string-based, leaving true semantic equivalence (such as notational variants or equivalent value formats) unaddressed.SCORE。 虽然 SCORE[16] 自称是“语义评估框架”，旨在解决 TEDS 和 GriTS 的格式刚性问题，但它将表格归一化为格式无关的单元格元组，并通过检查单元格是否占据正确的网格位置来评估索引准确性，内容准确性通过单元格文本的编辑距离衡量，对小的行/列偏移有容忍度。 这种方式避免了对不同输出格式的标记差异惩罚，但结构容忍度也可能掩盖真实错误，且底层的单元格比较仍基于字符串，未能处理诸如记号变体或等价数值格式等真正的语义等价性。

Text-based metrics such as Levenshtein edit distance [13] or BLEU [29] operate at a strictly lower level of granularity: while the metrics above preserve cell-level structure, text-based approaches flatten the table into a token sequence, making scores dependent on serialization order and unable to distinguish structural from content errors.基于文本的指标如 Levenshtein 编辑距离[13]或 BLEU[29]在粒度上更低：上述指标保留了单元格层级结构，而文本指标将表格展平成 token 序列，导致分数受序列化顺序影响，且无法区分结构错误和内容错误。

While LLM-based evaluation has recently shown promise for formula extraction from PDFs [10], substantially outperforming text-based, tree-based, and image-based metrics in correlation with human judgment, no comparable study exists for table extraction.虽然基于 LLM 的评估已在 PDF 公式提取上显示出显著优势[10]，在与人工判断的相关性上远超文本、树和图像指标，但尚未有针对表格提取的可比研究。

Our work addresses both gaps: on the benchmark side, we use synthetic PDFs whose LaTeX source serves as exact ground truth, eliminating the need for manual annotation. On the metric side, we apply LLM-as-a-judge for semantic table evaluation. Together with a broad comparison of 21 contemporary parsers, this yields a reproducible evaluation framework that we validate against human judgment in Section 4.我们的工作同时填补了两方面的空白：在基准方面，我们使用合成 PDF，其 LaTeX 源码作为精确的真实值，免除人工标注；在指标方面，我们采用 LLM 作为评审者进行语义表格评估。 结合对 21 种当代解析器的广泛比较，这提供了一个可复现的评估框架，并在第 4 节通过人工判断进行验证。

We collect LaTeX table sources from arXiv papers published in December 2025 to avoid overlap with established datasets [46, 36] that may already be part of parser training data. All top-level tabular/tabular* environments are extracted, cleaned of non-content commands (citations, cross-references), and compiled standalone to verify validity and record rendered dimensions; invalid tables are discarded. Each valid table is classified by structural complexity using an LLM-based classifier: simple (regular grid), moderate (limited cell merging), or complex (multi-dimensional merging, nested structures).我们收集了 2025 年 12 月发布的 arXiv 论文中的 LaTeX 表格源码，以避免与已存在的数据集[46, 36]重叠，这些数据集可能已被解析器的训练数据使用。 提取所有顶层的 tabular/tabular* 环境，去除非内容命令（引用、交叉引用），并单独编译以验证有效性并记录渲染尺寸；无效表格被丢弃。 每个有效表格使用基于 LLM 的分类器进行结构复杂度分类：简单（规则网格）、中等（有限单元合并）或复杂（多维合并、嵌套结构）。

Each benchmark page is generated by sampling a random layout configuration (document class, font family, page margins, font size, line spacing, and single- or two-column layout) and iteratively appending content blocks, either filler text or tables from the extracted pool. The document is recompiled with pdflatex after each addition; blocks that trigger overflow or typesetting warnings are discarded, and the process terminates when no further content fits. Tables are pre-filtered by their recorded dimensions against the remaining page space and scaled to column width via adjustbox when moderately oversized. To ensure deterministic positioning, tables are placed as non-floating centered blocks, avoiding the unpredictable reordering of LaTeX float environments that would complicate ground truth alignment. Figure 1 gives an overview of the pipeline.每页基准通过随机抽取布局配置（文档类、字体族、页边距、字号、行距、单/双栏布局），并迭代追加内容块，块可以是填充文本或来自提取池的表格。 每次添加后使用 pdflatex 重新编译；触发溢出或排版警告的块被丢弃，过程在无法再容纳内容时终止。 表格在加入前根据记录的尺寸与剩余页面空间进行预筛选，若略大则通过 adjustbox 按列宽缩放。 为确保位置确定性，表格以非浮动的居中块方式放置，避免 LaTeX 浮动环境的不可预测重排，简化真实值对齐。 图 1 展示了流水线概览。

Before tables can be evaluated, each ground truth table must be matched to its counterpart in the parser output. This is non-trivial because parsers produce tables in diverse formats (HTML, Markdown, LaTeX, plain text), may split or merge tables, reorder content in multi-column layouts, or fail to recognize tables entirely.在评估表格之前，必须将每个真实表格与解析器输出中的对应表格匹配。 这并非易事，因为解析器产生的表格格式多样（HTML、Markdown、LaTeX、纯文本），可能会拆分或合并表格，在多栏布局中重新排序内容，或根本未识别表格。

We address this with an LLM-based matching pipeline using Gemini-3-Flash-Preview [8]. Given the list of ground truth tables (as LaTeX) and the full parser output, the model identifies and extracts the corresponding parsed representation for each ground truth table. Since the LLM may introduce minor artifacts such as whitespace changes, a rule-based post-validation step verifies and corrects each returned table against the original parser output, yielding a robust mapping between ground truth and parsed tables.我们使用 Gemini-3-Flash-Preview[8] 构建基于 LLM 的匹配流水线。 给定真实表格列表（LaTeX）和完整的解析器输出，模型识别并提取每个真实表格对应的解析表示。 由于 LLM 可能引入空格等细微差异，随后进行基于规则的后验证步骤，对每个返回的表格与原始解析器输出进行核对和纠正，从而得到稳健的真实表格与解析表格映射。

Since TEDS, GriTS, and SCORE compare structure and content purely syntactically, they cannot distinguish between discrepancies that alter the semantic content of a table and those that are merely representational. Figure 2 illustrates this with a constructed example highlighting discrepancy patterns we frequently observed in parser outputs. Several difference types are semantically insignificant yet incur large edit distances:由于 TEDS、GriTS 和 SCORE 纯粹在语法层面比较结构和内容，它们无法区分改变表格语义的差异和仅是表现形式的差异。 图 2 通过构造的示例展示了我们在解析器输出中经常观察到的差异模式。 以下几类差异在语义上并不重要，却会导致巨大的编辑距离：

• •

  Structural reorganization: parsers flatten multi-level headers or resolve rowspans—often because the output format lacks support for spanning cells (e.g., Markdown, unlike HTML, has no colspan/rowspan mechanism), leading to workarounds such as repeating values or inserting empty padding cells.结构重组：解析器将多层标题展平或解决行跨（rowspan），常因输出格式不支持跨单元格（如 Markdown 没有 colspan/rowspan）而采用重复值或插入空填充单元格的变通。

• •

  Symbol encoding: formulas appear as Unicode symbols instead of LaTeX commands (e.g., $\alpha$ instead of $\alpha$).符号编码：公式以 Unicode 符号而非 LaTeX 命令出现（例如 α\alpha 而非 $\alpha$）。

• •

  Value equivalence: “85.0%” vs. “85%” or “—” vs. “N/A”.数值等价：“85.0%” 与 “85%”，或 “—” 与 “N/A”。

• •

  Markup artifact: visual attributes are encoded as raw commands (e.g., \textbf{}).标记伪影：视觉属性被编码为原始命令（例如 \textbf{}）。

In contrast, the only semantically critical differences, content errors such as a lost decimal point (1.12 $\to$ 112) and a flipped sign (+2.8 $\to$ $-$2.8), change merely a single character each, contributing minimally to the edit distance. String-based metrics consequently assign low scores driven by harmless representational variation, while the few-character errors that fundamentally alter the table’s meaning are barely reflected.相反，唯一语义关键的差异——如丢失小数点（1.12→112）和符号翻转（+2.8→−-2.8）——每个仅改变一个字符，对编辑距离贡献极小。 基于字符串的指标因此因无害的表现形式差异而给出低分，而那些仅改变单字符却根本改变表格意义的错误几乎未被反映。

Building on the LLM-as-a-judge paradigm [44, 14], we propose using LLMs to assess table extraction quality semantically. Given a ground truth table and its parsed counterpart, an LLM evaluates the pair on a 0–10 scale for content accuracy and structural preservation, i.e., whether every cell value can be unambiguously mapped to its row and column headers. We evaluate four popular LLMs as judges: DeepSeek-v3.2 [21], GPT-5-mini [34], Gemini-3-Flash-Preview [8], and Claude Opus 4.6 [2], selected for their strong performance on public benchmarks across different price points.基于 LLM‑as‑a‑judge 范式[44, 14]，我们提出使用 LLM 对表格提取质量进行语义评估。 给定真实表格和其解析对应，LLM 对该对进行 0–10 分的内容准确性和结构保真度评分，即每个单元格值是否能明确映射到其行列标题。 我们评估了四种流行的 LLM 作为评审者：DeepSeek‑v3.2[21]、GPT‑5‑mini[34]、Gemini‑3‑Flash‑Preview[8] 和 Claude Opus 4.6[2]，它们在不同价位上均在公开基准上表现出色。

To validate automated metrics against human judgment, we collected over 1,500 human ratings covering 518 pairs of ground truth and parsed tables. Each pair was rated on a 0–10 scale reflecting whether the semantic content of the table—all values, headers, and their associations—has been correctly, completely, and unambiguously preserved. The pairs were sampled across all parsers whose outputs span diverse formats (HTML, Markdown, LaTeX, plain text) and table complexities, ensuring broad coverage. Since tables can be large and discrepancies subtle, we prompted Claude Opus 4.6 to pre-identify potential differences in each pair. Evaluators were then presented with a web interface showing both tables alongside these LLM-generated hints on potential discrepancies, ensuring that subtle issues are surfaced for human judgment while the final score remains entirely a human decision.为验证自动指标与人工判断的对应关系，我们收集了超过 1,500 条人工评分，覆盖 518 对真实表格和解析表格。 每对表格均在 0–10 分尺度上评分，反映表格的语义内容——所有数值、标题及其关联——是否被正确、完整且明确地保留。 这些对从所有解析器的输出中抽样，覆盖 HTML、Markdown、LaTeX、纯文本等多种格式以及不同表格复杂度，确保广泛覆盖。 由于表格可能较大且差异细微，我们让 Claude Opus 4.6 预先识别每对中的潜在差异。 评估者随后在网页界面中同时看到两张表格以及 LLM 生成的潜在差异提示，以确保细微问题被呈现，最终分数完全由人工决定。

Inter-annotator agreement. To assess the reliability of the human reference scores, we report agreement among the three independent evaluators who each rated all 518 pairs. Krippendorff’s $\alpha$ (interval) is 0.77, indicating acceptable agreement [12]. Average pairwise Pearson correlation between annotators is $r=0.85$, with individual pairs ranging from 0.81 to 0.91 and a mean absolute score difference of 1.2 on the 0–10 scale. As a human performance ceiling, the leave-one-out correlation of each annotator with the mean of the other two yields an average Pearson $r=0.89$.评审者间一致性。 为评估人工参考分数的可靠性，我们报告了对所有 518 对进行评分的三位独立评审者之间的一致性。 Krippendorff 的 α（区间）为 0.77，表明一致性可接受[12]。 评审者两两之间的 Pearson 相关系数平均为 r=0.85，单对范围 0.81–0.91，平均绝对分差为 1.2（0–10 量表）。 作为人工表现上限，每位评审者与另外两位的均值的留一交叉相关为平均 Pearson r=0.89。

We compute Pearson, Spearman, and Kendall correlations between each automated metric and the human reference scores to quantify how well each approach captures human notions of table extraction quality. All metrics are scaled to a 0–10 range for comparability; Table 1 summarizes the results and Figure 3 visualizes the relationship for a subset of metrics.我们计算了每种自动指标与人工参考分数之间的 Pearson、Spearman 和 Kendall 相关系数，以量化各方法捕捉人工对表格提取质量认知的程度。 所有指标均缩放至 0–10 范围以便比较；表 1 汇总了结果，图 3 展示了部分指标的散点图。

Rule-based metrics achieve only moderate correlation with human judgment. Since GriTS and SCORE each decompose into separate structure and content sub-metrics, unlike TEDS and LLM-based judges which assess both aspects jointly, we additionally compute their arithmetic means (GriTS-Avg, SCORE-Avg) to enable direct comparison. All rule-based metrics, whether structure-focused, content-focused, or averaged, fall within a narrow band of $r=0.56$–$0.70$ (Table 1), confirming the limitations analyzed in Section 4.1.基于规则的指标仅与人工判断呈中等相关性。 由于 GriTS 和 SCORE 各自分解为结构子指标和内容子指标，而 TEDS 与 LLM 评审者同时评估两者，我们额外计算了它们的算术平均（GriTS‑Avg、SCORE‑Avg）以实现直接比较。 所有基于规则的指标，无论是结构、内容还是平均，均落在 r=0.56–0.70 的窄区间（表 1），验证了第 4.1 节分析的局限性。

LLM-based evaluation substantially outperforms all rule-based metrics. Even the weakest LLM judge (DeepSeek-v3.2, $r=0.80$) exceeds the best rule-based metric (GriTS-Avg, $r=0.70$), confirming that semantic assessment captures dimensions of table quality that string-level comparison systematically misses. Claude Opus 4.6 achieves the highest correlation ($r=0.94$), though the human reference scores may be skewed toward its assessments since it also generated the error hints shown to evaluators (^† in Table 1). Gemini-3-Flash-Preview ($r=0.93$) and GPT-5-mini ($r=0.89$), which had no role in the annotation process, still far surpass all rule-based metrics, confirming that the LLM advantage is independent of this confound.基于 LLM 的评估显著优于所有基于规则的指标。 即使是最弱的 LLM 评审者（DeepSeek‑v3.2，r=0.80）也超过了最佳的基于规则指标（GriTS‑Avg，r=0.70），证明语义评估捕捉了表格质量的维度，而字符串层面的比较系统性地遗漏。 Claude Opus 4.6 达到最高相关性（r=0.94），但由于它也生成了提供给评审者的错误提示，人工参考分数可能对其有偏（表 1 中的 †）。 Gemini‑3‑Flash‑Preview（r=0.93）和 GPT‑5‑mini（r=0.89）未参与标注过程，仍远超所有基于规则的指标，确认 LLM 优势独立于此混杂因素。

For the parser benchmark in Section 5, we adopt Gemini-3-Flash-Preview as it offers near-ceiling correlation at substantially lower inference cost than Claude Opus 4.6.在第 5 节的解析器基准中，我们采用 Gemini‑3‑Flash‑Preview，因其在接近上限的相关性下，推理成本远低于 Claude Opus 4.6。