A user types one string. “What is the maximum coverage amount? Don’t confuse it with the deductible, they’re often listed together.” The parser turns it into a row of typed columns: a topic, an expected answer shape (an amount), a scope hint (this contract), a negative cue (not the deductible) routed to the generation brief, and a layout hint (often listed together) that retrieval can use. Each piece becomes its own column on question_df. This article walks the five field families one at a time, with the code that fills each one and the typed schema that holds it.用户输入一个字符串。“最高承保金额是多少？不要与免赔额混淆，它们通常列在一起。”解析器将其转换为一行带类型的列：主题、预期答案形状（金额）、范围提示（本合同）、负面提示（不是免赔额）路由到生成简报，以及布局提示（通常列在一起）供检索使用。每个部分成为question_df上的独立列。本文逐一介绍五个字段族，包括填充每个字段的代码和保存它的类型化模式。
1. The five field families the parser fills1. 解析器填充的五个字段族
A question is more than its words. It also tells you what shape the answer should take, where to look in the document, whether it’s compound or too vague to act on. The parser captures each of these and writes it as a column on question_df. Read the rest of this article as a menu of what’s available, not as a checklist.问题不仅仅是词语的组合。它还告诉你答案应该是什么形状、在文档的哪里查找、是否是复合问题或过于模糊而无法处理。解析器捕获这些信息并将其写入question_df的列中。将本文的其余部分视为可用选项的菜单，而不是检查清单。
The columns fall into two groups.这些列分为两组。
What the parser reads from the question itself.解析器从问题本身读取的内容。
- Keywords: Tokens to feed retrieval. Several sources combine: explicit (the user named them), direct (extracted from the question), LLM rewrites, an expert concept dictionary, and high-signal regex anchors like
L131-1.关键词：供检索使用的标记。多个来源组合：显式（用户指定的）、直接（从问题中提取的）、LLM重写、专家概念词典以及高信号正则表达式锚点（如L131-1）。 - Answer shape and answer type: Two orthogonal axes: the expected cardinality (
single,listing,table,tree,nested_json) and the value type (text,amount,date,iban,address, …).答案形状和答案类型：两个正交轴：预期基数（单个、列表、表格、树、嵌套JSON）和值类型（文本、金额、日期、IBAN、地址……）。 - Scope: Where in the document to look: a page, a chapter, a section, a layout (table / image), a date range, a jurisdiction.范围：在文档中查找的位置：页面、章节、节、布局（表格/图像）、日期范围、管辖区域。
- Decomposition: Sub-questions when the question is compound.分解：当问题是复合时的子问题。
- Clarification: A short follow-up question when the input is too vague to act on.澄清：当输入过于模糊而无法处理时，一个简短的后续问题。
What the parser then decides (using the document’s profile on top of the above).解析器随后决定的内容（基于上述字段和文档配置文件）。
- Dispatch: How much surrounding context to read and return, which chunk strategy to use, which model to call. All cascaded from the answer type, the matched concept, and the project’s defaults.调度：读取和返回多少上下文、使用哪种分块策略、调用哪个模型。所有这些都根据答案类型、匹配的概念和项目的默认设置级联决定。
- Activations: Which bricks to run (TOC navigation, embeddings, cross-references, …), downgraded by what the document supports.激活：运行哪些模块（目录导航、嵌入、交叉引用……），根据文档支持的内容降级。
Each category becomes one or more columns on question_df. Projects pick what they need, skip the rest, and add new columns as failure modes show up: a policy_number for an insurance broker, a patient_id for medical RAG, a regulation_year for legal. The sub-sections walk each one.每个类别成为question_df上的一列或多列。项目选择所需内容，跳过其余部分，并根据失败模式添加新列：保险经纪人的policy_number、医疗RAG的patient_id、法律的regulation_year。各小节逐一介绍每个类别。
1.1 Keywords1.1 关键词
Retrieval needs words to search the document with. The parser picks them out of the question and hands them over. The user’s wording almost never matches the document’s wording on the first try, so the parser collects from several sources at once.检索需要词语来搜索文档。解析器从问题中提取这些词语并交给检索。用户的措辞几乎从不会与文档的措辞首次匹配，因此解析器同时从多个来源收集。
Here is the minimal schema we’ll grow as the section goes:以下是我们将随着本节内容扩展的最小模式：
class ParsedQuestion(BaseModel):
original_question: str
keywords: list[str]For “What is the maximum coverage amount?”, the parser produces:对于“最高承保金额是多少？”，解析器生成：
ParsedQuestion(
original_question="What is the maximum coverage amount?",
keywords=["maximum", "coverage", "amount"],
)Keywords inherit whatever typos the question carries. Pull tokens straight from “How does multi-head atention compare to self-atention?” and retrieval searches for atention, a string the document never contains. Zero hits, the system returns nothing, the user concludes the topic isn’t covered. The fix is a cheap pre-step that runs before keyword extraction: one LLM call that corrects typos and grammar without changing meaning, so the keywords come out clean.关键词继承问题中的任何拼写错误。直接从“多头注意力与自注意力相比如何？”中提取标记，检索搜索的是atention，而文档中从未包含该字符串。零命中，系统返回空，用户得出结论该主题未被覆盖。解决方法是在关键词提取之前运行一个廉价的预处理步骤：一次LLM调用，纠正拼写和语法而不改变含义，从而使关键词干净。
def correct_spelling(question: str) -> str:
"""Fix typos and grammar without changing meaning."""
prompt = f"""Fix any typos and grammar mistakes in the question below.
Do not change the meaning. Do not add or remove information. Return only
the corrected question, nothing else.
Question: {question}"""
resp = client.responses.create(model="gpt-4.1-mini", input=prompt)
return resp.output_text.strip()In production, this is cached (the same question typed by multiple users gets corrected once) and skipped when the input is clean.在生产中，这会被缓存（多个用户输入的相同问题只纠正一次），并在输入干净时跳过。
Let users name the keywords themselves. Some users (analysts, paralegals, anyone fluent in the document’s vocabulary) already know exactly which terms they want matched. A UI hint, “List exact terms to search for, separated by commas”, opens the highest-precision retrieval path the system has. The user’s tokens go in verbatim: weight 1.0, source direct, no LLM, no synonym expansion (unless they opt in). For “Please find ‘force majeure’, ‘rescission’, ‘event of default’ in this contract”, the parser pulls the three quoted phrases as-is. Faster, cheaper, more accurate than any LLM rewrite when the user can name the terms. The product side matters too: a “search terms” field next to the question box, or a system prompt instruction (“include the exact terms you want matched”), moves a measurable share of queries onto this path.让用户自己命名关键词。一些用户（分析师、律师助理、任何熟悉文档词汇的人）已经确切知道他们想要匹配哪些术语。一个UI提示，“列出要搜索的精确术语，用逗号分隔”，开启了系统拥有的最高精度检索路径。用户的标记原样输入：权重1.0，来源直接，无LLM，无同义词扩展（除非他们选择加入）。对于“请在此合同中查找‘不可抗力’、‘解除’、‘违约事件’”，解析器按原样提取三个引用的短语。当用户能够命名术语时，这比任何LLM重写更快、更便宜、更准确。产品方面也很重要：问题框旁边的“搜索词”字段，或系统提示指令（“包括您想要匹配的精确术语”），将可衡量的查询份额转移到此路径上。
When the user doesn’t name terms explicitly, three parser-side sources fill in: LLM rewrites, an expert concept dictionary, and anchor regex.当用户没有显式命名术语时，三个解析器端来源填补：LLM重写、专家概念词典和锚点正则表达式。
Vocabulary mismatch is the first thing to break. The user asks about “the cap on what the insurer will pay” and the document says “limit of indemnity per occurrence.” The gap shows up everywhere in enterprise:词汇不匹配是首先会出问题的地方。用户询问“保险公司支付的限额”，而文档说“每次事故赔偿限额”。这种差距在企业中随处可见：
- Insurance: “the cap on what the insurer will pay” → “limit of indemnity per occurrence”.保险：“保险公司支付的限额”→“每次事故赔偿限额”。
- Legal: “what happens if we exit early” → “early termination provisions” or “rights of rescission”.法律：“如果我们提前退出会怎样”→“提前终止条款”或“解除权”。
- Finance: “how much we’ll get paid back” → “principal repayment schedule” or “redemption terms”.金融：“我们能收回多少钱”→“本金偿还计划”或“赎回条款”。
- Medical: “side effects” → “adverse events” or “contraindications”.医疗：“副作用”→“不良事件”或“禁忌症”。
The keyword column has the wrong tokens; the search misses everything. Three sources combine to fill it with terms the document uses.关键词列包含错误的标记；搜索错过一切。三个来源结合以填充文档使用的术语。
Source A: LLM rewrites: Reformulate the question into 3-5 phrasings that match how the document is likely to phrase the answer. The trick is to generate the language that surrounds the answer, not the answer itself. (This is the idea behind HyDE, Hypothetical Document Embeddings: generating a plausible passage, then embedding it, rather than embedding the question directly.)来源A：LLM重写：将问题改写为3-5种与文档可能表述答案的方式相匹配的措辞。诀窍是生成围绕答案的语言，而不是答案本身。（这是HyDE，假设文档嵌入的思想：生成一个合理的段落，然后嵌入它，而不是直接嵌入问题。）
# src/question/rewrite.py
def rewrite_query(question: str, domain_hint: str = "") -> list[str]:
"""Rewrite a user question into 3-5 queries phrased the way the
relevant passage is likely to appear in the document."""
prompt = f"""You are translating a user question into search queries that match
how the answer would be phrased in a {domain_hint or 'professional'} document.
Return 3 to 5 alternative phrasings. Use vocabulary the document is likely to use,
not the user's casual phrasing. Output one phrasing per line, no numbering.
User question: {question}"""
resp = client.responses.create(model="gpt-4.1-mini", input=prompt)
return [line.strip() for line in resp.output_text.splitlines() if line.strip()]For “what happens if we exit early?” with domain_hint="commercial contract", the LLM rewrites the query into the five phrasings the document is likely to use: early termination provisions, conditions for exiting the agreement before the end of the term, termination for convenience, exit fees and penalties for early termination, rights of rescission and notice requirements.对于“如果我们提前退出会怎样？”且domain_hint="商业合同"，LLM将查询重写为文档可能使用的五种措辞：提前终止条款、在期限结束前退出协议的条件、便利终止、提前终止的退出费用和罚金、解除权和通知要求。
In our deployments, a small domain_hint-driven rewrite consistently moves more on the per-failure-mode evaluation than picking the next embedding model up the leaderboard. Cheap to add, easy to roll back if it hurts.在我们的部署中，一个小的domain_hint驱动的重写在每个失败模式评估中始终比选择排行榜上的下一个嵌入模型带来更多提升。添加成本低，如果效果不好也容易回滚。
Source B: the expert dictionary, our first satellite tables. LLM rewrites handle standard vocabulary. They don’t handle “premium” → “prime” (French insurance), “cotisation” (mutual societies), or “DDPE” (a specific insurance product code), because the LLM has rarely seen these mappings in its training data. Domain experts know the meaningful synonyms in their field. They maintain them in two satellite tables that together form the project’s keyword dictionary. This is where the system amplifies the expert at scale: year after year, the synonyms they collect accumulate into a relational asset the project owns, can grow, and can audit at any time.来源B：专家词典，我们的第一个卫星表。LLM重写处理标准词汇。但它们不处理“premium”→“prime”（法语保险）、“cotisation”（互助会）或“DDPE”（特定保险产品代码），因为LLM在训练数据中很少看到这些映射。领域专家知道他们领域中有意义的同义词。他们在两个卫星表中维护这些同义词，这两个表共同构成项目的关键词词典。这是系统大规模放大专家能力的地方：年复一年，他们收集的同义词积累成一个项目拥有的关系资产，可以增长，并且可以随时审计。
The first holds the concepts themselves: one row per concept, with its definition and the document family it belongs to.第一个表保存概念本身：每个概念一行，包含其定义和所属的文档族。
The second holds the keyword variants: one row per (concept, language, keyword), joined to concepts_df on the concept column.第二个表保存关键词变体：每个（概念、语言、关键词）一行，通过概念列连接到concepts_df。
The language column makes the dictionary work on mixed-language corpora. Think of a French insurance group whose contracts arrive in French, English, sometimes Spanish. Without the language column the parser would pull the wrong variants. The keyword_priority separates strong matches (primary) from weaker ones (secondary); weight is the numeric companion used directly in the lexical score. Splitting concepts and keywords this way keeps each concept’s metadata (definition, document type) in one place instead of repeating it on every keyword row.语言列使词典能够在混合语言语料库上工作。想象一个法国保险集团，其合同以法语、英语，有时西班牙语到达。没有语言列，解析器会提取错误的变体。keyword_priority将强匹配（primary）与弱匹配（secondary）分开；weight是直接用于词汇分数的数值伴侣。这样拆分概念和关键词可以将每个概念的元数据（定义、文档类型）保存在一个地方，而不是在每个关键词行上重复。
How the parser uses the two tables: When a keyword in the question matches a row in concept_keywords_df, the parser looks up the concept and pulls every variant (all languages, all priorities). Retrieval then searches for all of them at once. If a keyword could fit several concepts (a prime could be an insurance premium or a bonus or a primary number), the parser asks the LLM to pick, passing the definition column from concepts_df for each candidate:解析器如何使用这两个表：当问题中的关键词匹配concept_keywords_df中的一行时，解析器查找概念并提取所有变体（所有语言、所有优先级）。然后检索同时搜索所有这些变体。如果一个关键词可能适合多个概念（prime可以是保险费、奖金或质数），解析器要求LLM选择，传递concepts_df中每个候选的定义列：
def disambiguate_concept(
question: str,
candidates: pd.DataFrame,
*,
system_prompt: str = (
"Pick the concept that best fits the user's question. "
"Reply with the concept name only, no explanation."
),
) -> str:
"""`candidates`: rows from concepts_df that share a matched keyword."""
options = "\n".join(
f"- {r['concept']}: {r['definition']}" for _, r in candidates.iterrows()
)
user_msg = f"Question: {question}\n\nCandidates:\n{options}"
resp = client.responses.create(
model="gpt-4.1-mini",
input=[{"role": "system", "content": system_prompt},
{"role": "user", "content": user_msg}],
)
return resp.output_text.strip()The LLM picks the one that fits the user’s question and the rest of the row resolves from there.LLM选择适合用户问题和行其余部分的概念。
The two tables grow with the project: every missed retrieval that traces back to a vocabulary mismatch becomes a new row.这两个表随着项目增长：每次因词汇不匹配导致的检索失败都会成为新的一行。
Embeddings can help discover candidates: embed every distinct noun phrase in the corpus, cluster (HDBSCAN, ~5-min cluster size), find clusters that overlap a known concept’s existing keywords, and let the expert decide which new variants to admit. What retrieval reads is the validated table, not the raw embeddings. The expert validates each entry before it lands in the dictionary. That’s what makes the system fit the domain reliably.嵌入可以帮助发现候选：嵌入语料库中每个不同的名词短语，聚类（HDBSCAN，约5分钟聚类大小），找到与已知概念现有关键词重叠的聚类，让专家决定接受哪些新变体。检索读取的是经过验证的表，而不是原始嵌入。专家在每条条目进入词典之前进行验证。这就是使系统可靠地适应领域的原因。
Standard RAG tutorials assume embedding similarity will handle synonyms automatically. For some domains it does. For specialized enterprise vocabulary it doesn’t.标准的RAG教程假设嵌入相似性会自动处理同义词。对于某些领域确实如此。但对于专业的企业词汇则不然。
Here is where the series’s editorial position on embeddings shows up. The usual path is to pick the “best” embedding model by measuring recall@k on a labeled query/document set, then lean on that model for synonym matching. We do the opposite: solve synonyms with the validated dictionary, keep embeddings as a fallback for cases the dictionary doesn’t cover yet, and use them upstream as a discovery tool rather than the primary retrieval signal. The retrieval brick details where exactly embeddings sit in the funnel.这就是本系列关于嵌入的编辑立场出现的地方。通常的路径是通过在标记的查询/文档集上测量recall@k来选择“最佳”嵌入模型，然后依赖该模型进行同义词匹配。我们反其道而行之：使用经过验证的词典解决同义词，将嵌入作为词典尚未覆盖的情况的备选方案，并将其用作上游发现工具，而不是主要检索信号。检索模块详细说明了嵌入在漏斗中的确切位置。
When the value set is closed, enumerate it. Some concepts have a finite, known list of values: country names, currency codes, US state names plus abbreviations, insurance product codes, drug names from the company formulary, vehicle makes and models the broker sells. For these, the dictionary stops growing organically and becomes a one-shot bulk insert: list every value, every common spelling, every translation, every abbreviation, every variant that turns up in real documents.当值集是封闭的时，枚举它。一些概念具有有限、已知的值列表：国家名称、货币代码、美国州名及缩写、保险产品代码、公司处方药名称、经纪人销售的车辆品牌和型号。对于这些，词典停止有机增长，成为一次性批量插入：列出每个值、每个常见拼写、每个翻译、每个缩写、每个在真实文档中出现的变体。
Take countries. If a user asks “What is the coverage in Germany?”, the document almost certainly contains Germany, Allemagne, Deutschland, DE, or DEU as a literal token. There is no pattern to detect, no regex that captures “country-ness” across the 195 of them. The fix is to load all of them (or the subset the corpus uses) into concept_keywords_df with concept = "country", one row per (language, spelling). Any match in the question tells the parser the user is asking about a country, and retrieval scopes accordingly.以国家为例。如果用户问“德国的承保范围是多少？”，文档几乎肯定包含Germany、Allemagne、Deutschland、DE或DEU作为字面标记。没有可检测的模式，没有正则表达式能捕获195个国家中的“国家性”。解决方法是将所有国家（或语料库使用的子集）加载到concept_keywords_df中，概念为"country"，每个（语言、拼写）一行。问题中的任何匹配告诉解析器用户正在询问一个国家，检索相应地进行范围限定。
The contrast with answer_types_df is sharp. Amounts, dates, IBANs, percentages all share structural patterns that regex catches. Country names share no structure. The two satellite tables solve two different kinds of problem: one for things with patterns, the other for closed sets you can list end to end.与answer_types_df的对比很明显。金额、日期、IBAN、百分比都有正则表达式可以捕获的结构模式。国家名称没有共同结构。这两个卫星表解决两种不同的问题：一种针对有模式的事物，另一种针对可以完整列出的封闭集。
Source C: anchor keywords: Sometimes the user gives you tokens that are too important to risk losing in the embedding average: internal product codes, regulatory references, clause numbers, identifiers.来源C：锚点关键词：有时用户给出的标记太重要，不能冒险在嵌入平均中丢失：内部产品代码、法规引用、条款编号、标识符。
“Does article L131-1 of the insurance code apply here?”“保险法典第L131-1条适用吗？”
The token “L131-1” is the entire query. If you embed the whole sentence, that token gets diluted with “article”, “insurance code”, “apply here”. Extract the high-signal tokens and route them to a lexical index: BM25, the classical keyword-scoring algorithm that weights rare terms more heavily: alongside the embedding query.标记“L131-1”就是整个查询。如果你嵌入整个句子，该标记会被“article”、“insurance code”、“apply here”稀释。提取高信号标记并将其路由到词汇索引：BM25，经典的词汇评分算法，更重视稀有术语：与嵌入查询一起。
# src/question/keywords.py
import re
ANCHOR_PATTERNS = [
r"\b[A-Z]+\d+(?:[-/]\d+)*\b", # L131-1, ISO-9001, RC-2024 (any number of leading caps)
r"\b[A-Z]{2,}\.[A-Z]{2,}(?:-\d+)?\b", # NIST-style codes: ID.AM, PR.AC-1
r"\b[A-Z]{3,}\b", # GDPR, RCP, SLA
r"\b\d{4,}\b", # year, identifier numbers
]
def extract_anchor_keywords(question: str) -> list[str]:
"""Extract high-signal tokens for lexical retrieval."""
found: list[str] = []
for pattern in ANCHOR_PATTERNS:
found.extend(re.findall(pattern, question))
return list(dict.fromkeys(found)) # de-dup, preserve orderThe three sources combine in the parsed output:三个来源在解析输出中结合：
class Keyword(BaseModel):
text: str
weight: float = 1.0
source: Literal["direct", "llm_expansion", "expert_dictionary", "anchor"]
semantic_group: str | None = None
is_regex: bool = False
class ParsedQuestion(BaseModel):
original_question: str
keywords: list[Keyword] # now structuredWhy HyDE works, and why explicit keywords capture the same gain. A diagnostic on a real example, in Article 2 (embeddings’ failure modes), section 3.2, showed the raw query “how do I cancel my policy” losing to a lexical decoy. The HyDE rewrite injected rescission, terminate, written notice, renewal, and the target won by a margin of 0.169. The mechanism behind that effect is exactly what the satellite tables above already do, without the embedding round-trip.为什么HyDE有效，以及显式关键词如何捕获相同的增益。在真实示例上的诊断，见第2篇文章（嵌入的失败模式）第3.2节，显示了原始查询“how do I cancel my policy”输给了一个词汇干扰项。HyDE重写注入了rescission、terminate、written notice、renewal，目标以0.169的差距获胜。背后的机制正是上述卫星表已经做到的，无需嵌入往返。
Three mechanisms run at the same time when HyDE works:当HyDE有效时，三个机制同时运行：
- Keyword and synonym expansion: The LLM, generating a hypothetical answer, naturally uses the domain vocabulary the question lacks. “How do I exit a contract early?” yields “early termination, notice period, exit fees, written notice.” These are the words real passages contain. The HyDE embedding captures them, retrieval finds the matches.关键词和同义词扩展：LLM在生成假设答案时，自然使用了问题所缺乏的领域词汇。“How do I exit a contract early?”产生“early termination, notice period, exit fees, written notice。”这些是真实段落包含的词语。HyDE嵌入捕获它们，检索找到匹配。
- Register matching: The hypothetical answer adopts the document’s register (formal, technical, domain-specific). The conversational register of the question is replaced with the register of the answer. The distance shrinks in the embedding space. Real, but smaller than mechanism 1 on enterprise corpora with a bounded vocabulary.语域匹配：假设答案采用文档的语域（正式、技术、领域特定）。问题的会话语域被答案的语域取代。嵌入空间中的距离缩小。真实，但在词汇有限的企业语料库上比机制1小。
- Latent semantic associations: The LLM activates trained associations that are not lexical. The most-cited reason in the literature for HyDE’s gain, but the smallest contributor in a bounded domain.潜在语义关联：LLM激活了非词汇的训练关联。文献中HyDE增益最常被引用的原因，但在有限领域中贡献最小。
In enterprise contexts (one bounded domain: insurance, legal, medical) mechanism 1 explains most of the HyDE gain. The keywords the LLM produces in the hypothetical answer are exactly the ones the document uses. The fictitious text and the document share the same vocabulary. The embedding step captures that shared vocabulary, nothing more.在企业环境中（一个有限领域：保险、法律、医疗），机制1解释了HyDE增益的大部分。LLM在假设答案中产生的关键词正是文档使用的关键词。虚构文本和文档共享相同的词汇。嵌入步骤捕获了共享词汇，仅此而已。
If mechanism 1 dominates, extracting the keywords directly captures the same benefit at lower cost. The round-trip embedding disappears. Retrieval becomes auditable (the expert sees which keywords matched). What the project builds is a permanent asset (the dictionary in concept_keywords_df) instead of regenerating the hypothesis at every query.如果机制1占主导，直接提取关键词以更低的成本捕获相同的收益。嵌入往返消失。检索变得可审计（专家看到哪些关键词匹配）。项目构建的是一个永久资产（concept_keywords_df中的词典），而不是在每个查询时重新生成假设。
That is why explicit keyword extraction wins over HyDE in the contexts the series targets: bounded domain, expert available, auditable retrieval required. In open-domain consumer search without those constraints, HyDE keeps an edge because mechanisms 2 and 3 weigh more.这就是为什么在系列目标环境中显式关键词提取胜过HyDE：有限领域、专家可用、需要可审计检索。在没有这些约束的开放领域消费者搜索中，HyDE保持优势，因为机制2和3权重更大。
In practice the comparison is simple. HyDE asks for a rewrite, embeds it, runs cosine, one round-trip per query, no audit trail. The approach above makes one LLM call to extract the keywords once, looks them up in the dictionary, and the audit trail is the matched rows themselves. The dictionary is reusable across queries and grows with the project.在实践中，比较很简单。HyDE要求重写、嵌入、运行余弦，每个查询一次往返，无审计跟踪。上述方法进行一次LLM调用提取关键词，在词典中查找，审计跟踪就是匹配的行本身。词典可跨查询重用，并随项目增长。
1.2 Tagging the answer shape and the answer type1.2 标记答案形状和答案类型
Even with rich keywords, retrieval can return passages that match on the words but contain no actual answer. “What is the annual premium?” might match a sentence about premium quality with no monetary amount in sight. Or it matches a passage about an unrelated prime (a bonus, a primary). The system has no way to tell the right hit from the wrong one because it doesn’t know what kind of answer it’s looking for.即使有丰富的关键词，检索也可能返回在词语上匹配但不包含实际答案的段落。“年保费是多少？”可能匹配关于优质品质的句子，而没有货币金额。或者匹配关于无关prime（奖金、质数）的段落。系统无法区分正确和错误的命中，因为它不知道它在寻找什么类型的答案。
The fix is to tag every question on two independent axes: answer_shape (how the answer is laid out: one value? a list? a table?) and answer_type (what each value contains: text? amount? date?). “What is the annual premium?” is (single, amount). “List the annual premiums by year” is (listing, amount). “List the exclusions of the contract” is (listing, text). The same type can travel under any shape, and the same shape can carry any type. Keeping the two separate lets the parser tag each one on its own.解决方法是在两个独立轴上标记每个问题：answer_shape（答案如何布局：一个值？列表？表格？）和answer_type（每个值包含什么：文本？金额？日期？）。“年保费是多少？”是（单个，金额）。“按年份列出年保费”是（列表，金额）。“列出合同的除外责任”是（列表，文本）。相同的类型可以在任何形状下使用，相同的形状可以携带任何类型。保持两者分离让解析器独立标记每个轴。
Take amounts: A question of type amount triggers a regex pass alongside the keyword search, scanning for numeric tokens with currency symbols (\d[\d\s.,]*\s*(?:EUR|€|USD|\$)). “What is the annual premium?” now matches the line “Prime annuelle: 125 000 €” on two signals at once: the keyword prime is there AND the amount regex matches 125 000 € on the same line. Two-signal match is much stronger than keyword alone. Conversely, if retrieval finds keyword matches but no monetary amount anywhere in the candidate passages, the answer is probably not in the document; the system can return “no amount found” with confidence rather than guessing.以金额为例：类型为金额的问题触发正则表达式传递以及关键词搜索，扫描带有货币符号的数字标记（\d[\d\s.,]*\s*(?:EUR|€|USD|\$)）。“年保费是多少？”现在同时匹配行“Prime annuelle: 125 000 €”的两个信号：关键词prime存在，并且金额正则表达式在同一行匹配125 000 €。双信号匹配比单独关键词强得多。相反，如果检索找到关键词匹配但在候选段落中没有任何货币金额，答案可能不在文档中；系统可以自信地返回“未找到金额”，而不是猜测。
Same logic for dates: “When does coverage start?” expects a date. Retrieval scans for date patterns (ISO, locale, written-out) alongside keywords like date d’effet, commencement, start. If the keyword zone contains a clean parseable date, that’s almost certainly the answer; if not, the field is missing from the contract.日期同理：“承保何时开始？”期望一个日期。检索扫描日期模式（ISO、区域设置、书面形式）以及关键词如date d'effet、commencement、start。如果关键词区域包含一个干净可解析的日期，那几乎肯定是答案；如果没有，则该字段在合同中缺失。
The type axis is open: Anything you can name and write a regex (or LLM check) for, you can register: text, amount, date, boolean, email, iban, policy_number, siren, percentage, duration, address, … The registry lives in answer_types_df, one row per registered type:类型轴是开放的：任何你可以命名并编写正则表达式（或LLM检查）的内容，都可以注册：文本、金额、日期、布尔、电子邮件、IBAN、保单号、SIREN、百分比、持续时间、地址……注册表存在于answer_types_df中，每个注册类型一行：
The retrieval_patterns column (kept in the live DataFrame, omitted from the image for width) is what retrieval uses to confirm the type. The output_schema_ref column points to the Pydantic class generation renders into; the generation brick owns that side. The default_model column is the model the parser falls back to when no concept-level override applies: small types (amount, date, iban) land on a nano model, free-form text on mini. Adding a new type to the project is a single insert. Anything the parser can’t classify falls back to text and skips the regex confirmation.retrieval_patterns列（保存在实时DataFrame中，因宽度在图片中省略）是检索用来确认类型的。output_schema_ref列指向生成渲染成的Pydantic类；生成模块拥有那一侧。default_model列是解析器在没有概念级覆盖时回退的模型：小类型（金额、日期、IBAN）落在nano模型上，自由文本落在mini上。向项目添加新类型是一个单次插入。解析器无法分类的任何内容回退到文本并跳过正则表达式确认。
The shape axis is closed and tiny: Five values cover what we’ve seen across real corpora: single (one value, the default), listing (a flat enumeration), table (rows × columns), tree (nested hierarchy), nested_json (a structured object with named sub-fields, e.g. an address as {street, city, zip}). The registry is so small it lives in a sibling satellite with two columns of defaults:形状轴是封闭且很小的：五个值涵盖了我们在真实语料库中看到的情况：single（一个值，默认）、listing（平面枚举）、table（行×列）、tree（嵌套层次）、nested_json（具有命名子字段的结构化对象，例如地址为{street, city, zip}）。注册表非常小，它存在于一个具有两列默认值的兄弟卫星表中：
Single facts almost always live on one line in the top-ranked chunk, so sequential saves ⅔ of the tokens at k=3; listings, tables and trees need to be synthesised across passages, so combined is the safer default. The split is what lets “What is the annual premium?” (a (single, amount) question) and “List the annual premiums by year” (a (listing, amount) question) share the same type (amount, same regex, same value parsing) while routing differently at generation time.单个事实几乎总是位于排名最高的块中的一行上，因此sequential在k=3时节省了2/3的标记；列表、表格和树需要跨段落合成，因此combined是更安全的默认值。这种分割使得“年保费是多少？”（一个（单个，金额）问题）和“按年份列出年保费”（一个（列表，金额）问题）共享相同的类型（金额，相同的正则表达式，相同的值解析），同时在生成时路由不同。
class ParsedQuestion(BaseModel):
original_question: str
keywords: list[Keyword]
answer_shape: Literal["single", "listing", "table", "tree", "nested_json"] = "single"
answer_type: str = "text" # FK into answer_types_dfThe label is a property of the question, not of the document. “What is the premium?” is a (single, amount) question whether the contract is two pages or two hundred. The two fields are independent: “List the exclusions” is (listing, text), “List the annual premiums of the contract” is (listing, amount).标签是问题的属性，而不是文档的属性。“保费是多少？”是一个（单个，金额）问题，无论合同是两页还是两百页。这两个字段是独立的：“列出除外责任”是（列表，文本），“列出合同的年保费”是（列表，金额）。
Shape is a closed enum (five values, fixed for the project); type is open (one row per registered type in answer_types_df). The classification itself is folded into the consolidated parse_question call covered in Article 6_c (dispatch), with both registries injected into the LLM prompt so adding a new type or a new shape is a row insert, not a code change.形状是一个封闭枚举（五个值，项目固定）；类型是开放的（answer_types_df中每个注册类型一行）。分类本身被折叠到第6c篇文章（调度）中涵盖的合并parse_question调用中，两个注册表都注入到LLM提示中，因此添加新类型或新形状是行插入，而不是代码更改。
1.3 Scope: where to look in the document1.3 范围：在文档中查找的位置
A question often names where in the document to look. The parser captures these hints in two typed fields, both applied before keyword retrieval runs. StructuralHints holds the structural hints (page, TOC section, layout). ScopeFilters holds the corpus-level filters: the application layer can pass them in (when it knows the user’s jurisdiction, date range, etc.), or the parser can pull them out of the question when it names one explicitly.问题通常指定在文档中查找的位置。解析器在两个类型化字段中捕获这些提示，两者都在关键词检索运行之前应用。StructuralHints保存结构提示（页面、目录节、布局）。ScopeFilters保存语料库级过滤器：应用层可以传入它们（当它知道用户的管辖区域、日期范围等时），或者解析器可以在问题显式命名时从问题中提取它们。
- Pages: “Show me page 3”, “Summarize pages 5 to 7”, “Compare page 2 and page 9”. Single page, range, and explicit list all collapse to a flat list of integers.
pages_hint = [3],pages_hint = [5, 6, 7],pages_hint = [2, 9]. Retrieval then filters with one expression (page_df[page_df.page_num.isin(pages_hint)]) and does not branch on the shape of the hint. Hinted pages are kept even when no keyword matches them: the user pinned them explicitly, that is the answer surface.页面：“显示第3页”、“总结第5到7页”、“比较第2页和第9页”。单页、范围和显式列表都折叠为整数平面列表。pages_hint = [3]，pages_hint = [5, 6, 7]，pages_hint = [2, 9]。然后检索使用一个表达式（page_df[page_df.page_num.isin(pages_hint)]）进行过滤，并且不根据提示的形状分支。即使没有关键词匹配，提示的页面也会保留：用户显式固定了它们，那就是答案表面。 - Chapter / section: “What are the exclusions of this contract?”. Names a TOC entry.
toc_section_hint = "Exclusions". Retrieval matches against the document’s actual TOC.章节/节：“本合同的除外责任是什么？”命名一个目录条目。toc_section_hint = "Exclusions"。检索匹配文档的实际目录。 - Layout: “the schedule table at the end”. The answer lives in a table, an image, or a header.
layout_hint = "table". Tells retrieval to look at structured zones, not narrative text.布局：“末尾的附表”。答案位于表格、图像或标题中。layout_hint = "table"。告诉检索查看结构化区域，而不是叙述文本。 - Date range / parties / jurisdiction: “What did we sign with Acme between 2022 and 2024?”. Corpus-level filters that trim candidate documents before retrieval runs (handled by a corpus-level index ahead of the document-scoped pipeline).
ScopeFilters(date_range=..., parties=["Acme"]).日期范围/当事方/管辖区域：“我们在2022年至2024年间与Acme签署了什么？”在检索运行之前修剪候选文档的语料库级过滤器（由文档范围管道之前的语料库级索引处理）。ScopeFilters(date_range=..., parties=["Acme"])。
The same convention extends to other formats. sheets_hint: list[str] carries Excel sheet names pinned in the question (“on the Pricing sheet”); slides_hint: list[int] carries PowerPoint slide numbers (“on slide 4”, “slides 7 to 9”). Volume 2 picks up both. The unifying idea is that the user phrasing controls the scope. A consequence worth noting: on short documents (CV, single-page invoice, 1-2 page memo), the operator pins the only page (page 1) inside the question and the pipeline runs unchanged. No “short doc” mode, no chunk-strategy switch, no retrieval bypass. The same code path that handles a 1000-page corpus query happens to scope to a one-page document. Article 8 (generation) develops this convention from the dispatcher’s side.相同的约定扩展到其他格式。sheets_hint: list[str]携带问题中固定的Excel工作表名称（“在定价表上”）；slides_hint: list[int]携带PowerPoint幻灯片编号（“在第4张幻灯片上”、“第7到9张幻灯片”）。第2卷涵盖两者。统一思想是用户措辞控制范围。一个值得注意的后果：在短文档（简历、单页发票、1-2页备忘录）上，操作员在问题中固定唯一页面（第1页），管道不变运行。没有“短文档”模式，没有分块策略切换，没有检索旁路。处理1000页语料库查询的相同代码路径恰好范围限定到一页文档。第8篇文章（生成）从调度器端发展了这个约定。
class ScopeFilters(BaseModel):
sections: list[str] = Field(default_factory=list)
date_range: tuple[str, str] | None = None
parties: list[str] = Field(default_factory=list)
jurisdictions: list[str] = Field(default_factory=list)
page_range: tuple[int, int] | None = None
custom: dict = Field(default_factory=dict)
class StructuralHints(BaseModel):
# WHERE the answer lives
toc_section_hint: str | None = None
# The likely TOC section or chapter ("Exclusions", "Schedule A").
# Retrieval matches against the document's actual TOC.
pages_hint: list[int] | None = None
# Pages pinned by the question. Single ("page 3" -> [3]),
# range ("pages 5 to 7" -> [5, 6, 7]), or list ("page 2 and 9"
# -> [2, 9]) all collapse to a flat list at parse time.
sheets_hint: list[str] | None = None # XLSX, Volume 2
slides_hint: list[int] | None = None # PPTX, Volume 2
layout_hint: Literal["text", "table", "image", "header"] | None = None
document_version: str | None = None
# HOW MUCH context to read and return (retrieval consumes these)
detection_context: Literal["line", "sentence", "paragraph"] = "line"
# Granularity of the regex confirmation zone (section 2.2).
# "line" for amount/date, "paragraph" for narrative.
answer_context: Literal["line", "paragraph", "page", "section", "chapter", "document"] = "paragraph"
# How much surrounding text the generator receives.
needs_summary: bool = False
# True when the answer spans more than fits in a verbatim quote.chunk_strategy and suggested_model are also per-question dispatch decisions, but they’re not structural (they don’t describe the document, they describe how the pipeline calls the LLM). They live at the top level of ParsedQuestion, not on StructuralHints, and the dispatch companion (Article 6_c) walks the cascade that fills them. The same is true for three fields kept on StructuralHints (detection_context, answer_context, needs_summary), which describe how much text the regex pass and the generator read, not where the answer lives. Article 6_c covers their defaults too.chunk_strategy和suggested_model也是每个问题的调度决策，但它们不是结构性的（它们不描述文档，它们描述管道如何调用LLM）。它们位于ParsedQuestion的顶层，而不是StructuralHints上，调度伴侣（第6c篇）介绍了填充它们的级联。同样适用于StructuralHints上的三个字段（detection_context、answer_context、needs_summary），它们描述正则表达式传递和生成器读取多少文本，而不是答案所在位置。第6c篇也涵盖了它们的默认值。
A few examples of what the parser produces:解析器生成的一些示例：
Detection is one LLM call with structured Pydantic output. Regex was tempting (catch “page 3” with r"page\s+(\d+)") and works for a handful of trivial cases, but breaks on the rest: “in the warranty section” (modifier before noun), “the recap table at the end” (no number), “the chapter on liability” (synonym), “the appendix” (no keyword). The LLM covers all of them in one round-trip, returns clean Pydantic, and stays maintainable.检测是一次具有结构化Pydantic输出的LLM调用。正则表达式很诱人（用r"page\s+(\d+)"捕获“page 3”），并且对少数简单情况有效，但在其余情况下失败：“在保修部分”（修饰语在名词前）、“末尾的摘要表”（无数字）、“关于责任的章节”（同义词）、“附录”（无关键词）。LLM在一次往返中覆盖所有情况，返回干净的Pydantic，并保持可维护性。
# src/question/hints.py
HINTS_PROMPT = (
"Read the user's question and extract structural hints about WHERE the answer "
"lives in the document AND HOW MUCH context the answer needs.\n\n"
"- toc_section_hint: the section or chapter the user pointed at, matched against "
"typical document TOC entries (e.g. 'Exclusions', 'Limits', 'Schedule A'). null if "
"no section is implied.\n"
"- pages_hint: flat list of page numbers the user pinned. Single ('page 3' -> [3]), range ('pages 5 to 7' -> [5, 6, 7]) and list ('page 2 and 9' -> [2, 9]) all collapse to a list. null otherwise.\n"
"- layout_hint: 'table' / 'image' / 'header' if the question implies a layout.\n"
"- detection_context: granularity of the regex confirmation zone. 'line' for a "
"single fact, 'sentence' for short prose, 'paragraph' for narrative.\n"
"- answer_context: how much surrounding text the generator receives. 'line' for "
"a single value, 'paragraph' for an explanation, 'page' for a recap, 'section' "
"for a topic, 'chapter' or 'document' for a broad summary.\n"
"- needs_summary: True if the answer spans more than fits in a verbatim quote."
)
def extract_hints(question: str, *, system_prompt: str = HINTS_PROMPT) -> StructuralHints:
resp = client.responses.parse(
model="gpt-4.1-mini",
input=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": question},
],
text_format=StructuralHints,
)
return StructuralHints.model_validate_json(resp.output_text)Layout hints matter more than they look. If the user says “it’s usually in an image”, that’s a big clue. Most parsers strip images or replace them with placeholders. Knowing the answer lives in an image tells you to look at the OCR output of figures, or to flag the question for vision-language processing. Without the hint, the pipeline searches text-only and finds nothing.布局提示比看起来更重要。如果用户说“它通常在图像中”，那是一个重要线索。大多数解析器剥离图像或用占位符替换。知道答案存在于图像中告诉你查看图形的OCR输出，或标记问题用于视觉语言处理。没有提示，管道仅搜索文本，找不到任何东西。
One LLM call per concern, or one LLM call total? The article shows each concern separately so you can understand (and test) each piece on its own. That’s also how you’d build the pipeline: one helper at a time, validating each column before adding the next. In production, once you’re confident the schema is right, you fold everything into one consolidated call: one round-trip, one prompt, one place where the LLM has full context. Article 6_c (dispatch), section 3.1, shows that consolidated parse_question end to end.每个关注点一次LLM调用，还是一次LLM调用总计？本文分别展示每个关注点，以便你可以独立理解（和测试）每个部分。这也是构建管道的方式：一次一个辅助函数，在添加下一个之前验证每个列。在生产中，一旦你确信模式正确，你将所有内容折叠到一个合并调用中：一次往返，一个提示，LLM拥有完整上下文的一个地方。第6c篇文章（调度）第3.1节展示了端到端的合并parse_question。
At single-document scale, scope filters constrain where in the document retrieval looks. At corpus scale (Part IV), they become SQL clauses on the corpus index. Same idea, different machinery.在单文档规模上，范围过滤器约束检索在文档中查找的位置。在语料库规模上（第四部分），它们成为语料库索引上的SQL子句。相同的想法，不同的机制。
1.4 Compound questions1.4 复合问题
Some questions can’t be answered by retrieving any single passage, no matter how well you phrase the query. They contain multiple sub-questions packed into one.有些问题无法通过检索任何单个段落来回答，无论你如何精心措辞查询。它们包含多个子问题打包在一起。
“Are the indemnification and liability caps consistent in this contract?”“本合同中的赔偿和责任上限是否一致？”
A comparison. Retrieve the indemnification clause, retrieve the liability cap clause, then compare them.一个比较。检索赔偿条款，检索责任上限条款，然后比较它们。
“Does this contract include a non-compete clause, and if so, for how long?”“本合同是否包含竞业禁止条款，如果有，期限多长？”
A conditional question. Step one: is there a non-compete? Step two (only if yes): what’s the duration?一个条件问题。第一步：是否有竞业禁止？第二步（仅当是）：期限多长？
“What is the annual premium and what are the main exclusions?”“年保费是多少，主要除外责任是什么？”
Two unrelated facts joined by “and”. Different passages, independent answers.两个由“和”连接的不相关事实。不同的段落，独立的答案。
Four patterns come up often enough to name.四种模式经常出现，值得命名。
Independent: Two unrelated facts joined by “and”: “What is the premium and what are the exclusions?” The orchestrator runs pdf_qa twice in parallel; merging is just {"sub_questions": [{"q": ..., "answer": ...}, ...]} keyed by the parsed sub-question.独立：两个由“和”连接的不相关事实：“保费是多少，除外责任是什么？”编排器并行运行两次pdf_qa；合并只是{"sub_questions": [{"q": ..., "answer": ...}, ...]}，由解析的子问题键控。
Sequential: The second part depends on the first: “Who is the insured party, and what is their address?” The address is of the insured party: you have to identify the party first, then look up their address. The orchestrator runs the sub-questions in order and substitutes the previous answer into the next sub-question’s keywords.顺序：第二部分依赖于第一部分：“被保险人是谁，他们的地址是什么？”地址是被保险人的：你必须先识别当事方，然后查找他们的地址。编排器按顺序运行子问题，并将前一个答案替换到下一个子问题的关键词中。
Unified: Two terms that refer to the same concept, not two questions: “What are the exclusions and limitations?” In most policy documents, exclusions and limitations appear together in the same section. Decomposing this into two sub-questions duplicates work. Keep it as one question with both terms boosted in keyword retrieval.统一：指代同一概念的两个术语，而不是两个问题：“除外责任和限制是什么？”在大多数保单文件中，除外责任和限制出现在同一部分。将其分解为两个子问题会重复工作。保持为一个问题，在关键词检索中提升两个术语。
Conditional: A condition narrows the scope: “If the policy is for commercial property, what is the fire coverage limit?” The condition becomes a scope filter; the actual question is “what is the fire coverage limit”, run on the subset of the document that matches.条件：条件缩小范围：“如果保单是针对商业财产，火灾承保限额是多少？”条件成为范围过滤器；实际问题是“火灾承保限额是多少”，在文档的匹配子集上运行。
A cheap rule of thumb for picking the pattern is the “and” test: replace “and” with “; also”. If it still reads naturally, the parts are independent. If it reads awkwardly, they’re unified. For the harder cases, an LLM classifies. has_compound_indicators is the cheap pre-filter (a regex looking for \band\b, \bor\b, ?...?, multiple imperatives); llm_classify_decomposition is one structured-output call returning a Decomposition:选择模式的一个廉价经验法则是“and”测试：将“and”替换为“; also”。如果仍然读起来自然，各部分独立。如果读起来别扭，它们是统一的。对于更困难的情况，LLM进行分类。has_compound_indicators是廉价的预过滤器（一个查找\band\b、\bor\b、?...?、多个祈使句的正则表达式）；llm_classify_decomposition是一次结构化输出调用，返回一个Decomposition：
class Decomposition(BaseModel):
pattern: Literal['single', 'independent', 'sequential',
'unified', 'conditional'] = 'single'
sub_questions: list[str] = Field(default_factory=list)
conditional_filter: dict | None = None
def decompose(question: str) -> Decomposition:
if not has_compound_indicators(question):
return Decomposition(pattern='single')
return llm_classify_decomposition(question)Decomposition adds latency and cost. Detect compound structure first; only decompose when the pattern warrants it.分解增加了延迟和成本。首先检测复合结构；仅在模式需要时才分解。
In one production deployment, ~30% of user questions in the first month of beta were compound, and the pipeline was returning incomplete answers on most of them. Adding compound decomposition at the parsing layer raised user satisfaction sharply with no other change in the pipeline.在一个生产部署中，测试第一个月约30%的用户问题是复合的，管道在大多数问题上返回不完整的答案。在解析层添加复合分解显著提高了用户满意度，而管道没有其他变化。
1.5 Clarification: when the system asks back1.5 澄清：当系统反问时
Some questions are too vague to act on at all, and no amount of parsing makes them actionable.有些问题过于模糊，根本无法处理，再多的解析也无法使其可操作。
“What’s the cap?” Cap on what? Liability? Indemnification? Damages? Premium? “Show me the latest version.” Latest version of what document? Latest as of when? “Compare with last year’s.” Last year’s what?“上限是多少？”什么的上限？责任？赔偿？损害赔偿？保费？“显示最新版本。”什么文档的最新版本？截至何时的最新版本？“与去年的比较。”去年的什么？
If the system charges ahead and guesses, it produces subtly wrong answers users either don’t catch (worse) or do catch and stop trusting the system (also worse). The fix is the cheapest possible: detect that the question can’t be acted on, and ask the user back instead of running the pipeline.如果系统贸然猜测，会产生微妙的错误答案，用户要么没有发现（更糟），要么发现并停止信任系统（同样更糟）。解决方法是尽可能廉价：检测问题无法处理，并反问用户，而不是运行管道。
The parsed question carries this case in two fields:解析的问题在两个字段中携带这种情况：
class ParsedQuestion(BaseModel):
# ... other fields
suggested_clarification: str | None = None
ambiguity_reason: str | None = NoneWhen suggested_clarification is set, the orchestrator returns it before running the pipeline:当设置了suggested_clarification时，编排器在运行管道之前返回它：
“Several aspects of this question are ambiguous. Could you specify: which limit (coverage, deductible, sublimit), and which policy if you have multiple?”“这个问题的几个方面不明确。您能说明：哪个限额（承保范围、免赔额、子限额），以及如果您有多个保单，是哪个保单？”
A simple rule: if the question uses a word that points back to something else (this, that, the latest, last year’s, the cap) and the context isn’t available from conversation history or scope, ask. Detection runs inside the consolidated parse_question call covered in the dispatch companion (Article 6_c), as one sub-task of the shared parse prompt: “return a short follow-up question if the input is too vague, null otherwise”. The interface should make answering cheap: a short follow-up, two or three suggested options, done.一个简单的规则：如果问题使用指向其他事物的词（this、that、the latest、last year's、the cap），并且上下文无法从对话历史或范围中获得，则询问。检测在调度伴侣（第6c篇）中涵盖的合并parse_question调用内运行，作为共享解析提示的一个子任务：“如果输入过于模糊，返回一个简短的后续问题，否则返回null”。界面应使回答廉价：一个简短的后续问题，两三个建议选项，完成。
Few production systems do this. The default tends to be: charge ahead and answer something. The result is a steady drip of subtly wrong answers, and users who slowly lose trust in the system. Catching ambiguity at parsing time is cheaper than catching it after retrieval has returned the wrong passages.很少有生产系统这样做。默认趋势是：贸然前进并回答一些东西。结果是持续不断的微妙错误答案，以及逐渐失去对系统信任的用户。在解析时捕获歧义比在检索返回错误段落之后捕获更便宜。
2. Conclusion2. 结论
Question parsing turns one noisy user string into a typed, relational brief. Five families of columns on question_df carry what the parser reads straight from the user’s question:问题解析将一个嘈杂的用户字符串转换为类型化的关系简报。question_df上的五个字段族携带解析器直接从用户问题中读取的内容：
- Keywords (with expert-dictionary expansion) anchor the retrieval search.关键词（带有专家词典扩展）锚定检索搜索。
- Answer shape and answer type tell generation what schema to return.答案形状和答案类型告诉生成返回什么模式。
- Scope hints filter where in the document to look.范围提示过滤在文档中查找的位置。
- Compound decomposition breaks the question into sub-questions when needed.复合分解在需要时将问题分解为子问题。
- Clarification asks the user back when the question is too ambiguous.澄清在问题过于模糊时反问用户。
Each column is a place where the project’s expert vocabulary, the document’s structure, and the user’s intent meet. Adding a parsing capability means adding a column, not a new function.每个列都是项目专家词汇、文档结构和用户意图相遇的地方。添加解析能力意味着添加一列，而不是一个新函数。
Two more families sit on top of these, decided by the parser after it sees the document profile: the dispatch decisions (chunk strategy, model, answer-context window) and the activation flags that turn bricks off when they do not fit. Both belong to the dispatch companion (Article 6_c), which also covers the routing of each column to the brick that consumes it.另外两个族位于这些之上，由解析器在看到文档配置文件后决定：调度决策（分块策略、模型、答案上下文窗口）和当模块不适合时关闭它们的激活标志。两者都属于调度伴侣（第6c篇），该篇还涵盖了每个列到消费它的模块的路由。
Sources and further reading来源和进一步阅读
The article reframes the HyDE technique from Gao et al. (HyDE, ACL 2023) as offline keyword work: the load-bearing piece is the keywords the hypothetical answer contains, not the embedding step itself. The query-rewriting line (Ma et al., RRR, EMNLP 2023) is the closest published precedent for the corrected_question + rewrites part of the structured plan. The four compound-question patterns map onto Self-Ask (Press et al., Self-Ask, EMNLP Findings 2023) plus IRCoT (Trivedi et al., IRCoT, ACL 2023). The closest lineage to “question becomes a typed object that downstream code consumes” is text-to-SQL semantic parsing (Yu et al., Spider, EMNLP 2018). Volume 3 (Agentic Bricks) returns to runtime tool-picking on top of the structured plan defined here.文章将Gao等人（HyDE, ACL 2023）的HyDE技术重新定义为离线关键词工作：承载关键的是假设答案中包含的关键词，而非嵌入步骤本身。查询重写路线（Ma等人，RRR, EMNLP 2023）是结构化计划中corrected_question + rewrites部分最接近的已发表先例。四种复合问题模式映射到Self-Ask（Press等人，Self-Ask, EMNLP Findings 2023）和IRCoT（Trivedi等人，IRCoT, ACL 2023）。与“问题成为下游代码消费的类型化对象”最接近的谱系是文本到SQL语义解析（Yu等人，Spider, EMNLP 2018）。第3卷（智能体积木）回到在此定义的结构化计划之上进行运行时工具选择。
Same direction as the article:与文章方向相同：
- Gao, Ma, Lin, Callan, Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE), ACL 2023 (arXiv:2212.10496). The HyDE technique the article reframes: offline keyword work, not online hypothetical-document embedding.Gao, Ma, Lin, Callan, Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE), ACL 2023 (arXiv:2212.10496)。文章重新定义的HyDE技术：离线关键词工作，而非在线假设文档嵌入。
- Ma, Gong, He, Zhao, Duan, Query Rewriting for Retrieval-Augmented Large Language Models (RRR), EMNLP 2023 (arXiv:2305.14283). Closest published precedent for the
corrected_question + rewritespart of the structured plan.Ma, Gong, He, Zhao, Duan, Query Rewriting for Retrieval-Augmented Large Language Models (RRR), EMNLP 2023 (arXiv:2305.14283)。结构化计划中corrected_question + rewrites部分最接近的已发表先例。 - Press et al., Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask), EMNLP Findings 2023 (arXiv:2210.03350). Compound-question decomposition pattern that the four patterns in this article extend.Press等人, Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask), EMNLP Findings 2023 (arXiv:2210.03350)。本文四种模式扩展的复合问题分解模式。
- Trivedi et al., Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT), ACL 2023 (arXiv:2212.10509). Sequential sub-questions; complements Self-Ask for compound queries.Trivedi等人, Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT), ACL 2023 (arXiv:2212.10509)。顺序子问题；补充Self-Ask处理复合查询。
- Yu et al., Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task, EMNLP 2018 (arXiv:1809.08887). Canonical text-to-SQL benchmark; the closest lineage to “question becomes a typed object that downstream code consumes”.Yu等人, Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task, EMNLP 2018 (arXiv:1809.08887)。经典文本到SQL基准；与“问题成为下游代码消费的类型化对象”最接近的谱系。
Different angle, different context:不同角度，不同背景：
- Wang et al., Query2doc: Query Expansion with Large Language Models, EMNLP 2023 (arXiv:2303.07678). A single LLM-generated expansion of the user query is enough to lift retrieval. The context is in-domain open QA; this article handles enterprise corpora where a full structured plan (corrected question + suggested prompts + answer shape + generation brief) earns its keep.Wang等人, Query2doc: Query Expansion with Large Language Models, EMNLP 2023 (arXiv:2303.07678)。单个LLM生成的用户查询扩展足以提升检索。背景是领域内开放问答；本文处理企业语料库，其中完整结构化计划（修正问题+建议提示+答案形状+生成简报）有其价值。
- Schick et al., Toolformer: Language Models Can Teach Themselves to Use Tools, NeurIPS 2023 (arXiv:2302.04761). The model decides when and which tool to call inline, with no upfront question parsing. Volume 3 (Agentic Bricks) develops this line on top of the structured plan defined here.Schick等人, Toolformer: Language Models Can Teach Themselves to Use Tools, NeurIPS 2023 (arXiv:2302.04761)。模型自行决定何时调用何种工具，无需预先问题解析。第3卷（智能体积木）在此定义的结构化计划之上发展此路线。
Earlier in the series:系列前文：
- Document Intelligence: series intro. What the series builds, brick by brick, and in what order.文档智能：系列介绍。系列逐步构建的内容及顺序。
- Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out.基线企业RAG，从PDF到高亮答案。四积木流水线端到端：输入PDF，输出高亮答案。
- Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.嵌入并非魔法：RAG检索的可预测失败模式。嵌入相似性何时有效（同义词、拼写错误、释义），何时可预测地失败（未知术语、否定、术语与答案相关性），以及如何仍能使用。
- Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost. What a cross-encoder adds over bi-encoder embeddings, measured, and when it is worth the latency.重排序器也非魔法：交叉编码器层何时值得成本。交叉编码器相比双编码器嵌入的增益（经测量），以及何时值得延迟。
- RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.RAG不是机器学习，ML工具包解决错误问题。为何块大小扫描和微调优化了错误方向；应改为按问题类型路由。
- From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case.从正则表达式到视觉模型：哪种RAG技术适合哪种问题。两个轴——文档复杂度和问题控制度——为每种情况选择技术。
- 10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each.生产环境中我们持续看到的10个常见RAG错误。十个生产错误，按积木组织，附每个的修复方法。
- Beyond extract_text: the two layers of a PDF that drive RAG quality. The first half of the parsing brick: the document’s nature, signals, and summary.超越extract_text：驱动RAG质量的PDF两层结构。解析积木的前半部分：文档的性质、信号和摘要。
- Stop returning flat text from a PDF: the relational shape RAG needs. The second half of the parsing brick: the relational tables every downstream brick reads.停止从PDF返回纯文本：RAG所需的关系形状。解析积木的后半部分：每个下游积木读取的关系表。
