The mainstream story has retrieval as embed the question, return top-k by cosine, optionally rerank. We disagree with almost every part of it. Retrieval is filtering on structured tables, not searching free text. Embeddings are the optional fallback, not the foundation. Anchor and context are two granularities, not one. Each of these is a position we can defend, with consequences you can measure.主流叙事将检索描述为：对问题进行嵌入、按余弦相似度返回top-k、可选地重排序。我们几乎不同意其中的每一部分。检索是在结构化表格上进行过滤，而不是搜索自由文本。嵌入是可选的备选方案，而非基础。锚点和上下文是两种粒度，而非一种。这些立场我们都能用可度量的后果来辩护。
📓 Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1.📓 可运行的配套笔记本在GitHub上：doc-intel/notebooks-vol1。
The naive baseline this article pushes back on本文所反驳的朴素基线
The naive pipeline chunks the document, embeds every chunk, embeds the question, ranks by cosine. That single signal is opaque, and it throws away the document’s structure. We keep the document as line_df + toc_df and run three retrieval signals in parallel (keyword on lines, TOC reasoning, embedding cosine), then let an LLM arbiter rank once at the end with all three sets of hits in view.朴素流水线将文档分块，对每个块进行嵌入，对问题进行嵌入，然后按余弦相似度排序。那单一信号不透明，且丢弃了文档结构。我们将文档保留为 line_df + toc_df，并行运行三个检索信号（行关键词、目录推理、嵌入余弦），然后让LLM裁判在最后对三组命中结果进行一次排序。
Below are the six untaught lessons of this brick.以下是该砖块的六个未授之课。
Lesson 1 – Retrieval is filtering, not searching第一课 – 检索是过滤，而非搜索
Once parsing is done, retrieval is a SQL-like filtering problem over line_df and toc_df, the reverse of the chunk-embed-cosine-top-k framing. The shift is simple to state: the question has columns, the document has columns, and retrieval is the join.一旦解析完成，检索就是基于 line_df 和 toc_df 的类似SQL的过滤问题，与分块-嵌入-余弦-top-k框架相反。这一转变说起来很简单：问题有列，文档有列，检索就是连接。
Why it matters. Search and filter are not synonyms , the two operations have different mechanics. Search scores every candidate on a continuous similarity (cosine , BM25), forces a top-k cutoff, and always returns something, even when the answer is not in the document. Filter applies a boolean condition (line.contains("X") , toc.title in [...]), retains every row that matches and no more, and can return zero rows when the document does not carry the answer. The audit consequence is the largest part of the gap: a filter’s condition is one line of inspectable code that runs the same way in six months; a search’s ranking depends on which dimensions of the embedding mattered, and you cannot replay that judgment without re-running the model.为什么这很重要。搜索和过滤不是同义词，两种操作机制不同。搜索在连续相似度（余弦、BM25）上对每个候选进行评分，强制设定top-k截断，即使答案不在文档中也总是返回内容。过滤应用布尔条件（line.contains("X")、toc.title in [...]），仅保留匹配的行，不多不少，当文档不含答案时可以返回零行。审计后果是最大差距所在：过滤的条件是一行可检查的代码，六个月后仍以相同方式运行；搜索的排名取决于嵌入的哪些维度起了作用，且如果不重新运行模型，无法重放该判断。
Concrete contrast. The user asks “What positional encoding does the paper use?”. Naive RAG embeds the question, scores 300+ chunks, returns the top-5. Series RAG filters line_df where the line contains "positional encoding" (4 hits), filters toc_df where the section title contains "positional" (1 section, 3.5 Positional Encoding), and the arbiter sees both, anchor: the line; scope: the section. No cosine needed.具体对比。用户问“论文使用了什么位置编码？”。朴素RAG对问题嵌入，对300多个块评分，返回top-5。系列RAG过滤 line_df 中行包含“positional encoding”（4个命中），过滤 toc_df 中章节标题包含“positional”（1个章节，3.5 Positional Encoding），裁判看到两者：锚点：该行；范围：该章节。无需余弦。
→ Article 7A: Retrieval is filtering, not search lays out the mental model.→ 文章7A：检索是过滤，而非搜索，阐述了思维模型。
Lesson 2 – Anchor and context, kept apart第二课 – 锚点与上下文分离
You anchor on the single line that mentions “premium” (precise) but pass the whole surrounding section to generation (sufficient context); conflating them breaks precision and coverage in one move. Top-k forces you to pick: tiny chunks lose context, huge chunks lose precision. We get both, by keeping them apart.你锚定在提到“保费”的单一行（精确），但将整个周围章节传递给生成（充足上下文）；混为一谈会同时破坏精确性和覆盖度。Top-k迫使你选择：小块丢失上下文，大块丢失精确性。我们通过保持分离来兼得两者。
Concrete contrast. For a definition question, the anchor is the one line ( "the deductible is the amount the insured pays before coverage begins" ), the scope is the paragraph around it ( three sentences of context the LLM needs to phrase the answer ). Naive top-k either returns the line (no context) or the paragraph (anchor unclear). Series retrieval returns anchor + scope as a typed pair.具体对比。对于定义问题，锚点是那一行（“免赔额是被保险人在保险开始前支付的金额”），范围是周围的段落（LLM需要用来组织答案的三句上下文）。朴素top-k要么返回行（无上下文），要么返回段落（锚点不清）。系列检索返回锚点+范围作为类型化对。
→ Article 7A: Retrieval is filtering, not search draws the line between anchor and context.→ 文章7A：检索是过滤，而非搜索，划定了锚点和上下文之间的界限。
Lesson 3 – Embeddings come last, not first第三课 – 嵌入放在最后，而不是最先
Keywords always run (cheap, deterministic); the document’s own TOC is a first-class retrieval method; embeddings are the optional final signal, only when vocabulary mismatch is expected. The 2024-era reflex starts with embeddings; we leave them for the cases where the cheaper signals failed.关键词始终运行（廉价、确定）；文档自身的目录是一流检索方法；嵌入是可选的最终信号，仅在预期词汇不匹配时使用。2024年时代的直觉反应是从嵌入开始；我们将它们留给更廉价信号失败的案例。
Concrete contrast. A factual lookup on insurance policy: “effective date?”. Naive RAG embeds, returns 5 chunks. Series runs keyword on "effective" and "date" → 1 line found → done. Embeddings never ran. Cost: one regex pass over line_df; a few milliseconds. The 2-cent cosine search did not happen.具体对比。在保险单上事实查找：“生效日期？”。朴素RAG嵌入，返回5个块。系列对“effective”和“date”运行关键词 → 找到1行 → 完成。嵌入从未运行。成本：一次对line_df的正则遍历；几毫秒。2美分的余弦搜索没有发生。
→ Article 7B: Finding the right anchors builds the three-signal pipeline.→ 文章7B：找到正确的锚点构建了三信号流水线。
Lesson 4 – Keywords prove absence; embeddings cannot第四课 – 关键词证明缺失；嵌入不能
A zero on keyword search means the answer is genuinely not there; a zero on embedding similarity could be absence or just different words, so embeddings are a refinement, not a decision gate. This asymmetry is the case for keywords as the primary signal in enterprise RAG.关键词搜索的零结果意味着答案确实不存在；嵌入相似度的零结果可能是不存在或只是措辞不同，因此嵌入是细化手段，而非决策门。这种不对称性是关键词作为企业RAG主要信号的理由。
Concrete contrast. The user asks “does this contract cover earthquake damage?” on a flood-only policy. Keyword search for "earthquake" returns zero matches in line_df . The pipeline can ship answer_found = False confidently. Embedding cosine returns 5 chunks (the closest topically related lines about natural disasters ) and the LLM, seeing them, may infer a wrong yes. Keywords saved the day.具体对比。用户在一份仅保洪水的保单上问“这份合同覆盖地震损害吗？”。关键词搜索“earthquake”在line_df中返回零匹配。流水线可以自信地输出 answer_found = False。嵌入余弦返回5个块（关于自然灾害的话题最相关的行），LLM看到后可能推断出错误的肯定答案。关键词拯救了局面。
→ Article 7B: Finding the right anchors explains the keyword-first discipline.→ 文章7B：找到正确的锚点解释了关键词优先的原则。
Lesson 5 – Co-occurrence beats BM25 on narrow corpora第五课 – 共现在小规模语料上胜过BM25
BM25 ranks by term frequency, but the enterprise answer shape is one mention of a topic next to a specific value, so co-occurrence boosts and high-value regex anchors beat statistical IDF on narrow corpora. The IDF assumptions break on a 20-document corpus where every term is “rare” by Wikipedia standards.BM25按词频排序，但企业答案形态是主题与特定值相邻出现，因此共现提升和高价值正则锚点在小规模语料上胜过统计IDF。IDF假设在20个文档的语料上失效，其中每个词按维基百科标准都是“稀有”的。
Concrete contrast. The question is “what is the deductible amount?”. BM25 ranks by frequency of "deductible"; the line that appears 12 times in a glossary section ranks first. Co-occurrence search ranks lines that contain both "deductible" and a number; the actual policy line ( "the deductible is $1000" ) ranks first because it co-occurs with $1000 , and the LLM can extract the value cleanly.具体对比。问题是“免赔额是多少？”。BM25按“deductible”的频率排序；在术语表中出现12次的行排第一。共现搜索对同时包含“deductible”和数字的行排序；实际保单行（“the deductible is $1000”）因为与 $1000 共现而排第一，LLM可以干净地提取该值。
→ Article 7B: Finding the right anchors measures co-occurrence against BM25.→ 文章7B：找到正确的锚点测量了共现与BM25的比较。
Lesson 6 – One LLM pass over the TOC第六课 – 对目录进行一次LLM遍历
Handing the 20-100 row toc_df to a small model and asking which sections answer the question costs one cached call and catches the paraphrases (“exit early” ≈ “Termination”) keyword matching misses.将20-100行的toc_df交给一个小模型并询问哪些章节回答了问题，只需一次缓存调用，并能捕获关键词匹配遗漏的同义表述（“提前退出” ≈ “终止”）。
TOC reasoning is one of the most under-used retrieval signals in production RAG.目录推理是生产级RAG中最被低估的检索信号之一。
Concrete contrast. The user asks “when can I leave the policy early?”. Substring matching on "leave" returns zero TOC entries. An LLM call on the full TOC ( 28 rows, fits in a single small prompt ) returns section “Termination and Cancellation”, the correct paraphrase. One cached LLM call, deterministic afterwards, and the right anchor.具体对比。用户问“我何时可以提前退出保单？”。对“leave”的子串匹配返回零目录条目。对完整目录（28行，适合单个小型提示）的LLM调用返回章节“Termination and Cancellation”，正确的同义表达。一次缓存的LLM调用，后续确定，且得到正确的锚点。
→ Article 7B reasons over the TOC, and Article 7C: An LLM as arbiter adds the arbiter.→ 文章7B对目录进行推理，文章7C：LLM作为裁判添加了裁判。
The six lessons share one move: refuse the chunk-embed-cosine reflex, and treat retrieval as filtering on structured tables instead. Keywords always run because they prove absence; the TOC is a first-class signal because the document already declared its structure; embeddings are the optional refinement, not the foundation. The deep-dives (7A, 7B, 7C, 7bis) ship runnable code on real documents; this piece is the catalogue that points at them.这六课共享一个动作：拒绝分块-嵌入-余弦的直觉反应，而将检索视为在结构化表格上的过滤。关键词始终运行因为它们证明缺失；目录是一流信号因为文档已声明其结构；嵌入是可选的细化手段，而非基础。深度文章（7A、7B、7C、7bis）提供了在真实文档上可运行的代码；本文是指向它们的目录。
Across sectors and professions跨行业与职业
The same three-signal retrieval pattern ( keyword on line_df + reasoning on toc_df + embedding fallback ) holds in every domain. The vocabulary and the TOC depth differ; the signal hierarchy does not. Five sectors below, one retrieval pattern, one audit trace per call.相同的三信号检索模式（对line_df的关键词 + 对toc_df的推理 + 嵌入备选）在所有领域都适用。词汇和目录深度有所不同；信号层次不变。以下是五个行业，一种检索模式，每次调用一条审计追踪。
Embeddings fire only on the medical row, where the user’s vocabulary ( “tachycardia” ) diverges from the document’s ( “rapid heart rate” ). The other four rows resolve entirely on keyword + TOC. Keywords prove absence (Lesson 4), the TOC catches paraphrases (Lesson 6), and the anchor / scope split keeps precision and context apart (Lesson 2) in every row. The cost gradient is real: the four keyword-resolved rows run in milliseconds with zero LLM tokens; the medical row pays for one embedding pass and one arbiter call.嵌入仅在医疗行触发，其中用户的词汇（“心动过速”）与文档的（“心率过快”）不同。其他四行完全由关键词+目录解决。关键词证明缺失（第四课），目录捕获同义表达（第六课），锚点/范围分离保持精确性和上下文分离（第二课）在每一行中。成本梯度是真实的：四行由关键词解决，运行时间毫秒级，零LLM令牌；医疗行为一次嵌入传递和一次裁判调用付出成本。
Sources and further reading来源与延伸阅读
The mainstream literature on retrieval is shaped by web-scale search and shorter consumer corpora. The series stance assumes a small enterprise corpus where the structure is known and the vocabulary is the asset.关于检索的主流文献受网络规模搜索和较短消费者语料影响。该系列立场假设一个小的企业语料，其结构已知，词汇是资产。
- Retrieval is filtering, not search (Article 7A). The published mental-model article: retrieval as filtering on structured tables.检索是过滤，而非搜索（文章7A）。已发布的思维模型文章：检索视为在结构化表格上的过滤。
- Embeddings Aren’t Magic (Article 2). The published failure-modes catalogue for embedding similarity.嵌入并非魔法（文章2）。已发布的嵌入相似度故障模式目录。
- Rerankers Aren’t Magic Either (Article 2bis). When the cross-encoder pays off and when it does not.重排序器同样并非魔法（文章2bis）。交叉编码器何时有效，何时无效。
