How Anthropic enables self-service data analytics with Claude

Our agentic analytics stack 我们的代理分析栈

At Anthropic, the main way we minimize these three errors is via our agentic data stack. Each layer exists primarily to attack one or more of these problems: 在Anthropic，我们最小化这三种错误的主要方式是通过我们的代理数据栈。每一层主要针对其中一个或多个问题：

1. Entity ambiguity: data foundations and sources of truth shrink the space of plausible entities until there's a single governed answer. 实体模糊性：数据基础和事实来源缩小了可能实体的范围，直到只有一个受治理的答案。

2. Staleness: maintenance and validation processes keep everything from rotting as the business changes.过时：维护和验证流程确保随着业务变化，一切都不会腐烂。

3. Retrieval failure: skills make sure the agent reliably finds and correctly uses that answer. 检索失败：技能确保代理可靠地找到并正确使用该答案。

In this section, we’ll discuss how we built each layer. 在本节中，我们将讨论如何构建每一层。

Data foundations数据基础

The most important aspect of ensuring analytics agents are accurate is via strong data foundations, which include the data models, transforms, tests, and tables in a data warehouse, along with the metadata describing them. Standard data engineering and data quality practices such as dimensional modeling, shift-left testing, freshness and completeness checks on critical pipelines all still apply (and we won't relitigate these).确保分析代理准确性的最重要方面是通过强大的数据基础，这包括数据仓库中的数据模型、转换、测试和表，以及描述它们的元数据。标准的数据工程和数据质量实践，如维度建模、左移测试、关键管道的时效性和完整性检查，仍然适用（我们不会重新讨论这些）。

What does change is that the end user of your data model is no longer a data expert (e.g. data scientist), but rather agents acting on behalf of users with varying degrees of data expertise or understanding of the underlying infrastructure. This shift presents a challenge in that the results can’t require the user to validate the underlying correctness simply because the end user doesn’t know.变化的是，数据模型的最终用户不再是数据专家（例如数据科学家），而是代表具有不同数据专业知识或对底层基础设施理解程度的用户行事的代理。这种转变带来了一个挑战，即结果不能要求用户验证底层正确性，仅仅因为最终用户不知道。

The data foundations layer is aimed primarily at ambiguity: if revenue, for example, resolves to one governed dataset instead of forty plausible candidates, the problem largely disappears before the agent ever has to search. It's also where the first staleness defense lives, since the same repo that defines the canonical models is the natural place to enforce that they stay current.数据基础层主要针对模糊性：例如，如果收入解析为一个受治理的数据集而不是四十个可能的候选，那么问题在代理搜索之前就基本消失了。这也是第一个防过时防御所在，因为定义规范模型的同一仓库是强制执行它们保持最新的自然场所。

We’ve seen a few practices work especially well:我们看到一些实践特别有效：

• Create canonical datasets: By far the most common failure is that the agent can’t map a concept (“revenue for product X”) to the single correct table, column, and metric definition, usually because there are multiple plausible candidates with subtly different implementations. The fix is fewer, more heavily governed logical models: curate a small set of canonical, single source-of-truth datasets that are clearly owned, consumption-ready, and discoverable, then aggressively deprecate the near-duplicates. Physical rollups and caches still matter for cost and performance, but they should derive mechanically from the canonical models rather than living alongside them as alternatives. The goal is that when an agent searches for a concept, it finds a single governed answer.创建规范数据集：到目前为止，最常见的失败是代理无法将概念（“产品X的收入”）映射到单个正确的表、列和指标定义，通常是因为存在多个具有细微不同实现的可行候选。解决方案是更少、更受治理的逻辑模型：策划一小部分规范的、单一事实来源的数据集，这些数据集拥有明确的所有权、可供消费且可发现，然后积极弃用近似的重复项。物理汇总和缓存对于成本和性能仍然重要，但它们应从规范模型机械地派生，而不是作为替代方案与之共存。目标是当代理搜索一个概念时，它找到一个单一的受治理答案。
• Enforce your standards: We’ve found the foundations only hold if the canonical models and metric definitions are enforced by tooling (the agent is structurally routed to them first; more on that below), by CI (changes that bypass them fail review), and by mandate (downstream teams build on the governed layer or explain why not). Governance without enforcement otherwise quickly decays back to the multiple candidates problem.强制执行标准：我们发现，只有通过工具（代理在结构上首先被路由到它们，下面会详细介绍）、CI（绕过它们的更改会失败审查）和指令（下游团队在治理层上构建或解释原因）强制执行规范模型和指标定义，基础才能保持。否则，没有强制执行的治理很快就会退化回多个候选问题。
• Colocate artifacts: Our main defense against constantly changing data models and business logic is colocation. Nearly all data code (i.e., modeling, semantic layer, reference docs, canonical dashboard definitions) lives in a single repo, with CI checks that protect cross-layer integrity. If a modeling change would break a downstream dashboard or invalidate a documented metric, CI flags it and the fix ships in the same PR. (We’ll come back to the mechanics of this in the Skills section below.)共置工件：我们对抗不断变化的数据模型和业务逻辑的主要防御是共置。几乎所有数据代码（即建模、语义层、参考文档、规范仪表板定义）都位于一个仓库中，并带有保护跨层完整性的CI检查。如果建模更改会破坏下游仪表板或使记录的指标无效，CI会标记它，并且修复会在同一个PR中发布。（我们将在下面的技能部分回到这个机制。）
• Treat metadata as a first-class product: Coding agents perform well partly because codebases are legible: READMEs, type signatures, docstrings, etc. Your warehouse can be just as legible, but only if column and table descriptions, canonical metric definitions, grain documentation, valid value ranges, lineage, ownership, and model tiering are maintained with the same rigor as the transformations themselves. While not a new insight, good governance provides critical context that helps the agent choose the right dataset.将元数据视为一流产品：编码代理表现良好，部分原因是代码库易于理解：README、类型签名、文档字符串等。你的数据仓库也可以同样易于理解，但前提是列和表描述、规范指标定义、粒度文档、有效值范围、血缘、所有权和模型层级与转换本身一样严格维护。虽然这不是新见解，但良好的治理提供了关键上下文，帮助代理选择正确的数据集。

Sources of truth事实来源

If data foundations are the data warehouse itself, sources of truth are the reference surfaces the agent consults to navigate it. This layer reduces concept <> entity ambiguity and turns “weekly active users” in a stakeholder’s question into a specific, governed entity in your data model. Roughly in descending order of trust:如果数据基础是数据仓库本身，那么事实来源是代理咨询以导航它的参考表面。这一层减少了概念与实体之间的模糊性，并将利益相关者问题中的“每周活跃用户”转化为数据模型中的特定受治理实体。大致按信任度降序排列：

• Semantic layer: the compiled metric and dimension definitions. If a question maps cleanly to a defined metric, the agent calls a function and gets one number, the same number every other surface in the company produces. Our agents are structurally required (by skill instruction) to leverage the semantic layer first (see the appendix). One idea we tried that didn’t work: bootstrapping the semantic layer by having an LLM auto-generate metric definitions from raw tables and query logs. It produced plausible-looking definitions that encoded the very ambiguities we were trying to eliminate, and was net-negative on our evals versus a smaller, human-curated layer. Therefore we recommend generating the documentation with Claude, but having a human own the definition.语义层：编译的指标和维度定义。如果一个问题清晰地映射到一个定义的指标，代理调用一个函数并得到一个数字，与公司中其他任何表面产生的数字相同。我们的代理在结构上（通过技能指令）被要求首先利用语义层（见附录）。我们尝试过但未成功的一个想法：通过让LLM从原始表和查询日志自动生成指标定义来引导语义层。它产生了看似合理的定义，但编码了我们试图消除的模糊性，并且与较小的人工策划层相比，在我们的评估中净效果为负。因此，我们建议使用Claude生成文档，但由人类拥有定义。
• Lineage and the transformation graph: when the semantic layer doesn’t cover a question, lineage and table ranking (based on number of references) let the agent reason about which upstream models feed a concept, which are deprecated, and which share grain. This transforms “I don’t know the metric” into “I know which governed model to aggregate from.” It’s also the backbone of the freshness and provenance signals we surface in online validation below.血缘和转换图：当语义层不涵盖某个问题时，血缘和表排名（基于引用次数）让代理能够推理哪些上游模型为概念提供数据，哪些已被弃用，以及哪些共享粒度。这将“我不知道这个指标”转化为“我知道从哪个受治理模型进行聚合”。它也是我们在下面在线验证中展示的时效性和来源信号的主干。
• Query corpus: historical SQL from dashboards, notebooks, and prior analyses. Intuitively, this should be high-value: it’s a record of every question already answered correctly. In practice, we found that giving the agent raw retrieval access to thousands of prior queries moved accuracy by less than a point (we walk through that ablation in a later section below). Unstructured retrieval couldn’t map a new question to the right precedent. What does work is distilling that corpus into structured per-domain reference docs and reusable analysis patterns described in skills. Treat the query history as raw material for curation, not as a source of truth the agent reads directly.查询语料库：来自仪表板、笔记本和先前分析的历史SQL。直观上，这应该很有价值：它记录了每个已经正确回答的问题。在实践中，我们发现让代理对数千个先前查询进行原始检索访问，准确率提升不到一个百分点（我们在后面的部分中会介绍这个消融实验）。非结构化检索无法将新问题映射到正确的先例。有效的是将该语料库提炼为结构化的按领域参考文档和技能中描述的可重用分析模式。将查询历史视为策划的原材料，而不是代理直接读取的事实来源。
• Business context: the layer most teams skip, and the one we underrated the longest. An agent that doesn’t understand your business will answer what the user asked, but not what they meant. It won’t know that “the Q2 launch” refers to a specific product, that two teams define the same term differently, or that a question is being asked because a board meeting is on Thursday. We pipe in a company knowledge graph consisting of indexed docs, roadmaps, decision logs, and our organizational structure so the agent can resolve ambient references and ask better clarifying questions. 业务上下文：大多数团队跳过的一层，也是我们低估最久的一层。不了解你业务的代理会回答用户所问的问题，但不会回答他们真正意图的问题。它不会知道“Q2发布”指的是特定产品，两个团队对同一术语的定义不同，或者问题被提出是因为董事会会议在周四。我们引入了一个公司知识图谱，包含索引文档、路线图、决策日志和组织结构，以便代理能够解析环境引用并提出更好的澄清性问题。

The common failure pattern across all four is the same one from the data foundations layer: poor or stale documentation. Claude is exceptionally useful for closing the gap (drafting column descriptions, proposing metric docs from query patterns, flagging undocumented models in CI), but the curation and ownership are managed by humans. 所有四个层面的常见失败模式与数据基础层相同：文档质量差或过时。Claude在缩小差距方面非常有用（起草列描述、从查询模式提出指标文档、在CI中标记未记录的模型），但策划和所有权由人类管理。

In the next two sections, we discuss how to make that ownership cheap enough that it actually happens.在接下来的两节中，我们将讨论如何使这种所有权足够廉价，以便它真正发生。

Skills技能

If the sources of truth are the agent's declarative knowledge (i.e., what a metric means) then a skill is its procedural knowledge: which sources to consult in what order, how to navigate ambiguous data, and what a finished analysis looks like. 如果事实来源是代理的声明性知识（即指标的含义），那么技能就是它的程序性知识：按什么顺序咨询哪些来源，如何导航模糊数据，以及完成的分析应该是什么样子。

In Claude Code, a skill is a folder of markdown the agent reads on demand. At Anthropic, the skills we developed are hugely value additive. Without skills, Claude’s ability to answer analytics questions accurately didn’t exceed 21% on our evals. Adding skills gets these numbers consistently above 95% in aggregate and regularly around 99% in certain domains. See the appendix for a skeleton we use to create a majority of our skills.在Claude Code中，技能是一个按需由代理读取的markdown文件夹。在Anthropic，我们开发的技能非常有价值。没有技能，Claude准确回答分析问题的能力在我们的评估中不超过21%。添加技能后，这些数字在总体上持续超过95%，并且在某些领域经常达到99%左右。请参阅附录，了解我们用于创建大多数技能的框架。

Some best practices:一些最佳实践：

Create pairwise skills: a knowledge skill acts as a thin top-level router that allows additional domain details to load on demand. It says "try the semantic layer first, but if there’s no coverage, here are ~30 reference files for this domain describing the relevant tables, columns, joins and gotchas.” This router is, in effect, our answer to retrieval failure: rather than letting the agent search a million-field warehouse, it narrows the space to a few dozen curated files before a query is ever written. The unbook skill encodes the process a senior analyst would follow: clarify the question, find sources (via the knowledge skill), run the query, and then loop the result through adversarial review sub-agents. It also bundles a dozen reusable analysis patterns (retention curves, rate decomposition, funnel analysis) so that common requests don't get reinvented each time.创建成对技能：知识技能充当一个薄薄的顶层路由器，允许按需加载额外的领域细节。它说“先尝试语义层，但如果没有覆盖，这里有大约30个该领域的参考文件，描述相关的表、列、连接和注意事项。”这个路由器实际上是我们对检索失败的答案：与其让代理搜索一个拥有百万字段的仓库，它在编写查询之前就将空间缩小到几十个策划文件。非技能编码了高级分析师会遵循的过程：澄清问题，找到来源（通过知识技能），运行查询，然后将结果通过对抗性审查子代理循环。它还捆绑了十几个可重用的分析模式（留存曲线、比率分解、漏斗分析），这样常见请求就不会每次都重新发明。

Create proper reference docs: written for retrieval by an LLM. Our reference docs describe tables (grain, scope, and exclusions), the mechanics of gotchas (e.g., “exclude known free-email domains, but keep custom ones like anthropic.com”), and explicit routing triggers (e.g., “IF the question is about experiment lift… DO NOT use for raw event counts”) without prescriptive recipes that go stale. See below for a skeleton we use to create reference docs.创建适当的参考文档：为LLM检索而编写。我们的参考文档描述表（粒度、范围和排除项）、注意事项的机制（例如，“排除已知的免费电子邮件域名，但保留自定义域名如anthropic.com”），以及明确的路由触发器（例如，“如果问题涉及实验提升……不要用于原始事件计数”），而不包含会过时的规定性配方。请参阅下面我们用于创建参考文档的框架。

# [Domain] Tables

## Quick Reference
### Business Context — [what this domain means in plain words]
### Entity Grain — [what one row represents]
### Standard Hygiene Filter — [the filter every query in this domain applies]

## Dimensions
- [How the key dimensions are encoded, and how the same concept is named
  differently across tables]

## Key Tables
### [table_name]
- **Grain**: [...] · **Scope/exclusions**: [...]
- **Usage**: [when to use it, when NOT to, join keys, required filters]
[... one short section per governed table ...]

## Gotchas
- [The wrong-answer modes a senior analyst would warn you about]

## Best Practices / Common Query Patterns
- [Default choices, standard cuts, worked patterns where the exact query
  form is the hard part]

## Cross-References
- [Neighboring domain docs that own adjacent questions]

Treat skill maintenance as a first class citizen: Skill docs describe a data model that changes daily, so without active maintenance they're wrong within weeks. We watched our offline accuracy drift from ~95% at launch to ~65% over a month before we treated this as an engineering problem. That meant colocating skill markdown files in the same repo as our transformation models, so the PR that changes a model is the same PR that updates the doc describing it. A code-review hook flags any reporting-model change that doesn't touch a skill file. Roughly 90% of our data-model PRs now include a skill change in the same diff. We also regularly prune skill scaffolding as models improve and previous failure modes no longer apply.将技能维护视为一等公民：技能文档描述了一个每天都在变化的数据模型，因此如果没有主动维护，它们会在几周内出错。我们观察到离线准确率从发布时的约95%在一个月内下降到约65%，然后我们才将其视为一个工程问题。这意味着将技能markdown文件与我们的转换模型放在同一个仓库中，这样更改模型的PR就是更新描述它的文档的同一个PR。一个代码审查钩子会标记任何未触及技能文件的报告模型更改。大约90%的数据模型PR现在在同一个差异中包含技能更改。我们还定期修剪技能框架，因为模型改进且以前的失败模式不再适用。

Create a consistent and seamless experience across all surfaces: the same skill must provide the same answer to questions in Slack, in the IDE, in a dashboard tool, and in standalone agent sessions. We did this by ensuring one canonical source (the data repo) and that skill changes are synced automatically. On merge, the skill syncs to a plugin marketplace (for IDE users), to cloud-storage blobs (for hosted apps that read a single file), and is served directly as resources over MCP. We also designed for portability from the start by avoiding hardcoded repo paths and surface-specific namespaces.在所有表面上创建一致且无缝的体验：相同的技能必须在Slack、IDE、仪表板工具和独立代理会话中为问题提供相同的答案。我们通过确保一个规范来源（数据仓库）以及技能更改自动同步来实现这一点。合并后，技能同步到插件市场（供IDE用户使用）、云存储blob（供读取单个文件的托管应用使用），并作为资源通过MCP直接提供。我们还从一开始就设计了可移植性，避免硬编码的仓库路径和特定于表面的命名空间。

Validation验证

Finally, validation is how you find out which of the three failure modes is still leaking through. 最后，验证是你发现三种失败模式中哪一种仍然存在的方式。

Offline evaluations离线评估

A common pattern we see is that data teams will set up elaborate analytic environments without having any process to understand the accuracy of their analytics agents.我们看到的一个常见模式是，数据团队会建立复杂的分析环境，却没有任何流程来了解其分析代理的准确性。

One way of addressing this gap is via offline evals, which are simple question / answer pairs. You can think of offline evals similar to offline testing for an ML model in that they don’t tell you the performance of your online agents, but they do give you a good sense of whether you’ll have any critical gaps.解决这一差距的一种方法是通过离线评估，即简单的问题/答案对。你可以将离线评估视为类似于ML模型的离线测试：它们不会告诉你在线代理的性能，但确实能让你很好地了解是否存在任何关键差距。

We deploy two kinds of offline evals at Anthropic. Dashboard-based evals are auto-generated by Claude (then human validated), covering the most common stakeholder questions. Long tail evals are where we feed Claude business context (roadmaps, table docs) and have it generate plausible questions across the rest of the domain. We also continuously harvest every time a stakeholder corrects the agent in a thread as that correction is a candidate eval.我们在Anthropic部署了两种离线评估。基于仪表板的评估由Claude自动生成（然后由人工验证），涵盖最常见的利益相关者问题。长尾评估是我们向Claude提供业务上下文（路线图、表文档）并让它生成跨领域其余部分的合理问题。我们还持续收集每次利益相关者在线程中纠正代理的情况，因为该纠正是一个候选评估。

Other best practices, include:其他最佳实践包括：

• Anchor ground truth so it can't drift: An eval written against live data goes stale the moment the underlying number moves. Pin every eval to a snapshot date, write it against a stable fact table, or have the grader judge the agent's query rather than its number. Wire the suite into CI so a PR touching a dependency re-runs the affected evals.锚定真实答案以防止漂移：针对实时数据编写的评估在底层数字变化的那一刻就会过时。将每个评估固定到一个快照日期，针对稳定的事实表编写，或者让评分者判断代理的查询而不是其数字。将套件接入CI，以便触及依赖项的PR重新运行受影响的评估。
• Store results like telemetry, not like test logs: Every run lands in a warehouse table with the skill version, git SHA, model ID, per-assertion pass/fail, token count, and wall-clock. "Did that change help?" becomes a query, and you get the time-series to catch slow regressions that a single CI run won't.像存储遥测数据一样存储结果，而不是像测试日志：每次运行都会进入一个仓库表，包含技能版本、git SHA、模型ID、每个断言的通过/失败、令牌计数和挂钟时间。“那个更改有帮助吗？”变成了一个查询，并且你可以获得时间序列来捕捉单个CI运行无法发现的缓慢回归。
• Gate launches per domain: A domain owner can't announce the agent to their stakeholders until their slice of the eval set clears some threshold (we initially used ~90%). It forces reference-doc fixes before users see the failures.按领域门控发布：领域所有者不能向利益相关者宣布代理，直到他们那部分评估集超过某个阈值（我们最初使用约90%）。这迫使在用户看到失败之前修复参考文档。
• Create the appropriate number of evals: The number of evals you should have depends on the complexity of the business area and the complexity of the underlying data model. Calibrate by tracking how well offline accuracy predicts online accuracy: we’ve found there are diminishing returns past a few dozen per topic (e.g., “growth”), and that ceiling drops with each new model generation.创建适当数量的评估：你应该拥有的评估数量取决于业务领域的复杂性和底层数据模型的复杂性。通过跟踪离线准确性预测在线准确性的程度来校准：我们发现每个主题（例如，“增长”）超过几十个后收益递减，并且随着每个新模型代际，这个上限会下降。
• Offline eval accuracy should be ~100%; every correct answer should also be hitting your semantic layer (if you have one). Again, this level of accuracy doesn’t tell you your system isn’t going to produce a wrong answer, just that there are no obvious gaps, assuming you have proper eval coverage.离线评估准确率应接近100%；每个正确答案也应命中你的语义层（如果有的话）。再次强调，这种准确率水平并不能说明你的系统不会产生错误答案，只是说明没有明显的差距，假设你有适当的评估覆盖。

Ablation techniques消融技术

Every structural decision about the skill (e.g., which sources to expose, whether a sub-agent earns its latency, whether to merge two skills into one) is made by holding our offline eval set fixed. 关于技能的每个结构性决策（例如，暴露哪些来源，子代理是否值得其延迟，是否将两个技能合并为一个）都是通过保持离线评估集固定来做出的。

We vary exactly one component and compare pass rates. Each run only takes an hour and replaces a lot of arguments. The methodology matters more than any single result:我们只改变一个组件并比较通过率。每次运行只需一个小时，并取代了许多争论。方法论比任何单一结果更重要：

• Design for null results. Our most useful ablation was a negative one. We gave the agent direct grep access to our entire dashboard, transformation, and analyst-notebook SQL (thousands of files). We then verified in transcripts that it actually read them before every answer. Accuracy moved by less than a point in either direction. We then checked the obvious confounds: was the answer actually in the corpus for the questions it got wrong? About 80% of the time, yes. Did "answer present" predict "now gets it right"? No, the flip rate was flat. The information was there, the agent saw it, and it still didn’t use it. That single experiment told us our bottleneck wasn't access to prior work, it was structure (i.e., mapping a question to the right entity). That insight redirected months of roadmap.为无效结果设计。我们最有用的消融是一个负面消融。我们让代理直接grep访问我们整个仪表板、转换和分析师笔记本SQL（数千个文件）。然后我们在转录中验证它在每个答案之前确实读取了它们。准确率在任一方向上的变化不到一个百分点。然后我们检查了明显的混淆因素：对于它答错的问题，答案是否实际上在语料库中？大约80%的情况下，是的。“答案存在”是否预测“现在答对了”？不，翻转率是平的。信息在那里，代理看到了，但仍然没有使用它。那一个实验告诉我们，我们的瓶颈不是对先前工作的访问，而是结构（即将问题映射到正确的实体）。这一洞察重定向了数月的路线图。
• Ablate at PR granularity. Every meaningful skill edit gets a before / after run on the relevant eval slice, with the delta in the PR description. It keeps "I improved the docs" honest and catches the surprisingly common case where a well-intentioned addition makes things worse.在PR粒度上进行消融。每个有意义的技能编辑都会在相关评估切片上运行前后对比，并将差异放在PR描述中。它让“我改进了文档”保持诚实，并捕获了令人惊讶的常见情况，即善意的添加反而使事情变得更糟。
• Keep a short list of what didn't work. Two of ours: stacking additional rounds of doc refinement past a certain point (we hit three consecutive net-negative iterations: the docs were getting longer, not better), and swapping the adversarial reviewer to a cheaper model to cut latency (it lost most of the accuracy wins, for no real speedup). Negative results are cheap to record and they prevent the next person from re-running the same experiment.保留一份不起作用的内容的简短列表。我们的两个：超过某个点后堆叠额外的文档改进轮次（我们遇到了连续三次净负迭代：文档变得更长，而不是更好），以及将对抗性审查者换成更便宜的模型以减少延迟（它失去了大部分准确性收益，而没有真正的速度提升）。负面结果记录起来很便宜，并且可以防止下一个人重新运行相同的实验。

Online validation在线验证

The final step is ensuring the actual online system performance is as accurate as possible. Some of the steps we take include:最后一步是确保实际的在线系统性能尽可能准确。我们采取的一些步骤包括：

• Adversarial review: we’ve found that employing a Claude skill to aggressively challenge all underlying assumptions on a potential final answer increased accuracy by 6% within our eval set, but at the cost of 32% more tokens and 72% higher latency.对抗性审查：我们发现，使用Claude技能积极挑战潜在最终答案的所有底层假设，在我们的评估集中将准确率提高了6%，但代价是令牌增加32%，延迟增加72%。
• Provenance footer: every response carries a footer that contains which source tier it came from (semantic layer › curated reference › raw table), how fresh the underlying data is, and who owns the model. It doesn't make the answer more correct, but it does help the consumer judge how much they can trust the response. A "raw table, freshness unknown" footer is a signal to verify before forwarding upstream, and it's one of the few mitigations we have for silent failures.来源页脚：每个响应都带有一个页脚，包含它来自哪个来源层级（语义层›策划参考›原始表）、底层数据的新鲜度以及谁拥有该模型。它不会使答案更正确，但确实帮助消费者判断他们可以信任响应的程度。“原始表，新鲜度未知”的页脚是在向上游转发之前进行验证的信号，并且是我们针对静默失败为数不多的缓解措施之一。
• Data quality checks: it’s possible that your agent is using the right field in the appropriate way, but the data itself is incorrect. Adding basic data quality checks to ensure the referenced field is up-to-date, complete, and has no anomalies is generally good hygiene.数据质量检查：你的代理可能以适当的方式使用了正确的字段，但数据本身可能不正确。添加基本的数据质量检查以确保引用的字段是最新的、完整的且没有异常，通常是一种良好的卫生习惯。
• Passive monitoring: two production signals we track continuously are the share of agent queries that resolve through the semantic layer, and the share of responses that use correction language ("that's the wrong table," "you're missing the fraud filter"). Both feed a dashboard reviewed weekly alongside the offline pass rate.被动监控：我们持续跟踪的两个生产信号是通过语义层解析的代理查询比例，以及使用纠正语言（“那是错误的表”，“你缺少欺诈过滤器”）的响应比例。两者都输入到一个与离线通过率一起每周审查的仪表板。
• Active correction harvesting: the part that closes the loop. A scheduled agent scans stakeholder channels every few hours for similar correction language, drafts a one-line fix to the relevant reference doc, and opens a PR tagged to the domain owner. The fix path is deliberately boring — edit a markdown file, merge, auto-sync everywhere — so a domain owner doesn’t spend too much time on the task. The same corrections feed back into the offline eval set.主动纠正收集：这是闭环的部分。一个定时代理每隔几小时扫描利益相关者频道以查找类似的纠正语言，起草对相关参考文档的一行修复，并打开一个标记给领域所有者的PR。修复路径故意设计得很简单——编辑一个markdown文件，合并，自动同步到所有地方——这样领域所有者就不会花太多时间在这项任务上。相同的纠正会反馈到离线评估集中。

The failure mode none of this fully catches is the silent one. The answer is wrong, but looks plausible and is used without objection. Our mitigations are the provenance footer, explicit human sign-off on anything leadership-bound, and a standing eval for each domain's top KPIs that sanity-checks against the blessed dashboard daily, though we don’t have a robust solution yet.这些都无法完全捕获的失败模式是静默失败。答案错误，但看起来合理，并且在使用时没有异议。我们的缓解措施是来源页脚、对任何面向领导层的内容进行明确的人工签核，以及每个领域顶级KPI的常设评估，每天对照受祝福的仪表板进行合理性检查，尽管我们还没有一个稳健的解决方案。

Getting started 入门

If you're starting from zero, a handful of canonical datasets, a few dozen offline evals, and a thin knowledge skill will capture most of the upside; everything else in this post is what we added once those were built.如果你从零开始，一小部分规范数据集、几十个离线评估和一个薄薄的知识技能将捕获大部分收益；本文中的其他所有内容都是我们在这些构建完成后添加的。

We also shared many best practices, and not all of them will be appropriate for every data team. Align with your organization on a few principles that will affect your approach by asking:我们还分享了许多最佳实践，并非所有实践都适用于每个数据团队。通过与你的组织就一些会影响你方法的原則达成一致，通过提问：

• How important is a correct answer today vs. in the future? AI models are progressing at a rapid pace. We often see companies building a significant amount of infrastructure to account for current model shortfalls that become moot once those models improve. Knowing where models fall short, and waiting for model improvements to fill the gap has significantly less overhead, but may not fit your company’s risk tolerance.今天正确的答案与未来的正确答案有多重要？AI模型正在快速发展。我们经常看到公司构建大量基础设施来弥补当前模型的不足，而这些不足在模型改进后就会变得无关紧要。了解模型的不足之处，并等待模型改进来填补空白，开销要小得多，但可能不符合你公司的风险承受能力。
• How do you anticipate the complexity of your business to change over time? Some of the processes we discussed may be overkill if, for example, you don’t produce much data, you only have a few consumers of the output, or your data model is likely to remain simple.你预计你的业务复杂性将如何随时间变化？我们讨论的一些流程可能是多余的，例如，如果你不产生太多数据，只有少数输出消费者，或者你的数据模型可能保持简单。
• How technical is the intended audience of the output? Phrased differently, if you’re building this analytics system for data scientists who can recognize when an answer is incorrect, you may be more tolerant of errors compared to a situation in which the audience has no familiarity with the underlying data model. 输出的目标受众有多技术性？换句话说，如果你为能够识别答案何时错误的数据科学家构建这个分析系统，你可能比受众对底层数据模型不熟悉的情况更能容忍错误。
• How much are you willing to spend for improved accuracy? We’ve found certain processes like adversarial validation can significantly improve accuracy, but often at a higher cost and latency.你愿意为提高准确性花费多少？我们发现某些流程，如对抗性验证，可以显著提高准确性，但通常成本更高且延迟更高。
• What is your comfort around access controls and internal data privacy? Agents are often significantly more performant the more context they have; however, broad data access cuts against most companies' governance posture. This determines whether you're building one agent or many scoped ones.你对访问控制和内部数据隐私的舒适度如何？代理通常拥有的上下文越多，性能越好；然而，广泛的数据访问与大多数公司的治理姿态相悖。这决定了你是构建一个代理还是多个有范围的代理。

Whatever your route, our greatest gains have come from addressing each of the three failure modes: collapsing ambiguity into a single governed answer, making the answer easily discoverable, and flagging when either has gone stale.无论你选择哪条路，我们最大的收益来自于解决三种失败模式：将模糊性压缩为一个受治理的答案，使答案易于发现，并在其中任何一个过时时发出标记。

This article was written by Chen Chang, Clement Peng, Justin Leder, Johanne Jiao, and Josh Cherry, members of the Data Science and Data Engineering team. The authors would like to thank Michael Segner for his contributions.本文由数据科学和数据工程团队的陈畅、Clement Peng、Justin Leder、Johanne Jiao和Josh Cherry撰写。作者感谢Michael Segner的贡献。

Appendix附录

Skill File Skeleton技能文件框架

What follows is the skeleton of our main warehouse skill: the real file's structure, with internal specifics replaced by [bracketed placeholders]. It isn't meant to be copied verbatim; it's meant to show the kinds of sections we found worth writing down.以下是我们主要仓库技能的基本框架：真实文件的结构，内部细节替换为[括号占位符]。它并不意味着逐字复制；而是展示我们认为值得写下来的各种部分。

---
name: [warehouse-skill]
version: [x.y.z]
description: "IF the user asks to query [the company]'s data warehouse for any
  [list of business domains] question — THEN invoke this skill. DO NOT invoke
  for [adjacent engineering tasks] or questions with no data-warehouse component."
---

# [Warehouse] Skill Instructions

## Description
The single source of truth for safe and effective [warehouse] querying.
Referenced by other skills [listed] for query execution guidance.

Act as a Data Analyst, providing strategic insights and data-driven
recommendations but seek guidance along the way.

**Out-of-scope decisions**: [product areas, etc.] → surface data only,
state "decision is [owning team]'s call", do NOT take a position or author
code fixes.

## Executing queries
Priority:
1. **[Managed connection]** (if available): [query tool] / [schema tool]
2. **[CLI fallback]** (if installed): [default project, fallback project]
3. **Neither** — ask the user to authenticate, then stop

---

# Semantic Layer (REQUIRED first step)

The governed semantic layer is the **mandatory default path** for every data
question — same numbers as [the BI tool], joins/grain/filters baked in. Raw SQL
via the reference docs below is the **fallback**, used only after the
semantic-layer path is shown not to cover the ask.

## Required workflow
1. **Load** — [how to load the semantic layer in each runtime, with fallbacks]
2. **Discover** — search measures/dimensions by keyword; **always check
   segments** (the named canonical population filters — hand-rolled WHERE
   clauses for these are the dominant wrong-answer mode)
3. **Compile + run** — build the spec → compile to SQL → execute
4. **Fallback** — only if discovery finds no relevant metric or compile fails
   → raw SQL via `references/*.md` (PART 3 below)

> **Don't bail early.** Do NOT fall back to raw SQL on these grounds:
> - "[custom date filtering / cohorts]" → [covered by time-dimension specs]
> - "[needs a join]" → [the metric layer already encapsulates its joins]
> - [3–4 more pre-rebutted excuses agents use to skip the semantic layer]

### Date windows & timezone — decide before you query
- **As-of date vs trailing-N days**: [convention for each]
- **"Last week/month"** → the last *complete* calendar week/month, not trailing-7/30
- **Timezone default**: [TZ]; [exception for certain reporting rollups]
- **Freshness lag**: [some] tables settle late — anchor on MAX(date), not "yesterday"

---

# PART 1: MUST KNOW (Read First for Every Request)

## 🚀 Quick Start Workflow
1. **Check for red flags first**: [restricted/PII requests, gated domains,
   high-stakes asks that need extra validation]
2. **Out of scope — escalate, don't guess**: [access requests, pipeline
   troubleshooting, stale dashboards, root-cause assertions, product/pricing
   recommendations] → redirect to [the owning team], don't answer
3. **Clarify the request**: time period, segment, the business decision it informs
4. **Check for existing dashboards**: [per-domain dashboard catalogs]
5. **Identify the data source**: [navigation map below; prefer governed/aggregated tables]
6. **Execute the analysis**: [required filters + adversarial review]
7. **Deliver insights**: show methodology, differentiate observations from interpretations

## 🏢 Business Context

### Entity Disambiguation (MUST CLARIFY)
- **"[Term A]" can mean**: [entity 1] or [entity 2] — always clarify which
- **"[Term B]" can mean**: [entity 1] → [entity 2] → [entity 3] (one-to-many chain)
- **"Users"**: [which identifier gives accurate counts, and which ones inflate them]

### Business Terminology
- [Current product names vs deprecated aliases that still appear as frozen
  values in the data layer — write with the new names, filter with the old]
- [Key internal acronyms]
- **[Headline metric] calculations**: [monthly / default window / leading indicator]
- **Unfamiliar terms — search [internal docs], don't guess**

### Data Integrity Requirements ⚠️
- **NEVER**: make up data/columns; make speculative assertions beyond what data shows
- **ALWAYS**: use safe division; differentiate observations ("data shows X")
  from interpretations ("this suggests Y"); flag limitations

---

# PART 2: HOW TO DO (Follow During Execution)

## 🔧 Technical Execution Guide
- [Managed-connection tools and CLI invocation details]
- **PII protection**: for restricted data, return the SQL for the user to run
  themselves — do not return results

## 📊 Analysis Best Practices Guide
1. Clarify the ask before querying
2. Show your work (filters, inclusions/exclusions, freshness)
3. Clarify denominators
4. Consider sample bias
5. Connect to business impact
6. **Adversarial SQL review (MANDATORY)** — spawn the [sql-reviewer] sub-agent
   for every query before the final answer; blocking findings must be fixed
   and re-reviewed; do not self-certify
7. **Report with provenance** — every answer ends with a footer:
   > **Source:** [semantic layer | governed table | raw exploration] ·
   > **Confidence:** [tier] · **Reviewed:** [reviewer ✓, round N] ·
   > **Freshness:** [max date in the data] · **Owner:** [owning team]

---

# PART 3: DATA REFERENCES & RESOURCES

## 📚 Knowledge Base Navigation
### [Domain A] → `references/[domain_a].md`
- **Use for**: [kinds of questions]
- **Key tables**: [...]
- **Dashboards**: `references/[domain_a]_dashboards.json`

### [Domain B] → `references/[domain_b].md`
- **Use for**: [...]

[... one entry per business domain — a few dozen in total ...]

## ⚠️ Troubleshooting Guide

### When Information Is Missing
- [missing tables / access denied / outdated docs / unknown enum values → what to do]

### Field Naming Gotchas
- Use `[field_x_v2]` NOT `[field_x]`
- [Two similarly-named tables report the same metric at different grains — which to use]
- [Which of two plausible sources is canonical for the headline metric]
- [… a dozen more hard-won one-liners …]