Visibility is the easiest piece. The hard part is analyzing and understanding what you're observing. I've spoken to teams recording 100k+ traces every single day. What are they doing with those traces? Literally nothing. Because it's impossible to read and summarize 100,000 traces at any human scale.可见性是最容易实现的部分。难点在于分析并理解你所观察到的内容。我曾与一些团队交流，他们每天记录超过 10 万条追踪记录。他们用这些记录做了什么？实际上什么都没做。因为人类根本无法阅读和总结 10 万条追踪记录。
- Dev Shah- Dev Shah
Tracing is critical for agent development - it powers evaluation, debugging, and annotation. But when you have agents in production generating thousands of traces daily, manual review doesn't scale.追踪对于智能体开发至关重要——它为评估、调试和标注提供了支持。但是，当你的生产环境中的智能体每天生成数千条追踪记录时，人工审查就无法扩展了。
When building traditional software, we have product analytics to help with this problem. You can track agent metrics too—latency, error rates, feedback. But these metrics only surface problems. To understand the patterns behind them, you need to analyze unstructured conversations—not predefined event flows. Traditional product analytics wasn’t built for this. This is exactly why we built LangSmith Insights Agent.在构建传统软件时，我们有产品分析来帮助解决这个问题。你也可以追踪智能体指标——延迟、错误率、反馈。但这些指标只能浮现问题。要理解其背后的模式，你需要分析非结构化的对话，而不是预定义的事件流。传统的产品分析并非为此而生。这正是我们构建 LangSmith Insights Agent 的原因。
Why You Can’t Predict Agent Behavior为什么你无法预测智能体行为
Building agents is different than building software. There are three key differences from traditional software:构建智能体与构建软件不同。它与传统软件有三个关键区别：
Non-determinism. When you run software multiple times, you get the same result. This doesn’t happen with agents. Each call to an LLM may produce different results. When you let an agent run for a while and make hundreds of calls in a row, it is highly likely that the same input may produce very different paths.非确定性。当你多次运行软件时，你会得到相同的结果。智能体则不然。每次调用大语言模型（LLM）都可能产生不同的结果。当你让智能体运行一段时间并连续进行数百次调用时，相同的输入极有可能产生截然不同的路径。
Prompt sensitivity. Software doesn’t change dramatically with small input changes. It is robust to small changes in user input. LLMs have a characteristic called “prompt sensitivity”. This means small changes in the prompt space can produce large changes in output.提示词敏感性。软件不会因为输入的小幅变化而发生剧烈改变。它对用户输入的小幅变化具有鲁棒性。大语言模型具有一种称为“提示词敏感性”的特性。这意味着提示词空间中的微小变化可能会导致输出的巨大变化。
Unbounded input space. Software structures user input through UIs. The input space is naturally bounded and scoped by what is in the UI. Agents accept natural language, which is unbounded. Users can enter anything无界输入空间。软件通过用户界面（UI）来构建用户输入。输入空间自然地受到 UI 中内容的限制和约束。智能体接受自然语言，这是无界的。用户可以输入任何内容。
| Software | Agents | |
|---|---|---|
| Deterministic | Yes | No |
| Robust to small changes in user input | Yes | No |
| Bounded input space | Yes | No |
💡 Because agents are non-deterministic and accept unbounded input, you can’t predict what they’ll do or how users will use them until production.💡 由于智能体具有非确定性且接受无界输入，在进入生产环境之前，你无法预测它们会做什么，也无法预测用户将如何使用它们。
You Need Production Analytics你需要生产环境分析
Software is deterministic, robust to small changes in user input, and bounded in input space. As a result, you can be pretty sure software is behaving as you expect before you launch to production, and have a suite of tests to assert that. You can discover user behavior by observing production events, but this is constrained by the actions they can take in the UI (this is traditional product analytics).软件是确定性的，对用户输入的小幅变化具有鲁棒性，且输入空间有限。因此，在发布到生产环境之前，你可以相当确定软件的行为符合预期，并拥有一套测试来验证这一点。你可以通过观察生产环境事件来发现用户行为，但这受到他们在 UI 中可执行操作的限制（这就是传统的产品分析）。
Agents are non-deterministic and sensitive to small changes in user input. As a result - you don’t know what agents will actually do until production. Agents also usually have a natural language interface, giving them a completely unbounded input space. This means there is way more user intent you can capture and analyze.智能体是非确定性的，且对用户输入的小幅变化敏感。因此，在进入生产环境之前，你不知道智能体实际上会做什么。智能体通常还具有自然语言界面，这赋予了它们完全无界的输入空间。这意味着你可以捕获和分析更多的用户意图。
💡 When building agents, you need to iterate on production data much more than when building traditional software.💡 在构建智能体时，你需要比构建传统软件更频繁地根据生产数据进行迭代。
You need to iterate on how agents fail. With software, you catch correctness issues pre-production with tests. Maybe some edge cases slip through. With agents, it’s the opposite - most failures emerge in production.你需要针对智能体的失败方式进行迭代。对于软件，你可以在生产前通过测试捕获正确性问题。也许会有一些边缘情况漏掉。对于智能体，情况恰恰相反——大多数失败都出现在生产环境中。
You need to iterate on how users actually use agents. With software, users can only use the product as you let them. With agents, natural language means they can use it in far more ways.你需要针对用户实际使用智能体的方式进行迭代。对于软件，用户只能按照你允许的方式使用产品。对于智能体，自然语言意味着他们可以以更多样的方式使用它。
Traditional product analytics are necessary but not sufficient for agent iteration传统产品分析对于智能体迭代是必要的，但还不够
Traditional software generates tons of events (clicks, page views, sessions). Product analytics tools (Mixpanel, Amplitude, etc.) solve the "too much data" problem by:传统软件会生成大量事件（点击、页面浏览、会话）。产品分析工具（如 Mixpanel、Amplitude 等）通过以下方式解决“数据过多”的问题：
- Aggregating discrete events into metrics将离散事件聚合为指标
- Building funnels and cohorts构建漏斗和群组
- A/B testingA/B 测试
You can (and should) do this with agents as well. We’ve found that most companies in production track end user feedback, latency and tool calls.你也可以（并且应该）对智能体执行此操作。我们发现大多数生产环境中的公司都会追踪最终用户反馈、延迟和工具调用。
These metrics help you understand what is going on, but not why. In order to understand why these metrics are moving, you need to analyze the traces themselves.这些指标有助于你了解发生了什么，但无法解释原因。为了理解这些指标波动的原因，你需要分析追踪记录本身。
Online evals help with known questions. You can run evaluators over traces to score for specific things - user frustration, topic tags, success criteria. But online evals require you to know what you’re looking for upfront.在线评估有助于解决已知问题。你可以对追踪记录运行评估器，以针对特定事项进行评分——例如用户挫败感、主题标签、成功标准。但在线评估要求你预先知道自己在寻找什么。
What about exploratory questions? ‘How are users actually using my agent?’ ‘What failure patterns exist?’ You can’t write an evaluator for patterns you haven’t discovered yet.那么探索性问题呢？“用户实际上是如何使用我的智能体的？”“存在哪些失败模式？”你无法为你尚未发现的模式编写评估器。
So what does agent analytics actually look like? You need a tool that:那么智能体分析到底是什么样的？你需要一个能够实现以下功能的工具：
- Analyzes unstructured conversations, not discrete events分析非结构化对话，而非离散事件
- Discovers patterns you didn’t know to look for发现你未曾想到要寻找的模式
- Surfaces clusters at scale大规模浮现聚类
This is LangSmith Insights Agent.这就是 LangSmith Insights Agent。
LangSmith Insights AgentLangSmith Insights Agent
LangSmith Insights Agent uses clustering to automatically discover patterns in your traces. Instead of defining what to look for upfront, it analyzes thousands of conversations and surfaces the clusters that matter - usage patterns, error modes, or any dimension you specify. It handles the exploratory questions that would otherwise require manually reviewing hundreds of traces.LangSmith Insights Agent 使用聚类技术自动发现追踪记录中的模式。它无需预先定义要寻找的内容，而是分析数千次对话并浮现重要的聚类——例如使用模式、错误模式或你指定的任何维度。它处理了那些原本需要手动审查数百条追踪记录才能解决的探索性问题。
Insights Agent will look for patterns and produce a report. The report will be based around a clustered analysis of the runs. There will be multiple different hierarchies of clusters. There will be top level clusters, then a second level of more detailed groupings, and then individual runs beneath that. You can click through those levels to explore the data at whatever level you think best. This lets you zoom in on specific issues or zoom out for high-level patternsInsights Agent 将寻找模式并生成报告。该报告将基于运行记录的聚类分析。会有多个不同的聚类层级。会有顶层聚类，然后是更详细的二级分组，最后是底层的单个运行记录。你可以点击这些层级，以你认为最合适的层级探索数据。这使你可以放大查看特定问题，或缩小查看高层级模式。
You can configure Insights Agent to look for whatever patterns you want. The two most common patterns we see people looking for are:你可以配置 Insights Agent 来寻找你想要的任何模式。我们看到人们最常寻找的两种模式是：
- How are user's using my agent?用户是如何使用我的智能体的？
- How might my agent be failing?我的智能体可能在哪些方面失败？
We have prebuilt prompts for these two use cases, but if you want to configure it to look for something else you can specify an arbitrary prompt. Because every agent is different - you might care about compliance, tone, accuracy, or domain-specific failures我们针对这两种用例提供了预构建的提示词，但如果你想配置它来寻找其他内容，也可以指定任意提示词。因为每个智能体都不同——你可能关心合规性、语气、准确性或特定领域的失败。
Sometimes you may only want to run Insights Agent over a specific subset of runs. This lets you combine Insights Agent with online evals or traditional product analytics. For example, you may only want to investigate runs that had negative end user feedback (e.g a thumbs down). You can do this by specifying a filtered set of runs to run Insights Agent over. Combine quantitative signals (thumbs down) with qualitative analysis (what patterns exist in those thumbs down).有时你可能只想对特定子集的运行记录运行 Insights Agent。这使你可以将 Insights Agent 与在线评估或传统产品分析相结合。例如，你可能只想调查那些收到负面最终用户反馈（例如点踩）的运行记录。你可以通过指定一组过滤后的运行记录来运行 Insights Agent。将定量信号（点踩）与定性分析（这些点踩中存在什么模式）相结合。
You can also filter by attributes that don’t exist yet. Example: ‘Why are users frustrated?’ LangSmith Insights Agent can calculate ‘user is frustrated’ on the fly, then filter and cluster based on it.你还可以按尚不存在的属性进行过滤。例如：“为什么用户感到沮丧？”LangSmith Insights Agent 可以即时计算“用户感到沮丧”，然后基于此进行过滤和聚类。
When clusters are produced, it will then aggregate attributes associated with the traces in those groups so you can quickly spot outlier clusters. You can also specify attributes to calculate on the fly (that are then also included in the aggregated stats). Discover patterns you didn’t know to track.当生成聚类时，它会聚合这些组中与追踪记录相关的属性，以便你快速发现异常聚类。你还可以指定即时计算的属性（这些属性也会包含在聚合统计中）。发现你未曾想到要追踪的模式。
Try LangSmith Insights Agent试用 LangSmith Insights Agent
Agents generate thousands of unstructured traces daily. Traditional metrics tell you that something changed, but not what or why. Manual review doesn’t scale.智能体每天生成数千条非结构化追踪记录。传统指标告诉你发生了变化，但没告诉你是什么或为什么。人工审查无法扩展。
LangSmith Insights Agent makes pattern discovery practical - surfacing usage patterns and failure modes automatically. It’s what makes iterating on production agent data possible.LangSmith Insights Agent 使模式发现变得切实可行——自动浮现使用模式和失败模式。这使得对生产环境中的智能体数据进行迭代成为可能。
Try it today in LangSmith today.立即在 LangSmith 中试用。
