AI Agents to Make Sense of Data at OpenAIOpenAI中用于理解数据的AI代理

Bonnie Xu: Today I'll be talking about how OpenAI deployed AI agents to help our teams answer their data questions. Let me first paint you a picture. Your business lead comes to you and asks the question, how many ChatGPT pro users do we have in Italy? You consult a data scientist, but they're actually like, "This is hard. Let me get back to you". They don't know what table to look at, so they ask another engineer, but then the other engineer doesn't know. After three code deep dives, two quick meetings, and five Slack threads, we finally have an answer. Simple questions shouldn't be this difficult and this time consuming, but they are. The reason why this is hard is because there is so much data.Bonnie Xu：今天我将讨论OpenAI如何部署AI代理来帮助我们的团队回答数据问题。首先让我描绘一个场景。你的业务负责人来找你问：意大利有多少ChatGPT Pro用户？你咨询数据科学家，但他们可能会说：“这很难，我稍后回复你。”他们不知道要看哪个表，于是问另一个工程师，但那个工程师也不知道。经过三次代码深入分析、两次快速会议和五个Slack线程后，我们终于得到了答案。简单的问题不应该这么困难和耗时，但事实确实如此。之所以困难，是因为数据量太大了。

I'm Bonnie Xu. I'm on the data productivity team at OpenAI. I'm here to talk to you today about how we solved this problem. Here are some key takeaways I hope you get from my talk. Firstly, the importance of the right data context. Then, I'll be talking about how important memory is for self-learning, and then how important evals are for making sure that the model doesn't regress.我是Bonnie Xu，来自OpenAI的数据生产力团队。今天我来谈谈我们如何解决这个问题。以下是我希望你们从我的演讲中获得的一些关键要点。首先，正确的数据上下文的重要性。然后，我将讨论记忆对于自我学习的重要性，以及评估对于确保模型不退化的重要性。

Let me kick off with an overview of data platform to illustrate why we need an AI agent in the first place, then I'll go into implementation specifics, and then learnings, and next steps that we have. At OpenAI, 80% of the company directly uses our data platform. That's 80% of the company using our team's 15 tools to process over 600 petabytes of data a day across 70k total datasets. The data is just growing even more rapidly. That means we have so many more questions to answer, but there's so much more data now to sift through to get the right result. When ChatGPT launched in 2022, we were asking ourselves, how many users do we have? As the product has evolved, we've introduced more regions, different plans, more features.让我先概述数据平台，以说明为什么我们首先需要一个AI代理，然后我会深入实现细节，接着是经验教训和下一步计划。在OpenAI，80%的员工直接使用我们的数据平台。也就是说，80%的员工使用我们团队的15个工具，每天处理超过600PB的数据，涉及7万个数据集。数据还在快速增长。这意味着我们有更多的问题需要回答，但需要筛选的数据也更多了。当ChatGPT在2022年推出时，我们问自己：我们有多少用户？随着产品的发展，我们引入了更多地区、不同计划和更多功能。

We're now asking ourselves the question, how many daily active instant checkout users do we have in New York? That's a much harder question to answer now, but fundamentally, we're looking for the same type of answer. One of the reasons why this is hard is because table discovery becomes a lot harder at scale. Helene is a data scientist at OpenAI, and this is what she asked on Slack a few months ago. She's having difficulty finding the right table to use because there's a lot of similarly sounding tables, and it's unclear what data is in them. Here's Eric, another data scientist at OpenAI, struggling to make sense of the nuances of each table. This is a hard problem because some tables have encrypted IDs, some tables have unencrypted IDs, but we still might want to join them. Some tables have columns that adjust for fraud rates.现在我们问自己：纽约有多少每日活跃的即时结账用户？这个问题现在更难回答了，但本质上，我们在寻找相同类型的答案。困难的原因之一是，在规模扩大时，表发现变得非常困难。Helene是OpenAI的数据科学家，这是她几个月前在Slack上问的问题。她很难找到正确的表，因为有很多听起来相似的表，而且不清楚表中有什么数据。这是另一位数据科学家Eric，他正在努力理解每个表的细微差别。这是一个难题，因为有些表有加密ID，有些表有未加密ID，但我们可能仍然需要连接它们。有些表的列调整了欺诈率。

Some tables don't. Some tables are pre-filtered by feedback. Some are not. Missing one nuance can lead to an answer that is wrong by an order of magnitude, and this can be catastrophic when making important business decisions. Not to mention, writing SQL is hard. This SQL statement is 160 lines of code, and I have no idea if it's wrong or not. Because who can remember all the different ways we date format or writing performant queries or the aggravating fact that Trino arrays are one indexed.有些表没有。有些表按反馈预过滤，有些没有。忽略一个细微差别可能导致答案相差一个数量级，这在做出重要业务决策时可能是灾难性的。更不用说编写SQL很难了。这个SQL语句有160行代码，我不知道它是否正确。因为谁能记住所有不同的日期格式、编写高性能查询的方法，或者Trino数组从1开始索引这个恼人的事实呢？

What's a better way of doing this? At OpenAI, we built Kepler, an AI data analyst that takes the full context of data platform and answer these data questions for you. At its core, the Kepler service leverages the model to produce AI-powered results when you need them. This could be on Slack. For example, on the left here, we have a Slack agent to ping. This could be in your IDE, like Cursor, you hook it up, our MCP server. Maybe you are asking our web agent for table information, or you can connect Kepler to MCP platforms for workloads, for example. Let's go through an example. Let's see Kepler in action. Let's say I want to find for New York taxi trips what pickup, dropoff ZIP pairs are the most unreliable, so biggest spread between typical and worst-case duration and when that happens.有没有更好的方法？在OpenAI，我们构建了Kepler，一个AI数据分析师，它利用数据平台的完整上下文为你回答数据问题。其核心是，Kepler服务利用模型在你需要时生成AI驱动的结果。这可以在Slack上。例如，左边是一个Slack代理，你可以@它。这可以在你的IDE中，比如Cursor，你可以连接我们的MCP服务器。也许你通过我们的Web代理询问表信息，或者你可以将Kepler连接到MCP平台以处理工作负载。让我们看一个例子，看看Kepler的实际应用。假设我想找出纽约出租车行程中，哪些上下车邮政编码对最不可靠，即典型时长和最差时长之间的差距最大，以及这种情况何时发生。

You can see right now that Kepler first does an internal knowledge search. It's looking at, let's get some initial information and let's see what's out there. This is this chain of thought that you're seeing being streamed right now. Here we have the table schema, so Kepler knows how to write the query, and it's running through the query right now. Basically, it's writing all these different queries to try to get the right data. Bucketing is important here. We're looking at percentiles to determine the worst durations. You can see from the SQL, ok, we got back some results. We have to adjust the thresholds because those results weren't what we're looking for. Running some more queries, some sorting. You can imagine doing this manually yourself takes a lot of time.你现在可以看到Kepler首先进行内部知识搜索。它正在获取一些初始信息，看看有什么可用的。这是你正在看到的思维链。这里我们有表模式，所以Kepler知道如何编写查询，并且它正在运行查询。基本上，它正在编写所有这些不同的查询来获取正确的数据。分桶在这里很重要。我们正在查看百分位数以确定最差的时长。从SQL中你可以看到，我们得到了一些结果。我们必须调整阈值，因为这些结果不是我们想要的。运行更多查询，进行一些排序。你可以想象手动做这件事需要很多时间。

The agent is just going through these query and result set steps for you and on your behalf. Again, running another query. We have to adjust, so on and so forth. Some sorting going on here. There's a lot of steps in the analysis. Our agent tries to do a thorough job. If you read the SQL, you can tell that there's some pickup. Ratios, I think, are used for the duration. This is the CTE involved. Now there's the analysis part. We're satisfied with the results that we want and we need to produce the actual thing that we're going to show the user. That's the summary here that you see on the screen. Some light formatting, because we want to make sure that it looks pretty. That's always important in the analysis. You can see here we are looking at timestamp. Finally, we get the right answer.代理正在为你代劳，逐步执行这些查询和结果集步骤。再次运行另一个查询。我们必须调整，等等。这里进行了一些排序。分析中有很多步骤。我们的代理试图做得彻底。如果你阅读SQL，你可以看到有一些上下车。我认为比率用于时长。这是涉及的CTE。现在是分析部分。我们对想要的结果感到满意，需要生成实际要展示给用户的内容。这就是你在屏幕上看到的摘要。进行一些简单的格式化，因为我们希望它看起来漂亮。这在分析中总是很重要的。你可以看到我们正在查看时间戳。最后，我们得到了正确的答案。

We measure typical by p50 and reliability by the p95, p50 ratio. We get the right dataset and the amount in numbers. This was an example dataset from 2016 that we had available in our data catalog. The results are in New York time. Here's all the results, so that ZIP code and then those are the durations. This is the SQL that actually was used. As you can see, it's pretty big. Then we have a link to the raw results. Here's the final answer. Morning commutes, weekdays, at rush hour, and late nights are the most unreliable, for anyone taking a New York taxi this week. Let's say I also want to do something slightly different. Let's say I want to plot a graph representing these results. It's a little more, I want something visual to better understand. Kepler is also able to do that too.我们通过p50衡量典型值，通过p95/p50比率衡量可靠性。我们得到了正确的数据集和数字。这是2016年的示例数据集，在我们的数据目录中可用。结果是纽约时间。这是所有结果，包括邮政编码和时长。这是实际使用的SQL。如你所见，它相当大。然后我们有一个指向原始结果的链接。这是最终答案。对于本周乘坐纽约出租车的人来说，早高峰通勤、工作日、高峰时段和深夜是最不可靠的。假设我还想做点稍微不同的事情。假设我想绘制一个图表来表示这些结果。我想要一些视觉化的东西来更好地理解。Kepler也能做到这一点。

You see here it's figuring it out. It was able to come up with something. Here is the right query that it used to generate the chart result. If I scroll down here, then you can actually see the analysis on the left. It's the ZIP codes on the right. It's the unreliability.你看到它正在解决。它能够想出一些东西。这是它用来生成图表结果的正确查询。如果我向下滚动，你可以看到左边的分析。右边是邮政编码，这是不可靠性。

That was a toy example using a dataset. Let me give something maybe some of you folks might relate to more, debugging an anomaly. In this example, we're looking at what caused a big upward spike in ChatGPT user growth for weekly active users late March. I'm just going to be sharing screenshots of the chain of thought here. The first part is looking at the right table to check the spike. If there actually was a spike, we should see the numbers before spike and after spike increase. Another part of the chain of thought is actually delving into the data. How do we know that that table was correct? Here you can see Kepler is referencing a dashboard and a Notion document just to confirm that. Also, Kepler interactively delves into the data. It does this by actually just running different queries to slice it up here.那是一个使用数据集的玩具示例。让我举一个你们中一些人可能更相关的例子：调试异常。在这个例子中，我们正在查看是什么导致了3月下旬ChatGPT周活跃用户增长的大幅上升。我将分享思维链的截图。第一部分是查看正确的表以检查峰值。如果确实存在峰值，我们应该看到峰值前后的数字增加。思维链的另一部分实际上是深入研究数据。我们如何知道那个表是正确的？这里你可以看到Kepler正在引用一个仪表盘和一个Notion文档来确认。此外，Kepler交互式地深入研究数据。它通过实际运行不同的查询来切片数据。

That's what you see by dimensions. Kepler is running queries based on plan type, based on region. This is how it can see what motivated the spike. Like, was it a specific region that increased and that's what caused it? What a human might do for data analysis. Kepler also tries to come up with reasons now that it has all the data. An example hypothesis is, we logged too much. We have duplication issues. Kepler is looking around at internal company context to check that this is the case. Fortunately for our use case, it is not. Kepler was able to figure out that it was related to ImageGen that we launched. Now that Kepler is at this hypothesis, it actually does a web search to check for the timeline to cross reference itself.这就是你按维度看到的内容。Kepler根据计划类型、地区运行查询。这就是它如何看到是什么推动了峰值。比如，是否某个特定地区增加了，从而导致了峰值？人类进行数据分析时会做的事情。Kepler在拥有所有数据后也试图找出原因。一个示例假设是：我们记录太多了。我们有重复问题。Kepler正在查看内部公司上下文以检查是否如此。幸运的是，对于我们的用例，情况并非如此。Kepler能够找出这与我们推出的ImageGen有关。现在Kepler有了这个假设，它实际上进行网络搜索以检查时间线，进行交叉验证。

Here in this chain of thought, you can see Kepler references the release notes at around that timeframe and also a TechCrunch article because that's how it can really check the trends as well. Finally, we arrive at the result. Kepler was able to figure out that it was due to the ImageGen trend. It worked. I always get nervous whenever Kepler shows an answer. I'm like, is this right or not? Fortunately, it's right most of the time. I'll be talking about that as well.在这个思维链中，你可以看到Kepler引用了那个时间段的发布说明和一篇TechCrunch文章，因为它可以真正检查趋势。最后，我们得到了结果。Kepler能够找出这是由于ImageGen趋势。它成功了。每当Kepler显示答案时，我总是很紧张。我想，这是正确的吗？幸运的是，大多数时候它是正确的。我也会谈到这一点。

Let's talk about some other things that Kepler can do. Don't feel bad about asking Kepler questions. Kepler is available 24/7 for a quick conversation. You can also do follow-ups. That's how you get a thread with 56 replies like the one Dominique is having. The really lovely thing is that all follow-ups, Kepler just stores the whole context, so you don't need to repeat yourself. If you see Kepler veering off track because we stream up the chain of thoughts, you can see where Kepler is going, you can interrupt Kepler and Kepler will take that feedback and produce a different result. Here's a video example of what I mean by follow-ups. Here, similar thing. I'm asking about just the New York taxi scene because I'm really interested in that. I was asking about the results in the pickup. We have this graph.让我们谈谈Kepler可以做的其他事情。不要不好意思向Kepler提问。Kepler全天候可用，可以进行快速对话。你也可以进行后续提问。这就是为什么你会看到像Dominique那样有56条回复的线程。非常棒的是，所有后续问题，Kepler都会存储整个上下文，所以你不需要重复自己。如果你看到Kepler偏离轨道，因为我们流式传输思维链，你可以看到Kepler的进展，你可以打断Kepler，Kepler会接受反馈并产生不同的结果。这是一个视频示例，展示了我所说的后续提问。这里类似，我在询问纽约出租车的情况，因为我对此很感兴趣。我在询问关于上车点的结果。我们有这个图表。

Now let's say I wanted to dig into something a little more specific. I want to look specifically at pickup trends on February 14th, a very arbitrary day. I'm asking Kepler here. You can see Kepler is reasoning. This is our Slack agent interface that you're seeing here. There's a little bit of running queries. You can tell that, or hopefully you caught that. Kepler didn't need to do all the initial knowledge search again because we have the context from the previous response about the right table to use. Kepler was able to just immediately run the query for that answer and get the results that there were 375 total pickups in this dataset. Also, Kepler can handle follow-ups. If I ask a very incomplete question, Kepler just picks the best default if I don't follow up.现在假设我想深入一些更具体的内容。我想特别查看2月14日这个非常随意日期的上车趋势。我正在向Kepler提问。你可以看到Kepler正在推理。这是你看到的我们的Slack代理界面。有一些查询正在运行。你可以看出，或者希望你已经注意到，Kepler不需要再次进行所有初始知识搜索，因为我们有之前关于正确表的上下文。Kepler能够立即运行查询来获取答案，并得到结果：这个数据集中共有375次上车。此外，Kepler可以处理后续问题。如果我问一个非常不完整的问题，如果我不跟进，Kepler会选择最佳默认值。

The other thing that's nice and that we found useful, at least for our purposes, is there's a lot of commonly repeatable processes that people do. Things like feature product analysis or data validation on dev tables versus prod tables. Basically, we have these workflows which are like custom shareable instructions that you can just instantly rerun in the UI.另一件很好的事情，至少对我们的目的来说很有用，是存在许多常见的可重复流程。比如功能产品分析或开发表与生产表的数据验证。基本上，我们有这些工作流，它们就像自定义的可共享指令，你可以在UI中立即重新运行。

That was a lot of demoing and showing you guys what it might look like if you were using Kepler at the company. Let me now talk about how this all actually happens behind the scenes. This is a big picture diagram of how things happen. On the left here, we have the entry points. You've seen a couple of them. There's the UI. There's also the Slack agents. Then, you can hook up local or remote MCP. There's the top part which is the preprocessed offline information which I'll get to. This is the knowledgebases. There's the bottom part for sync calls, so I'm directly making API calls to our data warehouse or to other data platform sources like Spark or Airflow. This is really the core of it all. It's Kepler talking to the model armed with this toolbox.刚才演示了很多，向你们展示了在OpenAI使用Kepler会是什么样子。现在让我谈谈这一切在幕后是如何发生的。这是一个大图，展示了事情是如何发生的。左边是入口点。你们已经看到了一些。有UI。还有Slack代理。然后，你可以连接本地或远程MCP。顶部是预处理的离线信息，我稍后会讲到。这是知识库。底部是同步调用，我直接向数据仓库或其他数据平台源（如Spark或Airflow）发出API调用。这实际上是核心。Kepler与模型对话，并配备了这些工具。

I'm just going to take a step back here and explain why using MCP has been so helpful for us. The reason is because with all these tools, Kepler can use them to start off with. Then after realizing that something is wrong. For example, if Kepler just picks two tables to join but uses the wrong key, you'll get no results. Kepler can go back and repeat the steps. We internal knowledge search. We check the schemas, run the query. We got to redo it again. You can tell a little bit from my previous example where Kepler was just constantly running queries looking at results. Basically, this is the agent reasoning by itself. Instead of you giving the feedback, Kepler is running tools, giving feedback, then using the right tools to take the next potential steps depending on whatever feedback that's given.我先退一步，解释一下为什么使用MCP对我们如此有帮助。原因是，有了所有这些工具，Kepler可以一开始就使用它们。然后，当意识到某些东西出错时，例如，如果Kepler选择连接两个表但使用了错误的键，你会得到空结果。Kepler可以返回并重复步骤。我们进行内部知识搜索。我们检查模式，运行查询。我们必须重新做一遍。从之前的例子中你可以看出，Kepler不断运行查询并查看结果。基本上，这是代理自己推理。而不是你给出反馈，Kepler运行工具，给出反馈，然后根据给出的反馈使用正确的工具采取下一步可能的行动。

The really lovely thing is that Kepler can interactively explore the data itself and context is carried over the whole time. We have the whole steps which just make for a better answer at the end.非常棒的是，Kepler可以交互式地探索数据本身，并且上下文在整个过程中都会保留。我们有完整的步骤，最终得到更好的答案。

Agents without context can give wildly wrong answers. Take this example where somehow the agent thought that there were 5 million ChatGPT users compared to the actual 800 million answer. Just a minor rounding detail. Or in this case, when the agent thought that Sora was the Kingdom Hearts video game character. Maybe in my regular ChatGPT history, this would be more right. Yes, I really wanted to ask about Sora, the video gen product that we just released. As a start, we need table metadata context. As we saw earlier, fitting all 70k tables with their schemas and query history is just too much data to put into a model's context window. We have to do some preprocessing ahead of time. Table schema information is particularly important because that way the model knows how to actually query the table. That's where the SQL generation bits come from.没有上下文的代理可能会给出完全错误的答案。以这个例子为例，代理不知何故认为有500万ChatGPT用户，而实际答案是8亿。只是一个小的四舍五入细节。或者在这个例子中，代理认为Sora是《王国之心》电子游戏角色。也许在我的常规ChatGPT历史中，这会更正确。是的，我确实想问关于我们刚刚发布的视频生成产品Sora的问题。首先，我们需要表元数据上下文。正如我们之前看到的，将所有7万个表及其模式和查询历史放入模型的上下文窗口数据量太大。我们必须提前进行一些预处理。表模式信息尤其重要，因为这样模型才知道如何实际查询表。这就是SQL生成部分的来源。

It needs to know the columns and their types. Schemas alone aren't enough to understand the semantics and the relationships between data. That's why here you see query history, lineage. Those are important things to provide this extra context. All of this basically gets fed in into an embedding that we use the OpenAI API for, and then stored so that it can be live retrieved via specific table search and semantic search when the agent is actually answering questions. The other common problem with table metadata is that descriptions get easily outdated and often they become a burden to maintain. This leads to the agent getting bad results. It's incredibly tedious to manually update them, especially when you have 70k tables. We solve this by autogenerating as much as possible. We also include information beyond what is just in our data catalog.它需要知道列及其类型。仅凭模式不足以理解数据的语义和关系。这就是为什么这里你会看到查询历史、血缘关系。这些是提供额外上下文的重要内容。所有这些基本上都被输入到一个嵌入中，我们使用OpenAI API生成，然后存储起来，以便在代理实际回答问题时可实时检索，通过特定表搜索和语义搜索。表元数据的另一个常见问题是描述容易过时，并且维护起来往往成为负担。这导致代理得到糟糕的结果。手动更新它们非常繁琐，尤其是当你有7万个表时。我们通过尽可能自动生成来解决这个问题。我们还包含超出数据目录的信息。

That's actually what makes the generation so good. It's not enough to look at the table by itself just as is. You need to understand how the table was created and where it came from. This is the secret to the agent really understanding the differences between tables, knowing that a table was filtered down because it came from a subset of logs. We achieve this by essentially running an offline job that generates Codex tasks for tables. These Codex tasks are launched in parallel daily. They crawl the codebase to understand things like a table's purpose, downstream usage patterns, exact grain and primary keys, the freshness, when to use other tables, so on and so forth. Instead of knowing that a table is only about ChatGPT analytics, you know that the table contains first-party ChatGPT traffic and not third-party traffic, and that it's enriched by safety signals.这实际上是生成如此出色的原因。仅仅查看表本身是不够的。你需要了解表是如何创建的以及来自哪里。这是代理真正理解表之间差异的关键，知道一个表被过滤了，因为它来自日志的子集。我们通过运行一个离线作业来实现这一点，该作业为表生成Codex任务。这些Codex任务每天并行启动。它们爬取代码库以了解表的目的、下游使用模式、精确粒度和主键、新鲜度、何时使用其他表等等。而不是只知道一个表只关于ChatGPT分析，你知道该表包含第一方ChatGPT流量而不是第三方流量，并且它由安全信号丰富。

Or you know that some fields might be actually null because their upstream signals are missing or outside the hourly window. Here's an example Codex generation that gives information on a Spark observability table. As you can see, Codex easily goes through the agent files to crawl the codebase. It looks at the Airflow folder, looks at the projects. That's how we generate some tables. Gets this information, maybe it's job information, table info, stage usage, so on and so forth. Again, since this is all refreshed periodically by an offline job, the context stays fresh without any manual involvement. The really lovely thing about this is that you can also get lineage information, which is super useful in knowing how the tables relate. This is an example that I just pulled from one of our sample datasets.或者你知道某些字段可能为空，因为它们的上游信号缺失或超出小时窗口。这是一个Codex生成的示例，提供了关于Spark可观测性表的信息。如你所见，Codex轻松地通过代理文件爬取代码库。它查看Airflow文件夹，查看项目。这就是我们生成一些表的方式。获取这些信息，可能是作业信息、表信息、阶段使用情况等等。同样，由于这是由离线作业定期刷新的，上下文保持新鲜，无需手动参与。非常棒的是，你还可以获得血缘信息，这对于了解表之间的关系非常有用。这是我刚刚从我们的一个示例数据集中提取的例子。

As you can see here, it's a little richer than maybe what you might have if a human were just midnight writing this. This is all fed into the agent and also in our UI, so humans can also take advantage of it.如你所见，它比人类可能手动编写的内容更丰富。所有这些都被输入到代理中，也在我们的UI中，因此人类也可以利用它。

How do we get the company context? We actually have an internal knowledge service at OpenAI that ingests things like Slack threads, or Notion docs, or Google Drive docs. All of these go into blob storage with metadata so we can have the content and the source that it came from. Since these documents are quite large, they're entire docs sometimes, they're broken down into chunks and again embedded using the OpenAI embedding API. Then there's retrieval service that actually does similarly RAG search and does permissions checking and also caches so that we can pull these efficiently. That's why when we ask a question, we get the why and the context around just the what.我们如何获取公司上下文？实际上，我们在OpenAI有一个内部知识服务，它摄取Slack线程、Notion文档或Google Drive文档等内容。所有这些都进入带有元数据的blob存储，这样我们就可以拥有内容和来源。由于这些文档有时很大，它们被分成块，并再次使用OpenAI嵌入API进行嵌入。然后有一个检索服务，它进行类似的RAG搜索，进行权限检查，并缓存，以便我们可以高效地提取它们。这就是为什么当我们问一个问题时，我们得到的是“为什么”和“什么”周围的上下文。

If you see a dip in weekly active users, you might find that Slack thread that points to an incident or an outage, and that gives you a much richer analysis and understanding of the problem. Memory is useful for things like corrections and learnings. Here's an example. Before, we just had a Slack statement, we're looking at user IDs since we're looking at daily active users. Let's say that for this particular use case, we really mean we want to look at users that's about at least a message and we want to exclude external, and we care about PST time zone. After the memory, our system will always generate this sort of query instead of this one because we have this correction saved. That way, the agent is basically able to produce the right results each time. Memory is ingested similar to table knowledge.如果你看到周活跃用户下降，你可能会找到指向事件或中断的Slack线程，这为你提供了更丰富的分析和问题理解。记忆对于纠正和学习很有用。这是一个例子。以前，我们只有一个Slack语句，我们正在查看用户ID，因为我们正在查看每日活跃用户。假设对于这个特定用例，我们真正想要查看的是至少发送一条消息的用户，并且我们希望排除外部用户，并且我们关心PST时区。在记忆之后，我们的系统将始终生成这种查询而不是这种查询，因为我们保存了这个纠正。这样，代理基本上能够每次产生正确的结果。记忆的摄取类似于表知识。

We have all these corrections that we make. A user can submit a correction manually or the agent can also do so. We put things in embedding and then retrieve at runtime. We're relevant. For us, memory is really the mechanism that helps the agent continuously learn and improve. Context will get you maybe 80%, 90% of the way there. Sometimes you need those final little corrections that are just really hard to just infer. Let's say you rolled out a feature and you have a particular string for your stats and gates, that's an example of memory that you need to find the right result, but it's really tricky to know that otherwise. Here's an example. We have three scopes right now. There are user level ones because users might want their own customizations. We also want to protect potentially private information.我们进行了所有这些纠正。用户可以手动提交纠正，代理也可以这样做。我们将内容放入嵌入中，并在运行时检索。我们相关。对我们来说，记忆是帮助代理持续学习和改进的机制。上下文可能让你达到80%或90%。有时你需要那些最终的小纠正，这些纠正很难推断。假设你推出了一项功能，并且你的统计和门控有一个特定的字符串，这就是一个需要记忆才能找到正确结果的例子，否则很难知道。这是一个例子。目前我们有三个范围。有用户级别的，因为用户可能想要自己的定制。我们也想保护潜在的私人信息。

There's channel level for team scoped memories, and then also global memories for just general fixes that can benefit everyone. We're currently rolling out memory suggestions, so Kepler can prompt to generate a memory and then the user can confirm. Then it can be inserted in the right scope. We're also looking at ways we can create evals that only pass in the right memories just to make sure that all this is working correctly. The plan is to also have memory be compacted in case users generate a lot of the same memory. Also, sometimes there might be memories generated that are accidental or maybe they're just not that reusable. In that case, we want to prune them in an offline job. The other thing to make the memories richer and just to have a lot better signal is having them be edited.有团队范围的频道级别记忆，以及全局记忆，用于可以惠及所有人的通用修复。我们目前正在推出记忆建议，因此Kepler可以提示生成记忆，然后用户确认。然后它可以插入到正确的范围。我们也在寻找创建评估的方法，这些评估只传入正确的记忆，以确保一切正常工作。计划还包括在用户生成大量相同记忆时压缩记忆。此外，有时可能会生成偶然的记忆，或者它们可能不太可重用。在这种情况下，我们希望在离线作业中修剪它们。另一件让记忆更丰富、信号更好的事情是让它们可编辑。

In the UI right now, you can actually trigger an edit and then a resync if you want to make an update. Our users are really live helping contribute as well here. If all of the context isn't enough, our agent can make live calls to our data warehouse to find out what it needs. It's just a short API call away. One of the reasons, for example, why you might need this is if a table is new or something, like you generated the testing table, so it doesn't really exist in any of our services yet, or it doesn't exist in the offline job yet, you can just directly query the data warehouse and Kepler will get that information. Hopefully, this image really illustrates how important context is.在UI中，你现在可以触发编辑，然后重新同步，如果你想进行更新。我们的用户也在积极贡献。如果所有上下文都不够，我们的代理可以实时调用数据仓库来找出需要的内容。这只是一个简短的API调用。例如，你可能需要这个的原因之一是，如果表是新的，比如你生成了测试表，它还不存在于我们的任何服务中，或者离线作业中还没有，你可以直接查询数据仓库，Kepler将获取该信息。希望这张图片能真正说明上下文的重要性。

The next important factor to consider is how we ensure we don't cause regressions. Let me now talk about how we measure response quality. In the words of the wise Greg Brockman, evals are surprisingly often all you need. There's a lot of truth to that in our case. Our evals consist of sets of question-answer pairs. A question is usually some important metric we want to get right. Then we have a manually curated expected SQL statement that we want to be the correct answer to match. We hit our agent query generation endpoint to turn our natural language question to generate SQL and then run the query. We do the same with the expected query results. We have the generated query, the generated query results, expected SQL, expected SQL results, and we feed that all in to the OpenAI evals grader.下一个重要的考虑因素是我们如何确保不引起回归。现在让我谈谈我们如何衡量响应质量。用睿智的Greg Brockman的话来说，评估往往出乎意料地是你所需要的全部。在我们的案例中，这很有道理。我们的评估由一组问答对组成。一个问题通常是我们想要正确的重要指标。然后我们有一个手动策划的预期SQL语句，我们希望它是正确的答案。我们调用我们的代理查询生成端点，将自然语言问题转换为SQL并运行查询。我们对预期查询结果也做同样的事情。我们有生成的查询、生成的查询结果、预期SQL、预期SQL结果，然后将所有这些输入到OpenAI评估评分器中。

This evals grader is actually doing a model grading. This is particularly important because a lot of times generated SQL might differ by a little bit, but it still doesn't meaningfully change the results. All of this ends up giving us a score and a reason, so we can see how our evals did. Here are some key takeaways from our evals process. Firstly, I just want to say that exact SQL text equality isn't really a good representation, whether a SQL eval passed. Like you could write a date filter in multiple different ways and meaning is still the same. We normalize things. We convert things, everything into its AST representation. This helps us get around these minor SQL syntax different things. Also, when we compare result sets, we actually give a little wiggle room for things that don't meaningfully impact the answer.这个评估评分器实际上是在进行模型评分。这一点特别重要，因为很多时候生成的SQL可能略有不同，但不会显著改变结果。所有这些最终给我们一个分数和原因，这样我们就可以看到我们的评估表现如何。以下是我们评估过程中的一些关键要点。首先，我想说精确的SQL文本相等并不是SQL评估是否通过的良好表示。比如，你可以用多种不同的方式编写日期过滤器，但含义仍然相同。我们进行归一化。我们将所有内容转换为其AST表示。这有助于我们绕过这些小的SQL语法差异。此外，当我们比较结果集时，我们为那些不会对答案产生有意义影响的内容留出一点余地。

In some cases, a float or an int, it doesn't really matter. In some cases it does, but in some cases it doesn't. Again, the LLM reasoning is really good because it does a much better job where there's nuance. Like in the float in case, sometimes it actually is important for precision purposes, but sometimes it's not. The other thing is that because the model is pretty good at reasoning, it gives us a much more informative response from the results it's seen. The third thing is that we also expose the chain of thought in our evals. This is really helpful for debugging failures. In one case, we saw that an eval was failing because it was preferring a curated table over a raw table.在某些情况下，浮点数或整数并不重要。在某些情况下重要，但在某些情况下不重要。同样，LLM推理非常好，因为它能在有细微差别的地方做得更好。比如在浮点数的情况下，有时它确实对精度很重要，但有时不是。另一件事是，因为模型在推理方面相当不错，它从看到的结果中给了我们更有信息量的响应。第三件事是，我们还在评估中暴露了思维链。这对于调试失败非常有帮助。在一个案例中，我们看到一个评估失败，因为它更喜欢策划表而不是原始表。

After digging through the chain of thought, we realized it's because it thought that the question was more on the dashboarding side, and so that's why it picked that table. That wasn't immediately obvious in the beginning.在深入思维链后，我们意识到这是因为模型认为问题更偏向仪表盘方面，所以它选择了那个表。这在开始时并不明显。

With great data access comes great responsibility. Kepler does not provide any extra authorization. It actually does pass the authentication. This means that Kepler won't grant you extra access to tables you don't have access to. When you don't have access, Kepler will actually helpfully tell you which access group to join. Or it might use a similar table that you do have access to. Data security is also something that we take very seriously at OpenAI. Users should only be accessing the data that they have a legitimate purpose to do so. When we ingest internal knowledge, we actually ingest pre-sanitized queries so that important IDs, for example, aren't accidentally leaked. Especially on Slack, for a Slack agent, when the audience is more broad, we also redact sensitive outputs. This is done by intercepting the results and then passing it to our internal anonymization service that detects PII.拥有强大的数据访问权限意味着巨大的责任。Kepler不提供任何额外的授权。它实际上会传递身份验证。这意味着Kepler不会授予你对你没有访问权限的表的额外访问权限。当你没有访问权限时，Kepler会友好地告诉你加入哪个访问组。或者它可能会使用一个你有访问权限的类似表。数据安全也是我们在OpenAI非常重视的事情。用户只应该访问他们有合法目的的数据。当我们摄取内部知识时，我们实际上会摄取预先清理过的查询，这样重要的ID就不会意外泄露。特别是在Slack上，对于Slack代理，当受众更广泛时，我们还会编辑敏感输出。这是通过拦截结果并将其传递给我们检测PII的内部匿名化服务来实现的。

Sometimes there's actually a reasonable use case for the users to see the raw results. We do actually allow this by linking to an external UI where users who have permissions to those tables, so they would have been able to run the queries themselves anyway, we do permissions check there to make sure that they should actually be seeing that data. This is true for all the terrible pieces that the agent generates. Just like a human, Kepler can also make mistakes. That's why we stream Kepler's chain of thought as Kepler is answering the question, as you saw in those screenshots before. It's been really helpful as an audit and also because in a bunch of cases it's important to understand the assumptions that went into the answer.有时用户有合理的用例需要查看原始结果。我们确实允许这样做，通过链接到外部UI，在那里有权限的用户（他们本来就可以自己运行查询）会进行权限检查，以确保他们应该看到这些数据。这对于代理生成的所有可怕片段都是如此。就像人类一样，Kepler也可能犯错。这就是为什么我们在Kepler回答问题时会流式传输其思维链，正如你在之前的截图中看到的那样。这对于审计非常有帮助，而且在许多情况下，理解答案背后的假设也很重要。

If Kepler ran any queries that resulted in data, Kepler will link those and provide the reference ID so you can click into the raw results.如果Kepler运行了任何产生数据的查询，Kepler将链接这些查询并提供参考ID，以便你可以点击查看原始结果。

Now let's see what internal users are saying about Kepler. One of our users mentioned, for example, how Kepler they felt like was the most useful bot that we had at the company. Another user mentioned how Kepler was really good at writing SQL queries to the point where maybe writing them by hand is a total waste of time. At the bottom there, you can see Kepler is pretty good at sanity checking data and making sure your assumptions are correct about the data. This is my favorite quote, actually. Someone mentioned how Kepler to them felt like the closest thing to AGI that they've used. We have a lot of really nice people.现在让我们看看内部用户对Kepler的评价。例如，一位用户提到，他们认为Kepler是公司最有用的机器人。另一位用户提到，Kepler在编写SQL查询方面非常出色，以至于手动编写可能完全是浪费时间。在底部，你可以看到Kepler在数据合理性检查方面相当不错，可以确保你对数据的假设是正确的。实际上，这是我最喜欢的一句话。有人提到，对他们来说，Kepler感觉像是他们使用过的最接近AGI的东西。我们有很多非常友善的人。

The user love is really great to see. How did we get to this point? Let me talk about some key learnings we had along the way. We owe a lot of our initial success to just having a really quick feedback loop with our users. We partnered with a key team. They would give us feedback. We immediately proved feedback, immediately proved so on and so forth. Let me give an example of what I mean. Initially, we thought that all questions were just metrics questions, like you generate a SQL, gets a result, Kepler agent tells you. Actually, a lot of questions can just be answered by company context or a doc, or it's just table information, like an access group.用户的喜爱真是太好了。我们是如何达到这个地步的？让我谈谈我们一路走来的一些关键经验教训。我们早期的成功很大程度上归功于与用户建立非常快速的反馈循环。我们与一个关键团队合作。他们给我们反馈。我们立即改进反馈，立即改进，等等。让我举个例子说明我的意思。最初，我们认为所有问题都只是指标问题，比如你生成SQL，得到结果，Kepler代理告诉你。实际上，很多问题可以通过公司上下文或文档来回答，或者只是表信息，比如访问组。

As a result of this user feedback, we reworked our backend to also accommodate for these, so the model wouldn't waste things like running noop queries, SELECT 1s, doing table search when it's not really relevant. We can answer these types of important questions that Jimmy is asking right now. The other thing that really helped us was meeting users where they are. A lot of agent interfaces are actually web, for example, ChatGPT. We actually started with the Slack interface because OpenAI is a really Slack heavy company. People post analytics updates on Slack all the time. People ask data questions on Slack all the time. We realized the key to success was actually getting people to ping Kepler with their analytics questions instead. Now I'm going to talk about some lessons learned.由于用户反馈，我们重新设计了后端，以适应这些情况，这样模型就不会浪费资源运行无意义的查询、SELECT 1、进行不相关的表搜索。我们可以回答Jimmy现在问的这些重要问题。另一件对我们帮助很大的事情是满足用户的需求。许多代理界面实际上是Web，例如ChatGPT。我们实际上从Slack界面开始，因为OpenAI是一个非常依赖Slack的公司。人们经常在Slack上发布分析更新。人们经常在Slack上问数据问题。我们意识到成功的关键是让人们用他们的分析问题来@Kepler。现在我将谈谈一些经验教训。

It turns out if you give the model too much information, especially when it's overlapping, it can get really confused. We realized this initially because we have a lot of tool calls that are a little similar just because one might use service auth, one might use user auth, but the model was getting really confused because it just couldn't understand the little subtle nuances. We actually ratcheted down the tool calls that agent can use for easier tool discovery. The other thing is that we found the results were worse when there were really specific instructions. This is because there's just so many different types of questions that people ask. While there might be a similar general overall path, there's a lot of little branches in logic.事实证明，如果你给模型太多信息，尤其是重叠的信息，它可能会非常困惑。我们最初意识到这一点，因为我们有很多工具调用有点相似，只是因为一个可能使用服务认证，另一个使用用户认证，但模型非常困惑，因为它无法理解微小的细微差别。我们实际上减少了代理可以使用的工具调用，以便更容易发现工具。另一件事是，我们发现当指令非常具体时，结果更差。这是因为人们问的问题类型太多了。虽然可能有一个相似的大致路径，但逻辑中有很多小的分支。

Being overly prescriptive actually hurt us because the model would try to follow an exact set of instructions that didn't really make sense for maybe their question. We actually changed our prompting to be a little more general so that while Kepler would get a rough certain point, we leave it to the reasoning of GPT-5 to understand the exact path that it should take. Because at the end of the day, it does have all that context.过于具体反而有害，因为模型会试图遵循一套精确的指令，而这些指令可能不适用于他们的问题。我们实际上将提示改得更通用一些，这样虽然Kepler会有一个大致的起点，但我们让GPT-5的推理来决定应该采取的确切路径。因为归根结底，它拥有所有上下文。

What's next up for Kepler? One thing we'd like to do is to fine-tune a model dedicated specifically for Kepler. There's a lot of little areas where Kepler doesn't get things right. For example, some SQL quirks. We have a lot of data on the questions that users ask and the right SQL that can be generated. The plan is to use that data and train a model on it so that Kepler gets even better for internal use cases. We care a lot about our users' trust in Kepler. We want the responses to be validated and correct. That's super important to our success, because if people don't trust Kepler why would they use it? That's why we're planning to build in extra validation steps so Kepler can check itself like a human would. One example process is, you might look at a number at a dashboard somewhere after you run the query. Kepler can take this result and can do a comparison just to make sure that the number matches, and so that gives higher confidence that actually that is the right answer.Kepler的下一步是什么？我们想做的一件事是微调一个专门用于Kepler的模型。有很多小地方Kepler做得不对。例如，一些SQL怪癖。我们有很多关于用户问的问题和可以生成的正确SQL的数据。计划是使用这些数据训练一个模型，使Kepler在内部用例中变得更好。我们非常关心用户对Kepler的信任。我们希望响应得到验证并且正确。这对我们的成功至关重要，因为如果人们不信任Kepler，他们为什么要使用它？这就是为什么我们计划构建额外的验证步骤，这样Kepler可以像人类一样自我检查。一个示例过程是，你可以在运行查询后查看仪表盘上的某个数字。Kepler可以获取这个结果并进行比较，以确保数字匹配，从而增加对答案正确的信心。

If there's anything you take away from this talk, I hope these three things stick with you. Firstly, it's really important to have context beyond just table metadata. The code and rich context that I talked about and the company context goes a long way. Also, incorporating memory is really valuable so that your agent can continuously improve. Evals are particularly important to make sure that your model remains consistently good.如果你们从这次演讲中带走任何东西，我希望这三件事能留在你们心中。首先，拥有超越表元数据的上下文非常重要。我谈到的代码和丰富上下文以及公司上下文大有帮助。此外，整合记忆非常有价值，这样你的代理可以持续改进。评估对于确保你的模型保持一贯的良好表现尤其重要。

Participant 1: I'm curious about the user personas of the people that are using Kepler. Is it the data analysts that were doing analytics before? Is it the end users who now don't need them and they just ask Kepler, or somewhere in between?参与者1：我很好奇使用Kepler的用户画像。是之前做分析的数据分析师吗？还是现在不需要他们、直接问Kepler的最终用户，或者介于两者之间？

Bonnie Xu: We actually started out with the data scientists primarily because they have the most context and what a right answer might be. Especially because initially the responses just weren't as good. Now that Kepler has gotten a lot better, we've branched out. We now have users across GTM. We have users in finance and econ, API as well. There are folks working on Sora on ChatGPT that also ask Kepler questions. We've branched out to the whole company at this point.Bonnie Xu：我们实际上最初主要面向数据科学家，因为他们对正确答案最有上下文。特别是因为最初响应质量不太好。现在Kepler好多了，我们已经扩展到其他领域。我们现在有GTM的用户。我们有财务和经济部门的用户，还有API部门的用户。也有在Sora和ChatGPT上工作的人向Kepler提问。目前我们已经扩展到全公司。

Participant 2: What strategies do you have to manage the context window for your agent?参与者2：你们有什么策略来管理代理的上下文窗口？

Bonnie Xu: I think putting everything into the embedding and then returning that at runtime is really helpful. The right search is actually pretty good. Then the other nice thing is that it can just pick up little pieces. That's how we do it efficiently. I think the other thing, we also have like some tagging actually when looking things up so that it can find the right information, and we do limit the search to an extent. We're also looking into reducing the amount of tool calls or just getting to the right answer faster. That's also one way we reduce the context window size as well just to make sure that the responses still work at the end of the day.Bonnie Xu：我认为将所有内容放入嵌入中，然后在运行时返回非常有帮助。正确的搜索实际上相当不错。另一件好事是它可以只提取小片段。这就是我们高效做到这一点的方法。我认为另一件事，我们在查找时也有一些标签，以便找到正确的信息，并且我们会在一定程度上限制搜索。我们也在考虑减少工具调用次数，或者更快地得到正确答案。这也是我们减少上下文窗口大小的一种方式，以确保响应最终仍然有效。

Participant 3: Do you use the questions asked to the bot to then drive the data modeling? Is that agentic as well? If you've got lots of users consistently asking a question about some tables that haven't been joined, do you then bring that background? For the fine-tuning, what signal are you using? Is there a user action that happens in Slack after they've got their analysis that gives an insight into whether you've got good feedback or not?参与者3：你们是否使用向机器人提出的问题来驱动数据建模？这也是代理式的吗？如果很多用户持续问关于某些尚未连接的表的问题，你们是否会将其纳入背景？对于微调，你们使用什么信号？用户在Slack上获得分析后是否有操作可以表明是否得到了良好反馈？

Bonnie Xu: Your first question is about whether we use the top questions for evals? We actually do. Our evals, they're manually generated now just so we can get the right set, but they're pretty much based on the top common questions that people might answer and also the important ones to get right. Like ChatGPT weekly active users, or what's our revenue for Sora, or something else. Yes, we do actually use that input as informing. That's a lot of the basis of our eval question pairs.Bonnie Xu：你的第一个问题是关于我们是否使用最常见的问题来进行评估？我们确实这样做。我们的评估目前是手动生成的，以便获得正确的集合，但它们基本上基于人们可能回答的常见问题以及需要正确的重要问题。比如ChatGPT周活跃用户，或者Sora的收入等等。是的，我们确实使用这些输入作为参考。这是我们评估问题对的基础。

Then for your second question on fine-tuning and how we get the right inputs for that, we do actually have feedback. There is a manual feedback mechanism that people can use to upvote or downvote. Maybe you've seen this in ChatGPT where you can thumbs up, thumbs down, that's like a really important source for us. We also do get just feedback from Slack threads, for example, so actually in the thread. Those are also inputs, and even just the conversations. We'd like to extend this though, but we could actually look at all the conversations and do a semantic analysis, for example. That's one other way. We aren't currently doing that right now, but we'd like to move in that direction.至于你的第二个问题，关于微调以及如何获得正确的输入，我们确实有反馈。有一个手动反馈机制，人们可以用来点赞或点踩。也许你在ChatGPT中见过，你可以点赞或点踩，这对我们来说是一个非常重要的来源。我们也从Slack线程中获得反馈，例如，实际上在线程中。这些也是输入，甚至包括对话本身。不过我们希望扩展这一点，但实际上我们可以查看所有对话并进行语义分析。这是另一种方式。我们目前还没有这样做，但我们希望朝这个方向发展。

Participant 4: I'm assuming that for your evals you do them before, like assuming that there isn't any memory. How do you make sure that when there is memory included in the context that the memory didn't actually regress the performance of the tool?参与者4：我假设你们的评估是在没有记忆的情况下进行的。你们如何确保当记忆包含在上下文中时，记忆实际上不会降低工具的性能？

Bonnie Xu: The memory check is actually pretty lightweight since it just does RAG search. Yes, that's true. Sometimes maybe it's like a bad memory, it actually does meaningfully present a failure that shouldn't be the case. In those cases though, that's again why the chain of thought is useful. We can inspect what's going on. Then the memory pruning to make sure that these memories make sense. Pretty much it's just a little bit of auditing in our side to make sure that that happens. There are bad memories too, absolutely.Bonnie Xu: 内存检查实际上非常轻量，因为它只是进行RAG搜索。是的，确实如此。有时它可能像一段糟糕的记忆，实际上会错误地呈现一个本不该出现的失败。但在这些情况下，这又体现了思维链的用处。我们可以检查发生了什么。然后通过内存修剪来确保这些记忆有意义。基本上，我们这边只需要做一些审计来确保这一点。当然，也存在不好的记忆。

Participant 5: A question around memory. You talked about having different levels of memory, user, team, and global scope. I'm assuming that you generate these memories in some kind of offline process. You also allow editing on these memories so that users can make any small changes. If that be the case, in this memory pipeline process, if you're rerunning this pipeline process, how do you ensure that the user edited memories are not overwritten? Because like, now, if the memories are going to be idempotent in nature, meaning if I have a conversation, and if I'm generating a memory, even if I rerun, I should be getting the same memory UUID or something of that sort. That's what my assumption is. If that is not the case, then how are you dealing with the conflicts? Now, if you rerun the same pipeline, you might actually generate a different memory, then you would end up in a conflict. First is, how do you avoid overwriting? If not, how are you avoiding the conflicts?参与者5：关于内存的一个问题。你提到了不同级别的内存：用户、团队和全局范围。我假设你是通过某种离线过程生成这些内存的。你还允许用户编辑这些内存，以便他们可以进行小的修改。如果是这样，在这个内存管道过程中，如果你重新运行这个管道，如何确保用户编辑过的内存不会被覆盖？因为现在，如果内存本质上是幂等的，也就是说，如果我进行了一次对话并生成了一个内存，即使我重新运行，也应该得到相同的内存UUID或类似的东西。这是我的假设。如果不是这样，那么你如何处理冲突？如果你重新运行同一个管道，可能会生成不同的内存，从而导致冲突。首先，如何避免覆盖？如果不能，如何避免冲突？

Bonnie Xu: Your question is basically about conflicting memories. I think there's a couple of things that we do in this case. Firstly, we are actually planning to introduce a way that users can see their memories. The whole reason for scopes is that you don't pollute global memory if you shouldn't. If I'm a user, and I'm like, I have this very specific analytics case. This other user is like, no, actually do something very differently. Those are things that are more meant for personal memories. Even within scopes, memories can conflict. Let me take a step back here. For memories, we actually do insert them at runtime, and then we prune them offline, since there could be a lot of them. It's that offline pruning process that helps take care of a bit of that.Bonnie Xu: 你的问题基本上是关于冲突的内存。我认为我们在这方面做了几件事。首先，我们实际上计划引入一种方式，让用户可以看到他们的内存。引入作用域的全部原因是，你不应该在不该污染全局内存的时候去污染它。如果我是用户，我说，我有一个非常具体的分析案例。另一个用户说，不，实际上要做一些非常不同的事情。这些更适合个人内存。即使在作用域内，内存也可能冲突。让我退一步说。对于内存，我们实际上是在运行时插入它们，然后离线修剪，因为可能会有很多。正是这个离线修剪过程有助于处理一部分问题。

Obviously, there still is that gap between when that offline job runs and when it live runs. The other nice thing is that we have the model to fall back on when it does retrieve a memory that is incorrect, or just doesn't make sense in the context of the question, because that will sometimes happen even in its own knowledge search. It is able to use that as just a signal and either disregard it, or maybe don't take it as highly. We've seen that actually in the chain of thought. In most cases, again, it works out. In these edge cases, like you mentioned, it's just another signal. That's why we provide so many signals, because if at least 80% of the signals are good, that's usually enough to push in the right direction.显然，离线作业运行和实时运行之间仍然存在差距。另一个好处是，当模型检索到不正确或与问题上下文不符的内存时，我们有模型可以依赖，因为这种情况有时甚至会在其自身的知识搜索中发生。模型能够将其仅作为一个信号，要么忽略它，要么不将其视为高优先级。我们在思维链中确实看到了这一点。在大多数情况下，它仍然有效。在你提到的这些边缘情况下，它只是另一个信号。这就是为什么我们提供这么多信号，因为如果至少80%的信号是好的，通常就足以推动正确的方向。

Participant 6: Is there any thinking about either open sourcing it as a framework, or maybe as a service for enterprise? Are you finding that users are using Kepler instead of the source systems themselves, just because it's easier?参与者6：有没有考虑过将其作为框架开源，或者作为企业服务提供？你们是否发现用户因为更容易而使用Kepler而不是源系统本身？

Bonnie Xu: Yes, of course. The first question was open source. I do love open source. I don't know if I'm the right person with the right authority, unfortunately, to make that decision.Bonnie Xu: 当然。第一个问题是关于开源的。我确实热爱开源。但不幸的是，我不知道我是否是拥有适当权限来做这个决定的人。

Then on your second question of using Kepler versus just directly querying the source. I think, at least from what we've heard from our users, directly using Kepler is a lot faster. It's more productive, just because when you're looking at different sources, you have to go across like that. All these different sources, you might be doing some curation yourself. Let's say, you're doing Kepler DAC. You're looking at Databricks for your data catalog, and then you're looking at Codex for some code files or whatever. Then maybe you're looking at Airflow for your Airflow job. That's a lot of stuff you have to do. Then you have to connect the dots. Versus Kepler is really that layer on top, that abstraction that does it for you. That's why it's just a lot faster. We've actually seen a lot of folks too, because Kepler will operate independent of them, they just launch a couple of Kepler questions, and they come back to it. You just become so much more productive that way.至于你的第二个问题，关于使用Kepler与直接查询源系统的比较。我认为，至少从我们用户那里听到的反馈来看，直接使用Kepler要快得多，效率更高。因为当你查看不同的源时，你需要跨多个源进行操作。所有这些不同的源，你可能需要自己做一些整理。比如，你使用Kepler DAC，查看Databricks的数据目录，然后查看Codex的代码文件，再查看Airflow的作业。这有很多事情要做，而且你需要连接这些点。而Kepler正是位于其上的一层抽象，为你完成这些工作。这就是为什么它快得多。我们还看到很多人，因为Kepler可以独立运行，他们只需提出几个Kepler问题，然后回来查看结果。这样你的效率就大大提高了。

See more presentations with transcripts查看更多带文字记录的演示