Ontology‐Driven Observability: Building the E2E Knowledge Graph at Netflix Scale本体驱动的可观测性：在 Netflix 规模下构建端到端知识图谱

Prasanna Vijayanathan: My name is Prasanna Vijayanathan. I'm part of the observability team at Netflix. I work on observability, performance, and quality of experience across all our member applications. What that means is from the point you click on the Netflix logo until you start watching something, how does your experience feel on Netflix is what I look into. What metrics should we look at? What tools should we have? What insights should we provide to our users to make sure that we understand what you all like? If you do like watching Netflix, you're welcome. If you don't, you can blame it on my colleague, Renzo.Prasanna Vijayanathan：我叫 Prasanna Vijayanathan，是 Netflix 可观测性团队的一员。我负责所有会员应用程序的可观测性、性能和体验质量。这意味着从你点击 Netflix 图标的那一刻起，直到你开始观看内容，我对你在 Netflix 上的体验感受负责。我们应该关注哪些指标？我们应该拥有哪些工具？我们应该为用户提供哪些洞察，以确保我们了解大家的喜好？如果你喜欢看 Netflix，欢迎；如果你不喜欢，那你可以怪我的同事 Renzo。

Renzo Sanchez-Silva: I'm Renzo Sanchez-Silva. I've been at Netflix for almost 10 years now with the monitoring and alerting team within observability, building several alerting infrastructure, and now getting into AIOps space. I've done also some research back in the day in ontology. With AI, everything is converging.Renzo Sanchez-Silva：我是 Renzo Sanchez-Silva。我在 Netflix 工作快 10 年了，一直身处可观测性部门的监控和告警团队，构建了多种告警基础设施，现在开始涉足 AIOps 领域。我过去也做过一些本体论方面的研究。随着 AI 的发展，一切都在融合。

Prasanna Vijayanathan: Today we're going to talk to you about a problem that we've been tackling, which at the surface feels like an observability challenge, but it's really a hard data engineering problem. Before I get into that, let me set the stage up. Why is observability hard at scale? What scale are we even talking about? Let's do a pop-up quiz. Netflix launched live events. End of 2024 was one of our first major big live events, a boxing match between Jake Paul and Mike Tyson. A few of you might have watched it. What do you think was our peak traffic during that event? It's definitely in the millions. It's 65 million concurrent streams. What that means is there are 65 million users clicking watch and watching the stream at the same time. That was our peak traffic. One more. When you log into Netflix, the apps are sending requests into our infrastructure for our backends to respond and stuff.Prasanna Vijayanathan：今天我们要和大家讨论一个我们一直在解决的问题，表面上看这是一个可观测性挑战，但本质上是一个艰巨的数据工程问题。在深入探讨之前，先让我铺垫一下。为什么大规模可观测性很难？我们谈论的规模到底有多大？让我们做一个小测验。Netflix 推出了直播活动。2024 年底是我们首批大型直播活动之一，杰克·保罗（Jake Paul）和迈克·泰森（Mike Tyson）之间的拳击赛。你们中可能有人看过。你们认为那次活动的峰值流量是多少？肯定是以百万计。是 6500 万并发流。这意味着有 6500 万用户同时点击观看并播放流媒体。那是我们的流量峰值。再来一个。当你登录 Netflix 时，应用程序会向我们的基础设施发送请求，以便我们的后端进行响应等。

How many requests do you think we handle per day coming from our applications to our servers? We handle more than 2 billion requests a day. Last one. With all those requests happening, we obviously have our own observability systems where we log metrics, a lot of different things. How many real-time logging events do you think we have to handle per second? It's upward of 38 million events per second. That is the scale we're talking about. Numerous client platforms and devices across all the TVs all over the world, mobile phones, laptops, you name it. Thousands of microservices emitting all these billions of telemetry events. Every single member touches so many different parts of our infrastructure, our CDNs, nodes, and so on. On top of all that, we have thousands of experiments running through A/B tests throughout our ecosystem. Managing all of that at scale. It's everything, everywhere, all at once.你认为我们每天处理多少来自应用程序发往服务器的请求？我们每天处理超过 20 亿次请求。最后一个。在所有这些请求发生的同时，我们显然有自己的可观测性系统，记录指标以及许多其他内容。你认为我们每秒需要处理多少实时日志事件？超过 3800 万次事件每秒。这就是我们所说的规模。全球范围内所有的电视、手机、笔记本电脑等无数客户端平台和设备。数以千计的微服务都在产生所有这些数十亿条遥测事件。每一位会员都会触及我们基础设施、CDN、节点等不同部分。除此之外，我们整个生态系统中还在运行数千个 A/B 测试实验。在大规模环境下管理这一切，就像是“瞬息全宇宙”一样。

Even at that scale, our goal at Netflix is to deliver the best possible experience to every user every single time. Our strategy towards observability is simple. Do we have the right tools, metrics, and insights to enable us to do that? The strategy is simple. It is not easy. At this scale, correlating user experience with system performance is a fundamentally different challenge. It is not a tooling challenge anymore. It is a data engineering challenge. Traditionally, observability systems have been monitoring and alerting tools where you observe your apps and services. Something goes wrong, you get flagged. Then you look at what's going on, triage it to the right teams, look for the right developers to fix it if needed, roll out a fix, and then see if the data comes back to where it is expected to be. That's how traditionally observability systems have worked. With end-to-end observability, we wanted to rethink how we do observability, and change it from a reactive monitoring experience to a proactive insights engine.即使在这样的规模下，Netflix 的目标依然是每次都为每位用户提供尽可能最好的体验。我们对可观测性的策略很简单：我们是否拥有合适的工具、指标和洞察来实现这一目标？策略很简单，但执行起来并不容易。在这种规模下，将用户体验与系统性能相关联是一个本质上完全不同的挑战。这不再是一个工具问题，而是一个数据工程问题。传统的可观测性系统是监控和告警工具，用于观察应用程序和服务。一旦出问题，你就会收到标记。然后你查看发生了什么，分诊给正确的团队，寻找合适的开发人员修复（如果需要），发布修复程序，然后观察数据是否恢复到预期水平。这就是传统可观测性系统的工作方式。通过端到端可观测性，我们希望重新思考如何进行可观测性，并将其从被动监控体验转变为主动洞察引擎。

Before I show you how we do that, I want to talk about the vision that we created. Imagine a system that can detect issues automatically in a few minutes across the entire stack. It's not just one service or app, but across your stack. Wherever an issue happens, you know that this is what is happening. Then you go into automatically prioritizing those issues based on how it's impacting our users. You also can triage it to the right team automatically. You don't have to look at one app and one error and say, I think I'm going to follow this needle through this haystack and pick out different services and loop them in. Then, can we automatically root cause? Not just triage, but can we actually identify the root cause of the issue? Or, even better, can we predict issues even before they would actually start impacting our users and maybe even start suggesting or taking corrective actions?在展示我们如何做到这一点之前，我想谈谈我们创建的愿景。想象一个系统，能够自动在几分钟内检测整个技术栈中的问题。不仅仅是一个服务或应用程序，而是整个技术栈。无论问题发生在哪里，你都知道正在发生什么。然后，根据对用户的影响程度自动对这些问题进行优先级排序。你还可以自动将其分诊给正确的团队。你无需盯着一个应用程序和一个错误，然后说：“我想我得顺着这根针穿过这片干草堆，挑出不同的服务并将它们关联起来。” 那么，我们能否自动找出根本原因呢？不仅仅是分诊，我们能否真正识别问题的根本原因？或者更好的是，我们能否在问题开始影响用户之前就预测到它们，甚至可能开始建议或采取纠正措施？

Is this possible? We wanted to do this. Building the system would give us seamless visibility from the user's device through our networks into our gateway services into the deep trenches of our backend dependencies, services. All connected, all in real time. Users would have no disruptions to their experience on Netflix. Every time you watch Netflix, you will love it. Your experience is going to be awesome, every single time. This is our vision.这可能吗？我们想做到这一点。构建这个系统将使我们能够从用户设备、穿过我们的网络、进入网关服务，再到后端依赖项和服务的深处，获得无缝的可见性。所有这些都是实时连接的。用户在 Netflix 上的体验将不会受到任何干扰。每次你观看 Netflix，你都会喜欢它。你的体验每次都会非常棒。这就是我们的愿景。

Sounds great. That's far from reality. It's very different today. I'll walk you through a timeline of a recent investigation, probably within the last year or so. There was an alert that got triggered on one of our client platforms, I'd say around 3:30. Typically, on-calls look at it and they're like, let me wait for a while. Let me see if this is going to sustain. Maybe something happens magically and it'll go away on its own. Sometimes it does. Sometimes it doesn't. A couple of hours later, they're actually looking at it, "I think it's a real issue. I need to go figure out, follow this needle and see if there's another service that's impacting." In the average case, usually it does. Then it's triaged to the right team. Then it turns out that team was already debugging this issue from like 1:30 that day. People are already on it.听起来很棒。但这离现实还很远。今天的情况截然不同。我将带你回顾最近一次调查的时间线，大概在过去一年左右。我们的一个客户端平台触发了一个告警，时间大约是 3:30。通常，值班人员看到后会想：“让我等一会儿。看看它是否会持续。也许什么东西会神奇地发生，然后它自己就会消失。” 有时确实会这样。有时则不会。几个小时后，他们才真正开始查看：“我想这是一个真正的问题。我需要去弄清楚，顺着这根针找，看看是否有另一个服务在产生影响。” 在一般情况下，通常确实如此。然后它被分诊给正确的团队。结果发现，那个团队从当天 1:30 开始就已经在调试这个问题了。人们已经在处理了。

Then, multiple teams are involved now. There's a lot more debugging happening between all these different teams. Then once the debugging starts, they root cause the actual issue, follow the needle down to the actual offending service. In this case, it turned out there was some service that thought they're not as important and cut down their capacities. That caused backward propagation errors. That's around like 6:30, three hours since the client application actually noticed real user impact. We rolled out a fix. It was resolved within the hour. This is an average case. It doesn't sound too bad. It's like four hours. Not too bad. Then when you consider what it took, during this time, there are nine teams that were paged. More than 30 engineers were involved in the entire debugging process in different capacities. Then three other related incidents that were also flagged. There's panic everywhere across teams. This is typically, and this is an average case. This is not even a bad example. Then when you put together the four hours and all the resources that get involved, it just feels wasteful. This is what you want to solve with end-to-end observability.然后，现在涉及了多个团队。所有这些不同团队之间进行了更多的调试。调试开始后，他们找出了实际问题的根本原因，顺着针找到了实际出问题的服务。在这种情况下，结果发现某个服务认为自己没那么重要，削减了容量。这导致了反向传播错误。那大约是 6:30，距离客户端应用程序真正注意到用户受到实际影响已经过去了三个小时。我们发布了修复程序。在一小时内解决了。这只是一个平均情况。听起来还不算太糟。大约四个小时。不算太糟。但当你考虑所付出的代价时，在此期间，有九个团队收到了传呼。超过 30 名工程师以不同的身份参与了整个调试过程。还有三个其他相关事件也被标记出来。团队之间到处都是恐慌。这通常是平均情况。这甚至还不是一个糟糕的例子。当你把这四个小时和所有涉及的资源放在一起时，感觉真是浪费。这就是你想要通过端到端可观测性来解决的问题。

We looked at why. Why does this happen? We came up with four main themes that existed. One, you've got too many siloed data sources. We got roughly logs, metrics, events, and traces that are stored in different forms in different datastores. They are in silos. They don't talk to each other. They're not standardized. Structures are different, and so on. That leads to disconnected and non-contextual alerting, which means each team follows a data source that works well for them in their silo and then set up alerts. Then when alerts trigger, they are looking at the same problem in one way compared to another team that's looking at the same problem from a different perspective. They should be talking to each other, but that doesn't happen. What that leads to is complexity in how teams triage and actually troubleshoot the issue. Ultimately, what that means is we actually don't detect what is the impact for our users. If you're a downstream service, 5 hops, 10 hops away from the user, you almost never understand how your system and service is affecting the end user.我们研究了原因。为什么会发生这种情况？我们总结了四个主要主题。第一，你有太多孤立的数据源。我们大概有日志、指标、事件和追踪，它们以不同的形式存储在不同的数据存储中。它们处于孤岛中。它们互不通信。它们没有标准化。结构各不相同，等等。这导致了脱节且缺乏上下文的告警，这意味着每个团队都遵循适合他们孤岛的数据源，然后设置告警。当告警触发时，他们以一种方式看待同一个问题，而另一个团队则从不同的角度看待同一个问题。他们应该进行沟通，但这并没有发生。这导致团队在分诊和实际排查问题时变得复杂。最终，这意味着我们实际上并没有检测到对用户的影响是什么。如果你是一个距离用户 5 跳、10 跳的下游服务，你几乎永远无法理解你的系统和服务是如何影响最终用户的。

Given the vision that we saw, that we have for end-to-end observability, how do we solve this and move from the state of reality to the vision? The one-word answer for that is connectedness. That's what we thought. The common theme is that. We want connected data, connected alerting, connected debugging systems, workflows, and root cause analysis. If we create that theme for end-to-end observability, we don't need 10 different data sources giving rise to 6 different tools. We want to create an integrated observability layer. One abstraction that can tell you everything about our system's observability. What that means, then, is our dashboards, alerts, debugging workflows, and incident response tools, they all speak the same language. Then duplicating efforts in debugging and triaging, the same issues, and the same symptoms across the system is reduced, which means you can follow a breadcrumb trail, and then get into the same path from the user's device to the actual root cause.鉴于我们看到的端到端可观测性的愿景，我们如何解决这个问题并从现实状态迈向愿景？用一个词回答就是：连通性。这就是我们的想法。共同的主题就是这个。我们需要连接的数据、连接的告警、连接的调试系统、工作流和根本原因分析。如果我们为端到端可观测性创建该主题，我们就不需要 10 个不同的数据源产生 6 个不同的工具。我们想要创建一个集成的可观测性层。一种可以告诉你关于我们系统可观测性一切的抽象。这意味着，我们的仪表板、告警、调试工作流和事件响应工具都使用相同的语言。这样，在调试和分诊系统中的相同问题和相同症状时，重复的工作就会减少，这意味着你可以沿着面包屑路径，从用户设备一直追踪到实际的根本原因。

You can follow the breadcrumb from different trails, but get on the same main path of root causing the issue. That leads to reducing the time it takes to resolve issues and ultimately will improve the accuracy in diagnostics, and give our users a less disrupted and better experience. The core piece of all this starts with the data, connected data. We got our users, our devices and applications, and our services and infrastructure, all emitting these broadly four different categories of data: metrics, events, logs, and traces. We unify them and put them in one layer, the MELT layer. Once we unify all these sources, we solve the problem. It's important that even if all of this exists together, we need them to be accurate and actionable.你可以从不同的路径追踪面包屑，但最终会走上查找问题根源的同一条主路径。这有助于减少解决问题所需的时间，并最终提高诊断的准确性，为我们的用户提供更少干扰、更好的体验。所有这一切的核心始于数据，即连接的数据。我们拥有用户、设备和应用程序，以及服务和基础设施，它们都在产生这四大类数据：指标、事件、日志和追踪。我们将它们统一起来并放入一层，即 MELT 层。一旦我们统一了所有这些来源，我们就解决了问题。即使所有这些都共存，它们也必须准确且可操作，这一点很重要。

Let me walk you through another scenario. Let's say we have this layer already. Let's say there is a regression we see, LOLOMO TTR. LOLOMO is the metrics of the list-of-list-of-movies that you see on Netflix. TTR, some rendering time for that list-of-list-of-movies. You see a regression there. There's a slow leak, and there's a big regression. Developer notices this and they ask our unified data layer, what happened? Now all the observability data lives in one layer, so tell me what happened. Then the metrics system would tell you, yes, I see LOLOMO TTR regressing. I also see 12 other metrics across 5 different services correlating, happening at the same time. They are also regressing. We find some correlations. That's good. Then events could tell you there are 47 events across all these different apps and services that also happened right before this regression started. Yes, I can see it on a graph.让我带你了解另一个场景。假设我们已经有了这一层。假设我们看到了一个回归，LOLOMO TTR。LOLOMO 是你在 Netflix 上看到的电影列表的指标。TTR 是该电影列表的某种渲染时间。你在那里看到了一个回归。有一个缓慢的泄漏，并且有一个大的回归。开发人员注意到了这一点，他们询问我们的统一数据层：发生了什么？现在所有的可观测性数据都存在于一层中，所以告诉我发生了什么。然后指标系统会告诉你，是的，我看到 LOLOMO TTR 在回归。我还看到跨 5 个不同服务的 12 个其他指标在同一时间相关联，它们也在回归。我们发现了一些相关性。这很好。然后事件可以告诉你，在这次回归开始之前，所有这些不同的应用程序和服务中也发生了 47 个事件。是的，我可以在图表上看到它。

I can see all these events along with the metrics. Then logs tells me, I think there is a connection pool exhaustion error that spiked up at this time in one of those 7 services. Maybe that's what it is. There are also all these timeout errors in all these other services at the same time too. Then traces can tell you a different story. If I'm going to talk about the work we've done with traces, that will be a different session on its own. To give you an idea, a trace gives you a path for a particular request and response from the user to all the services that touched it. You can trace that path. The challenge with tracing is, unless you know exactly what trace you're looking for, it is super hard to use it to debug an issue accurately. It's not even a needle in a haystack. It's like looking for a leaf in a forest in the dark and you're blindfolded. It's hard.我可以看到所有这些事件以及指标。然后日志告诉我，我认为这 7 个服务中的一个在此时出现了连接池耗尽错误。也许这就是原因。同时，所有其他服务中也出现了所有这些超时错误。然后追踪可以告诉你一个不同的故事。如果要谈论我们在追踪方面所做的工作，那将是另一个单独的会议。给你一个概念，追踪为你提供了一条从用户到触及它的所有服务的特定请求和响应的路径。你可以追踪那条路径。追踪的挑战在于，除非你确切知道你在寻找什么追踪，否则很难用它来准确调试问题。这甚至不是大海捞针。这就像在黑暗中寻找森林里的一片叶子，而且你还被蒙住了眼睛。这很难。

What we are really missing is not just connectedness, but the relationships between all these different pieces of this puzzle. We need to know the connections from these components, yes, but we also want to know how they interact with each other and how they are related to each other. This we call the observability ontology. Renzo will walk you through how we build these relationships and how we're putting this to use every day.我们真正缺失的不仅仅是连通性，而是这个谜题所有这些不同部分之间的关系。我们需要知道这些组件之间的连接，是的，但我们也想知道它们是如何相互作用的，以及它们是如何相互关联的。我们称之为可观测性本体。Renzo 将带你了解我们如何构建这些关系，以及我们如何每天使用它们。

Renzo Sanchez-Silva: What is an ontology? It's just a formal specification of types, properties, and relationships. It's how we encode meaning. The basic construct of it is what you call the triple. The triple is a subject, a predicate, and an object, one fact per triple. It can be, a person is an employee. A service is important, or tier-0, for example. It can be properties too. This is the way we encode knowledge. It's very basic, very primordial, actually. It's like saying Tarzan loves Jane. Mommy loves Daddy. Babies have mini ontologies in their head. We just start collecting all these symbolic triples in our heads, and we start making propositions and reasoning and so forth. Ontologies are a way to encode knowledge. We're using W3C standards, RDF, OWL, web ontology language, SPARQL, the query language, SHACL is shapes constraint language, where we can say a service can be terminated in AWS, but then another constraint is if the service has more or less than 10% of capacity, you cannot terminate.Renzo Sanchez-Silva：什么是本体？它只是类型、属性和关系的正式规范。它是我们编码意义的方式。它的基本结构就是你所说的“三元组”。三元组是主体、谓词和对象，每个三元组一个事实。它可以是：某人是员工。某服务很重要，或者例如属于 tier-0。它也可以是属性。这就是我们编码知识的方式。它实际上非常基础，非常原始。就像说“泰山爱简”、“妈妈爱爸爸”。婴儿头脑中也有微型本体。我们只是开始在头脑中收集所有这些符号化的三元组，并开始做出命题和推理等等。本体是编码知识的一种方式。我们正在使用 W3C 标准：RDF、OWL（网络本体语言）、SPARQL（查询语言）、SHACL（形状约束语言），我们可以说一个服务可以在 AWS 中被终止，但另一个约束是如果该服务的容量超过或低于 10%，则不能终止。

You can encode all these things. You can describe your services, the whole infrastructure. For example, we have API gateway. It's of type application. It's owned by a team. It is deployed in a region, and so forth. We can go and describe every single piece of the infrastructure, and every piece there is namespaced as well. We started building this baby ontology by describing incidents. We have identified a few namespaces. We're building an operational ontology. The goal, again, going back to Prasanna's vision, is to start finding this end-to-end observability graph of all the things connected in our infra. Let's call that the operational ontology. For instance, we identify these namespaces, and they can be referenced formally in the triples.你可以对所有这些进行编码。你可以描述你的服务，整个基础设施。例如，我们有 API 网关。它是应用程序类型。它由一个团队拥有。它部署在一个区域中，等等。我们可以去描述基础设施的每一个部分，而且每一部分也都进行了命名空间划分。我们开始通过描述事件来构建这个初级本体。我们已经确定了一些命名空间。我们正在构建一个运营本体。目标，回到 Prasanna 的愿景，是开始寻找我们基础设施中所有连接在一起的事物的端到端可观测性图谱。我们称之为运营本体。例如，我们识别这些命名空间，它们可以在三元组中被正式引用。

The problem is how to bring order to this operational chaos, in incidents scattered across Slack, PagerDuty, the MELT layer, metrics, traces, logs, and events. We all gather in the channel together. There are hundreds of messages, many services, people interacting. It's crazy until we find root cause, and then we fix the issue many hours later. What we want to do is observe that, start encoding in real time. Capturing the structures and how we preserve this in machine-readable types. A message can be, obviously, in English. Then someone says, API gateway has latency spiked after this deployment and so forth, and then Jane is rolling back. Then you can describe that in triples. That's the idea of doing this. Doing this by hand is hard. In fact, back in the day when there was no AI, you needed to have an ontologist who was always annotating, this is that, this is a person, and this is a service, and so forth.问题是如何将秩序带入这种运营混乱中，这些事件分散在 Slack、PagerDuty、MELT 层、指标、追踪、日志和事件中。我们都在频道里聚在一起。有数百条消息，许多服务，人们在互动。这很疯狂，直到我们找到根本原因，然后几小时后我们修复了这个问题。我们想做的是观察它，开始实时编码。捕获结构，以及我们如何以机器可读的类型保存它。一条消息显然可以用英语。然后有人说，“API 网关在这次部署后延迟激增”，等等，然后简正在回滚。然后你可以用三元组来描述它。这就是这样做的想法。手工做这件事很难。事实上，在没有 AI 的时候，你需要一位本体学家一直进行注释：这是那个，这是人，这是服务，等等。

With AI, we can do this much faster. For this, we created what we call a harvest pipeline. The incidents occur in Slack. That's where the action happens. For each channel, we transform every message, and we send every message to a bunch of enrichers. We call observability APIs. We use LLMs on top to find some extra classification, we correlate that. We save that in our graph database, we call it QuipuDB. The enrichers in green are very deterministic. We want to identify — for example, this is on incident.io, we use them as our incident management software — it's of this ID. We have the PagerDuty alert that was fired, for example, we extract that. We have various very deterministic, small algorithms that classify things because we know URLs, for example, we extract IDs. We extract, for example, also the sentiment of what is happening in the thread or the message.有了 AI，我们可以更快地做到这一点。为此，我们创建了所谓的“收获流水线”（harvest pipeline）。事件发生在 Slack 中。那是行动发生的地方。对于每个频道，我们将每条消息转换，并将每条消息发送给一组丰富程序（enrichers）。我们调用可观测性 API。我们在上面使用 LLM 来找到一些额外的分类，我们将此相关联。我们将其保存在我们的图数据库中，我们称之为 QuipuDB。绿色的丰富程序是非常确定性的。我们想要识别——例如，这是 incident.io，我们使用它们作为我们的事件管理软件——它是这个 ID。我们有触发的 PagerDuty 告警，例如，我们提取它。我们有各种非常确定性的小算法来分类事物，因为我们知道 URL，例如，我们提取 ID。我们也提取，例如，线程或消息中正在发生的事情的情绪。

In blue, we have alerts and the timeline that is occurring. People are sharing links to our dashboard. We encode that. We know what that means. Someone uploads a snapshot or a screenshot, we save that. It's a screenshot related to this graph, this metric. Then we use also LLMs at the end of this process where we have identified all these basic patterns or entities, and start finding new relationships or concepts, or new definitions. The LLM, you can see great things like this has canary analysis, deploy failed, and things like that. It starts finding concepts from what has been currently harvested. If it's new, we'll try to propose a new definition. This is what it's called in ontology, the open world assumption, where you are somewhat free to add relationships and we have ways to control that. It's a very controlled process. This is sequential. Sometimes if something fails, we just leave it. For example, we cannot grab some data, for example, you would leave it there. Then we check against our main metadata sources. For example, the software catalog is where we have the definition of all our services. We identify every service, and we know which type they are, and so forth. It's very controlled.在蓝色部分，我们有告警和正在发生的时间线。人们正在分享指向我们仪表板的链接。我们对其进行编码。我们知道这意味着什么。有人上传快照或截图，我们保存它。这是一个与该图表、该指标相关的截图。然后，在这个过程的最后，我们还使用 LLM，当我们识别了所有这些基本模式或实体时，开始寻找新的关系或概念，或新的定义。LLM，你可以看到很棒的事情，比如“这有金丝雀分析”、“部署失败”等等。它开始从当前已收获的内容中寻找概念。如果是新的，我们会尝试提出一个新的定义。这就是本体论中所说的“开放世界假设”，你在某种程度上可以自由添加关系，我们有方法来控制它。这是一个非常受控的过程。这是顺序的。有时如果某事失败了，我们只是离开它。例如，我们无法获取某些数据，例如，你会把它留在那里。然后我们针对我们的主要元数据源进行检查。例如，软件目录是我们拥有所有服务定义的地方。我们识别每个服务，我们知道它们是什么类型，等等。这非常受控。

The goal is to create determinism from non-determinism. The non-determinism starts with the chaos in the channel, for example. That is the first source of non-determinism. Then we have a bunch of rules where we extract the data. Then, at the very end, the icing on the cake is the LLM coming and saying, this is probably a new relationship. There's a new connection here, and so forth. The paradox here is that we use stochastic tools to produce deterministic SPARQL queryable graph. The ontology is the bridge. It's where we save all the data. We identify five layers. This is a bootstrapping process. Deterministic extraction is those steps in green. We have a bunch of regexes, for example, that extract what a Jenkins job is by ID, by server. That's one entity. Those are very well-known patterns. Then the catalog validation is cataloging or checking against the service is a valid service deployed in this region and so forth.目标是从非确定性中创造确定性。非确定性始于频道中的混乱，例如。那是第一个非确定性来源。然后我们有一堆规则，我们在其中提取数据。然后，在最后，锦上添花的是 LLM 过来并说，这可能是一个新的关系。这里有一个新的连接，等等。这里的悖论在于，我们使用随机工具来产生确定性的 SPARQL 可查询图谱。本体是桥梁。它是我们保存所有数据的地方。我们确定了五个层级。这是一个引导过程。确定性提取是那些绿色的步骤。例如，我们有一堆正则表达式，通过 ID、服务器提取 Jenkins 作业是什么。那是一个实体。那些是非常众所周知的模式。然后目录验证是对服务进行编目或检查，以确保它是部署在该区域中的有效服务，等等。

We enrich with our APIs, observability APIs, and then we pass the LLM to classify and find new predicates. We also have a final step, what we call the probabilistic correlation, where we do the first search in all the conversation. We find chains where the incident started with this one application that affected application B, the original application was rolled back, and then the issue was fixed. That chain is identified also as an entity, a pattern that will be saved so that in the future, maybe we will find similarities, this incident with something in the future. Also, everything is constrained. We have categories, some taxonomies, and they are all constrained. When the LLM comes in, it wants to classify some category, say whether this conversation is an incident, question, discussion, or decision. It can only choose from those four values. We don't let it hallucinate new things unless maybe it tells us that, and then we'll see how we can manage that. Everything is typed through, for example, thread 123 is related, true is a Boolean.我们使用我们的 API、可观测性 API 进行丰富，然后我们传递 LLM 进行分类并找到新的谓词。我们还有一个最后一步，我们称之为概率相关性，我们在所有对话中进行第一次搜索。我们找到链条，其中事件始于这个影响了应用程序 B 的应用程序，原始应用程序被回滚，然后问题得到解决。该链条也被识别为一个实体，一个将在未来保存的模式，以便将来我们可能会发现相似之处，这个事件与未来的某些事物。此外，一切都受到约束。我们有类别、一些分类法，它们都受到约束。当 LLM 进来时，它想要分类某些类别，说这次对话是事件、问题、讨论还是决策。它只能从那四个值中选择。我们不让它产生新的幻觉，除非它告诉我们，然后我们会看看我们如何管理它。一切都通过类型化，例如，线程 123 相关，true 是布尔值。

The process we're describing is a bootstrapping process, what we call the knowledge flywheel, where every rotation, say a thread starts. We start identifying entities. More data comes in, we go around this flywheel by observing, enriching, inferring, and adapting. We observe the MELT data. We enrich with all these enrichers. We infer possibly new patterns or old patterns just to apply them, and we adapt to see what knowledge we can gather. The causal chain from one incident becomes a known pattern for the next. Observe, enrich, infer, that's the main process. In every iteration, we encode knowledge. Each cycle teaches the system, how the ontology will evolve this schema into memory. We have a process also where data is converting information, that information is encoded in knowledge. The fourth is wisdom, wisdom in the sense that we can find root cause. We can encode that today, but also in the future.我们描述的过程是一个引导过程，我们称之为“知识飞轮”，每转一圈，比如一个线程开始。我们开始识别实体。更多数据进来，我们通过观察、丰富、推理和适应来绕过这个飞轮。我们观察 MELT 数据。我们用所有这些丰富程序进行丰富。我们推理可能的模式或旧模式只是为了应用它们，我们适应以查看我们可以收集什么知识。从一个事件产生的因果链成为下一个事件的已知模式。观察、丰富、推理，这是主要过程。在每次迭代中，我们编码知识。每个周期都教导系统，本体将如何将此模式演变为内存。我们还有一个过程，其中数据正在转换信息，该信息被编码在知识中。第四是智慧，智慧的意义在于我们可以找到根本原因。我们今天可以编码它，但将来也可以。

We're not predicting here yet. We're just describing always. Describing first. That's what a baby does. Daddy loves mommy, he knows that is a fact, but then more things will happen later. Everything is queryable via SPARQL. SPARQL is a weird syntax, but you get used to it. LLMs love it. We don't care. I don't even read it sometimes. Again, the ontology accumulates the structure of what keeps happening. Patterns become queryable facts.我们在这里还没有预测。我们只是在描述。先描述。这就是婴儿所做的。爸爸爱妈妈，他知道那是一个事实，但后来会有更多的事情发生。一切都可以通过 SPARQL 查询。SPARQL 是一种奇怪的语法，但你会习惯它。LLM 喜欢它。我们不在乎。我有时甚至不读它。同样，本体积累了持续发生的事物的结构。模式变成了可查询的事实。

It's not surprising that we use AI with this. We use Claude as our co-developer. When we get a new thread or a new message in an incident, we run this harvester. It creates a git worktree because we don't want to conflict with other branches, because there are many incidents happening, or maybe there's another iteration coming, the previous harvest can be working still on something and then it doesn't finish yet. It creates a git worktree or a branch. Then we use Claude also to harvest. We are not just running a script. We are running Claude with the harvesting script to extract the data because there's the LLM step that finds new patterns. At the same time, we use it to find maybe bugs in this very same script, so that it actually fixes itself. Then, say, that whole loop ends, and then we ended up with 15 branches only for this series of iterations, for one incident, and then human in the loop comes in.我们使用 AI 来做这件事并不奇怪。我们使用 Claude 作为我们的共同开发者。当我们收到一个新的线程或事件中的新消息时，我们运行这个收获程序。它创建一个 git 工作树，因为我们不想与其他分支冲突，因为有许多事件发生，或者也许有另一个迭代正在到来，之前的收获可能还在处理某些事情，并且还没有完成。它创建一个 git 工作树或分支。然后我们也使用 Claude 来收获。我们不仅仅是在运行脚本。我们正在运行带有收获脚本的 Claude 来提取数据，因为有 LLM 步骤可以找到新的模式。同时，我们使用它来找到这个脚本中可能存在的错误，以便它实际上可以自我修复。然后，说，整个循环结束了，然后我们最终为这一系列迭代得到了 15 个分支，针对一个事件，然后人类参与循环。

Either me or Prasanna can come in and say, we got all these branches, and then we ask Claude to cherry pick and see what happened. It finds very interesting things. It finds new facts. Some, it says, these five branches did almost the same. They found this pattern. This other branch realized it was running on Sonnet 4.5, and it decided to upgrade itself to 4.6, which is cool. It's self-healing at the same time. You say, ok, merge. Then you keep going. It's a meta loop. The flywheel produces the knowledge, and the second one improves the machinery, and uses us for now. The idea is actually to make that non-dependent on humans and merge by itself. Then we continue iterating on top, so many flywheels on top, kind of a pyramid. Until it's completely autonomous, and then we move on to other domains or namespaces. This is how we see that in Slack, because Slack is convenient.我或 Prasanna 可以进来并说，我们得到了所有这些分支，然后我们要求 Claude 进行挑选并查看发生了什么。它发现了非常有趣的事情。它发现了新的事实。有些，它说，这五个分支做了几乎相同的事情。它们发现了这个模式。另一个分支意识到它在 Sonnet 4.5 上运行，它决定将自己升级到 4.6，这很酷。它同时是自愈的。你说，好的，合并。然后你继续前进。这是一个元循环。飞轮产生知识，第二个改进机器，并且暂时使用我们。这个想法实际上是使它不依赖于人类并自行合并。然后我们继续在上面迭代，上面有很多飞轮，就像金字塔一样。直到它完全自主，然后我们转向其他领域或命名空间。这就是我们在 Slack 中看到它的方式，因为 Slack 很方便。

This is how it looks. For example, at each iteration, we find a few channels, and then, this happened to be a platform operations channel, a metrics/analytics engineering, all these classifications are something that it has found before. I never told it what it was, but I accepted it. I looked and spot checked. This is an alert management channel. It's not incident channels. These are general channels. We're also watching all the Slack channels as well. Because, remember, the idea is not only start with a small ontology and then go up into bigger parts of the infrastructure to describe them. This is just for general channels. This is what happens at the end of one iteration. For example, in this run it found some Turtle. Turtle is the way we serialize RDF in ontology language. It found some issues with the regex patterns, and the URIs were not saving correctly to valid Turtle.它看起来是这样的。例如，在每次迭代中，我们找到几个频道，然后，这恰好是一个平台运营频道、指标/分析工程，所有这些分类都是它以前发现的东西。我从来没有告诉它那是什么，但我接受了它。我观察并进行了抽查。这是一个告警管理频道。这不是事件频道。这些是通用频道。我们也正在观察所有的 Slack 频道。因为，记住，想法不仅是始于一个小本体，然后上升到基础设施的更大部分来描述它们。这只是针对通用频道。这是在一次迭代结束时发生的事情。例如，在这次运行中，它发现了一些 Turtle。Turtle 是我们在本体语言中序列化 RDF 的方式。它发现正则表达式模式存在一些问题，并且 URI 没有正确保存为有效的 Turtle。

It found a diff and so forth. That's what I can continue and say, you can merge this, and it goes and does it. All tests passed. The merge is complete and successful. Let me create a summary. That means for the next iteration, this code is already in the main branch. The next iteration, someone is adding a new message in the channel or a new discovery. Someone found a root cause that will actually run this new code. The next git worktree that spins up, it will already merge this fix.它发现了一个差异，等等。这就是我可以继续并说，你可以合并这个，然后它去做了。所有测试通过。合并完成且成功。让我创建一个摘要。这意味着对于下一次迭代，这段代码已经在主分支中了。下一次迭代，有人在频道中添加了一条新消息或一个新发现。有人找到了一个将实际运行这段新代码的根本原因。下一个启动的 git 工作树，它将已经合并了这个修复。

The part that extracts these chains that I talked about, it's part of our correlation engine. It finds all these chains. For example, commit happened with retry logic. It was deployed with this version. Three minutes later there was an error, connection timeout, an alert fired, and the incident was started. Then the LLM comes up with its own confidence, but we save this chain. Many chains can happen in one incident. Imagine all these chains together, we'll have a lot of data in the future for fine-tuning and similarity. Agents are actually entropy reducers. Because, like I said, we receive noisy input. We spit out triples at the end, with new relationships, but at the same time, we find better ways of extracting data, because the agentic loop is happening. We do things, for example, like SME, Subject Matter Expert detection. This person is actually a Subject Matter Expert, and we find the topic too.提取我谈到的这些链条的部分，是我们相关性引擎的一部分。它找到所有这些链条。例如，提交发生了，带有重试逻辑。它以这个版本部署。三分钟后出现错误，连接超时，触发了告警，事件开始了。然后 LLM 带着它自己的信心出现，但我们保存了这个链条。一个事件中可能会发生许多链条。想象一下所有这些链条在一起，我们将来会有很多数据用于微调和相似性。代理实际上是熵减器。因为，正如我所说，我们接收嘈杂的输入。我们在最后吐出三元组，带有新的关系，但同时，我们找到提取数据的更好方法，因为代理循环正在发生。我们做事情，例如，SME（主题专家）检测。这个人实际上是主题专家，我们也找到了主题。我们也有这些 Turtle 上的服务拓扑，因为它是已经存在的，因为我们使用 eBPF、日志和追踪从头开始构建它。我们创建了这个由我们数千个服务依赖项组成的图谱，哪个服务调用哪个服务。这是它的一部分。合约是针对丰富程序的，我们有一个类，它是一个接口，你可以在其中看到丰富部分，你可以向图谱添加三元组等等。我们有很多这样的东西，随着更多数据进来，会有更多的丰富程序。Claude 实际上会告诉我们我们需要构建一个新的丰富程序，因为我们正在看到这个我不知道的、来自之前历史的新事物等等。它不在代码中。它不在本体中。我们需要创建。实体中的每个 URI 都是稳定的。我们有域和 ID。我们总是可以通过命名空间引用它们。

The accumulation of those triples will tell us eventually, this person is an expert in the service systems, or in Kafka and so forth. We also have these service topologies on a Turtle that already exists, because we build it from the ground up from many sources, using eBPF and logs and tracing. We created this graph of all our thousands of dependencies of services, which service calls each other. That's part of it. The contract is for the enrichers, we have a class, which is an interface where you can see the enrichment part, you can add triples to the graph and so forth. We have many of these, and more enrichers come as more data come in. Claude will actually tell us we need to build a new enricher because we're seeing this new thing that I don't know about from previous history and so forth. It's not in the code. It's not in the ontology. We need to create. Every URI in the entities are stable. We have a domain and IDs. We can always reference them by namespace.从图谱，我们进入知识库。同样的过程，在收获程序的每次运行中，我们将 Turtle 文件保存在 S3 中。我们创建一个 Parquet 文件。我们有我们的图数据库，它不断地轮询该流。它实际上是一个 CDC（变更数据捕获）流，将这些三元组加载到内存中。我们可以实时查询，因为那时我们有了数据。例如，我们描述了事件。来自 Claude Code 会话、Cursor 或其他工具的其他人可以查询事件 123 发生了什么。我们已经有了那些数据。LLM 将去进行 SPARQL 查询并开始提出问题，然后也许将其与过去的其他事件相关联，等等。你使用其他 MCP 工具等等。我们也保存 Blobs。例如，当有人上传图表时，该文件将被保存在 S3 中，并且我们知道它与指标有关。

From a graph, we go into a knowledge base. The process again, at every run of the harvester, we save the Turtle files in S3. We create a Parquet file. We have our graph database that is constantly polling that stream. It's actually a CDC, change data capture stream that loads those triples in memory. We can query in real time, because then we have the data. We have described the incident, for example. Someone else from a Claude Code session, or Cursor, or whatever can be querying what's going on with incident 123. We already have that data. The LLM will go make SPARQL queries and start asking questions, and then maybe correlate it to another incident in the past, and so forth. You use other MCP tools and so forth. We also save Blobs. For example, when someone uploads a graph, that file will be saved in S3, and we know it's related to a metric.这是使用的查询。所有这些都被描述了。实际图谱，我指的是仪表板图谱，它有一个截图。截图被保存，字节在 S3 中。该 Blob 也是一个实体。一切都链接在一起。我们可以恢复所有这些数据。也许你想要进行取证分析并将其带回，并准确获取事情发生时发生了什么。本体是混乱与理解之间的合约。我们的随机引擎被限制在这个基于本体的非常强大的类型系统中，它们发出可以查询的确定性三元组。这就是我们如何从事件本体、几个命名空间、许多代理和信号开始。

This is the query used. All that is described. The actual graph, the graph I mean the dashboard graph, it has a screenshot. The screenshot is saved, the bytes in S3. That Blob is also an entity. Everything is linked. We can recover all that data. Maybe you want to do forensic analysis and bring it back and get what happened exactly when things occurred. The ontology is a contract between chaos and understanding. Our stochastic engines are constrained to this very strong type system grounded on the ontology, and they emit deterministic triples that can be queryable. That's how we started with the incident ontology, a few namespaces, many agents and signals.这是我们每次运行产生的人工制品之一。这只是一个保存在 S3 中的 HTML 文件。我们使用这个来调试，因为我刚才在看，它是否在提取正确的东西？我知道一定有 Jira 问题，或 Slack 用户或文件附件以及所有这些东西。每次运行，有时我意识到，它没有找到文件附件。我需要有一些人工制品，一些目标，因为我一直在看代码。我只是在看这些结果，因为我知道里面必须有什么。如果它不在那里，我们需要继续修复。最后，通过使用这个人工制品，我可以检查。这只是事件的时间线。一切都已类型化并链接。这是一个有很多关系的重大事件。拓扑、服务依赖关系也编码在其中。任何人都可以问任何问题，哪些团队参与了。

This is one of the artifacts that we produce on every run. This is just an HTML file saved in S3. We use this to debug, because I was just looking, is it extracting the right things? I know there has to be a Jira issue, or Slack users or file attachments and all those things. Every run, sometimes I realize, it's not finding file attachments. I need to have some artifacts, some goal, because I'm looking at the code all the time. I'm just looking at these results because I know what has to be in there. If it's not there, we need to keep fixing. Finally, by using this artifact, I could inspect. This is just a timeline of the incident. Everything is typed and linked. This is a big incident with many relationships. The topology, the service dependency is also encoded in there. Anyone can ask any questions, which teams are involved.谁在这里情绪低落？谁压力很大，所有这些？一切都链接在一起。想象一下所有这些事件在大图谱中粘合在一起，这样我们就可以同时查询它们。挑战实际上是让所有这些数据随时间流动，我们称之为时间旅行。目前，我们只是保存了几周的数据。所有这些数据都在 S3 中，所以最终我们可以得出一个非常大的数据库，我们可以随时查询。

Who has low sentiment here? Who's very stressed, and all that? Everything is linked. Imagine all these incidents gluing up in the big graph so we can query them at the same time. The challenge is actually to keep all this data flowing in time, what we call the time travel. For now, we're just saving a few weeks of data. All this data is in S3, so eventually we can come up with a really big database that we can query at any time.AutoSRE——将本体投入使用

How do we use the ontology? One of the applications is what we call the AutoSRE. It's our AI SRE solution. That's the name now in the industry, for site reliability, aiding SRE with AI. We call it that. What AutoSRE does is taps into the metric, logs, and traces in real time, in also Slack. It has the topology, it has devices as well, and ways of looking at the end-to-end graph, which is the devices. Taps into the ontology, so in the knowledge graph, which is higher creative relationships as domain concepts. Going back to the LOLOMO incident or issue, AutoSRE comes in, not only the on-call, and it analyzes all the MELT data and versions of all the A/B tests that's running, and then it does correlation much faster than humans. It finds what incidents or apps are involved in it, and so forth. It finds logs as well.它使用那个。我们有一堆 MCP 和 CLI 工具查询基础设施，不仅在本体中，而且实际上是查询一般数据。它也查看代码，正如那里的许多解决方案所做的那样，因为如果你不深入到代码中，你就无法理解系统。这就是它产生的结果。它说，此记录操作存在 GraphQL 错误激增，接下来的步骤是这些。我们实际上已经增加了交互性，用户可以提出后续问题，因为今天很难让任何 AI 在第一次尝试、第一个循环中给你正确的答案。你需要经历许多迭代来挖掘，因为 LLM 可能会游离等等。

It uses that. We have a bunch of MCPs and CLI tools querying the infrastructure, not only in ontology, but also really the data in general. It also looks at code, as many solutions out there do, because you cannot understand systems if you don't go down into the code. This is what it produces. It says, there was a GraphQL error spike for this record operation, and the next steps are these. We have actually added interactivity to this, where users can ask follow-up questions, because it's really hard to get any AI today to give you the right answer on the first go, in the first loop. You need to go through many iterations to dig in, because the LLMs can wander and stuff.未来会怎样？

What the future holds for us? There are three things. We want to find automatic root cause analysis. We don't want to keep digging. We can just give the crystal ball, this is the root cause. Next is to actually do auto-remediations. For example, if we know these patterns occur, the ontology, we'll say, in this situation, the best thing to do is just to terminate the instance. Fine. Do it. Auto-remediation. Without human interaction, even. The third one is self-healing infrastructure. That's the dream. We don't even want to get to an incident. We don't even want to get an alert. We want to keep shifting left and left, all the way to the point that the infrastructure is healing itself, because by the time we describe not only our incidents or our operational data, we will merge ontologies all the way to the data lineage to the point where, say, a movie is created.在由该团队、该人在该时间运行的流水线中存在数据错误，最终级联导致用户服务上的错误点击。这是我们想要创建的端到端梦想。不仅是形成本体，而且合并到其他更大的领域并创建这个描述一切的大型企业本体。这回到了 Prasanna 展示的原始端到端轮子。找到这个将所有较小本体连接成大的巨大循环。

There was a data error in a pipeline that was run by this team, by this person at this time, that cascaded into finally bad clicks on the user service. That's the end-to-end dream that we want to create. Not only forming the ontology, but merging into other bigger domains and create this big enterprise ontology that describes everything. That's going back to the original end-to-end wheel that Prasanna showed. Finding this huge cycle of connecting all the smaller ontologies into big.有人认识“暗黑”（Dark）中的这个角色吗？他们实际上是时间旅行者。有一个阴谋集团能够驾驭时间。他们能够进入未来，而且实际上，还有知识。这里的灵感是我正在使用他们的咒语之一。它说，“Der Anfang ist das Ende und das Ende ist der Anfang。” “开始就是结束，结束就是开始。” 这意味着观察、丰富、推理。我只是想用那个类比来结束我们的节目。

Does anyone know this character from "Dark?" They are actually time travelers. There is a cabal that were able to harness time. They were able to go in the future, and actually, also knowledge. The inspiration here is I'm using one of their mantras. It says, "Der Anfang ist das Ende und das Ende ist der Anfang." "The beginning is the end and the end is the beginning." That means observe, enrich, infer. I just wanted to end that with that analogy to our show.问答

Participant 1: How do you deal with deduplication and disambiguation in the graph on those incidents?Renzo Sanchez-Silva：现在，我们正在让 LLM 进入这个开放世界。它进行查找，它将创建三元组，并说，不，这个关系已经存在了，我们让它创建一个。它将是一个必须合并的分支，但我只是让它通过，它只是不断附加到本体。我们可能会得到许多相似的概念，也许。我们还没有检查过，但理想情况下，例如，对于这个较小的本体，过程已经停止，因为它正在变得稳定。我看到需要合并的 PR 变少了，等等。它最终确实变得稳定。它确实必须经过人类。比如本体学家需要真正看到这个，仔细观察它。我所看到的是，去重是非常受控的。这是可能的。

Renzo Sanchez-Silva: Right now, we're letting the LLM go into this open world. It does lookup, and it will create the triples, and say, no, this relationship is already there, and we let it create one. It is going to be a branch that has to be merged, but I'm just letting it pass, and it just keeps appending to the ontology. We may get to many similar concepts, maybe. We haven't checked that, but ideally, for example, for this smaller ontology, the process has stopped in the sense that it's becoming stable. I see less PRs that we need to merge, and so forth. It does eventually become stable. It does have to go through a human. Say an ontologist needs to actually see this, look at it closely. What I see is that deduplication is very controlled. This is possible.Renzo Sanchez-Silva：目前，我们让大语言模型（LLM）进入这个开放世界。它执行查找操作，创建三元组，并判断某种关系是否已存在；如果不存在，我们就让它创建一个。这会形成一个需要合并的分支，但我目前只是让它通过，它会持续向本体中追加内容。我们可能会得到许多相似的概念。虽然我们尚未进行核实，但理想情况下，例如对于这个较小的本体，该过程在趋于稳定这一层面已经停止。我看到需要合并的 PR 越来越少，诸如此类。它最终确实会变得稳定。它确实需要经过人工审核，比如需要一位本体论专家来实际查看并仔细审视。我所看到的是，去重过程得到了很好的控制。这是可以实现的。

Prasanna Vijayanathan: In the future, we also see the need to have a pattern extraction layer that can generalize patterns to help with deduplication. We're not there yet. We're able to manage.Prasanna Vijayanathan：展望未来，我们认为还需要一个模式提取层，能够归纳模式以辅助去重。我们目前还没做到这一步，但现有的手段尚能应对。

Participant 2: In these graphs, everything can start relating to everything else. How do you fine-tune your infra to give you more signal over the noise, so that the things that it outputs are actually not something that has been hallucinated or possibly related, that it's actually actionable, real, and a signal for the user.参与者 2：在这些图中，任何事物都可能开始与其他事物产生关联。你们如何微调基础设施，从而从噪声中提取更多信号，确保输出的内容不是幻觉或无关的联想，而是对用户而言切实可行、真实且有价值的信号？

Anand: How do you remove false correlations, is that possible?Anand：你们如何消除错误的相关性，这可能吗？

Prasanna Vijayanathan: Going back to old testing systems, we have a Swiss cheese model. We want to be able to give as much accurate and actionable information as possible, and the best person to tell us that would be the users right now. Like Renzo mentioned, it creates and self-heals itself, so that also exists. We have different quality metrics that we track. We do online logs, and then extract metrics from that, and then we have evals systems. It's like, we freeze our inputs and outputs, and then keep testing how our model is doing. Then we have human-in-the-loop feedback. All of those systems feed back into our model, and we haven't done that extensively yet, but it'll get into some kind of reinforcement learning to use those signals, and improve itself.Prasanna Vijayanathan：回到传统的测试系统，我们采用的是“瑞士奶酪模型”。我们希望尽可能提供准确且可操作的信息，而目前最了解这些信息的人就是用户。正如 Renzo 所提到的，它具有创建和自我修复的能力，所以这方面也是存在的。我们追踪了不同的质量指标，通过在线日志提取指标，并拥有评估系统。具体来说，我们会冻结输入和输出，不断测试模型的表现。此外，我们还有“人在回路”的反馈机制。所有这些系统都会反馈给模型，虽然我们目前还没有大规模应用，但未来会引入某种强化学习来利用这些信号，从而实现自我优化。

Participant 3: I'm a bit intrigued about the scale piece, because it seems like the whole graph is in Turtle, created by PRs like code. This is a small subset, then, of those 32 million per second queries, and we're not relating those bits. It's a much higher-level construct. Is that correct?参与者 3：我对规模化部分很感兴趣，因为整个图似乎都是用 Turtle 格式，像代码一样通过 PR 创建的。那么，这只是那每秒 3200 万次查询中的一小部分，而且我们并没有将这些片段关联起来。这是一种更高级别的结构，对吗？

Renzo Sanchez-Silva: The idea is, for this high-volume throughput, we're just going to be collecting metadata, fingerprints of something has started. We have found this pattern, but we are not going to get the data on Turtle. There's no way to do that. We're going to find the fingerprint of what's happening, and then relate that for the rest of the graph. It's just always controlling cardinality as well.Renzo Sanchez-Silva：我们的思路是，对于这种大容量吞吐量，我们只收集元数据，即事物开始的指纹。我们发现了这种模式，但不会在 Turtle 上获取数据，因为这根本做不到。我们将找到事件发生的指纹，然后将其与图的其余部分关联起来。这始终也是在控制基数。

Participant 4: How do you detect that something important didn't happen, instead of just picking up something bad that happened?参与者 4：你们如何检测到“重要事件未发生”的情况，而不是仅仅捕捉到“发生了坏事”？

Renzo Sanchez-Silva: The ontologies actually have a very powerful formal reasoning. It's not AI-related. This is actually proportional, logical. Very hard algorithms that actually can reason, can find something that is incongruent or transitive or contradictory. We also plan to use formal reasoning to catch these things as things progress. Not only to find inconsistency in data, but also to find new connections. We can do formal reasoning with non-AI, but we can also use AI to help that base reasoning.Renzo Sanchez-Silva：本体论实际上拥有非常强大的形式化推理能力。这与人工智能无关，而是比例逻辑。这些是非常严谨的算法，能够进行推理，发现不一致、传递性错误或矛盾之处。我们还计划随着进展使用形式化推理来捕捉这些问题。这不仅是为了发现数据中的不一致性，也是为了发现新的联系。我们可以利用非人工智能手段进行形式化推理，也可以利用人工智能来辅助这种基础推理。

Prasanna Vijayanathan: A human in the loop also helps a lot in that system, where a developer notices something that was not caught, or we notice an incident or an issue that came up that actually had to be solved. Then we would look at why the system didn't catch it. In fact, recently we had a case where someone flagged it and said, why did you not tell me this? Then it was able to correct itself, and then create a new pair. That's the other side of it.Prasanna Vijayanathan：“人在回路”的机制在系统中也大有裨益。当开发者注意到某些未被捕获的情况，或者我们发现了一个必须解决的突发事件或问题时，我们会分析系统为何未能捕捉到它。事实上，最近我们遇到过一个案例，有人指出“为什么没告诉我这个？”，随后系统能够自我修正并创建新的对子。这就是该机制的另一面。

Participant 5: Could you tell me how often you would allow your ontology to change? How does that affect your knowledge graph rebuild?参与者 5：你能告诉我你们允许本体论变更的频率吗？这如何影响你们的知识图谱重建？

Renzo Sanchez-Silva: We are letting it go with the open world assumption. It seems stable, at least for the incidents, but we're also going to find connections with alerts. That's a very small ontology, because all we want to catch is the metadata of the alerts. There are very few keywords or concepts in the alert, for example, and the signals. Then for other ones, like pipelines and jobs and all that, there's also a way to control that. It's very small. What is harder is the incidents, because anything can happen. The concepts are growing, but like I said, it becomes stable. At some point, you need to go back and look, we need to compress that with humans, and then it will stay controlled. We need human in the loop, for sure.Renzo Sanchez-Silva：我们采取的是“开放世界假设”。对于事故而言，它似乎是稳定的。但我们也会通过警报来寻找联系。这是一个非常小的本体，因为我们只想捕捉警报的元数据。例如，警报中的关键词或概念以及信号非常少。对于其他部分，如流水线和任务等，也有相应的控制方法。它非常小。比较困难的是事故部分，因为任何事情都可能发生。概念在不断增长，但正如我所说，它会趋于稳定。在某个阶段，你需要回过头来看看，我们需要人工介入进行压缩，这样它就能保持受控。毫无疑问，我们需要“人在回路”。

Participant 6: Any plans on open-sourcing?参与者 6：有开源计划吗？

Prasanna Vijayanathan: Maybe. Not yet.Prasanna Vijayanathan：也许吧，但目前还没有。

See more presentations with transcripts查看更多带有文字记录的演示