Two weeks ago, we launched the langchain-benchmarks package, along with a Q&A dataset over the LangChain docs. Today we’re releasing a new extraction dataset that measures LLMs' ability to infer the correct structured information from chat logs.两旬之前，我等发布了 langchain-benchmarks 包，并附以 LangChain 文档之问答数据集。今日，又推出一全新提取数据集，用以衡量大语言模型从聊天日志中推断正确结构化信息之能。
The new dataset offers a practical environment to test common challenges in LLM application development like classifying unstructured text, generating machine-readable information, and reasoning over multiple tasks with distracting information. 此新数据集，为测试大语言模型应用开发中常见之挑战，提供了实战之境。诸如非结构化文本之分类、机器可读信息之生成，以及于干扰信息中多任务推理等，皆在其中。
In the rest of this post, I'll walk through how we created the dataset and share some initial benchmark results. We hope you find this useful for your own conversational app development and would love your feedback!下文之中，我将详述数据集之创建过程，并分享初步基准测试之结果。望诸位于自身对话应用之开发有所裨益，亦盼不吝赐教！
Motivation for the dataset数据集之缘起
We wanted to design the dataset schema around a real-world problem: gleaning structured insights from chat bot interactions. 我等欲以现实问题为基，设计数据集之架构：从聊天机器人交互中，提炼结构化之洞见。
Over the summer, our excellent intern Molly helped us refresh Chat LangChain (repo), a retrieval-augmented generation (RAG) application over LangChain's python docs. It’s an “LLM with a search engine”, so you can ask it questions like "How do I add memory to an agent?”, and it will tell you an answer based on whatever it can find in the docs. 今夏，我辈杰出实习生 Molly 助我等焕新了 Chat LangChain（代码库），此乃基于 LangChain Python 文档之检索增强生成应用。其犹如“内嵌搜索引擎之大语言模型”，可问诸如“如何为智能体添加记忆？”之类问题，模型便依据文档中所寻，作答以告。
The real test of such a project begins post-deployment, when you begin to observe how it's used and refine it further. Typically, users won't provide explicit feedback, but their conversations reveal a lot, and while you can try just “putting the logs into an LLM” to summarize it, you can also often benefit from extracting structured content to monitor and analyze. This could help drive analytic dashboards or fine-tuning data collection pipelines, since the structured values can easily be used by traditional software.此类项目之真正考验，始于部署之后。彼时，你方始观察其如何被用，并加以精进。通常，用户不会明示反馈，然其对话却泄露诸多信息。虽可尝试“将日志投入大语言模型”以作总结，但若能提取结构化内容以监控分析，则往往获益更多。此举可助驱动分析仪表盘或微调数据收集管道，因结构化数值易于传统软件所用。
The Chat Extraction dataset is designed around testing how well today's crop of LLMs are able to extract and categorize relevant information from this type of data. In the following section, I’ll walk through how we created the dataset. If you just want to see the results, check out the summary graph below. You can feel free to jump to the experiments section for an analysis of the results.聊天提取数据集，旨在测试当今一众大语言模型，从此类数据中提取并分类相关信息之能力。下节之中，我将详述数据集之创建过程。若君只欲观其结果，可直览下方汇总图表。亦可径跳至实验章节，以观结果之分析。
Creating the Dataset数据集之创建
The main steps for creating the dataset were:创建数据集之主要步骤如下：
- Settle on a data model to represent the structured output.定下数据模型，以表征结构化输出。
- Seed with Q&A pairs.以问答对作为种子。
- Generate candidate answers using an LLM.使用大语言模型生成候选答案。
- Manually review the results in the annotation queue, updating the taxonomy where necessary.在标注队列中人工审核结果，必要时更新分类体系。
LangChain has long had synthetic dataset generation utilities that help you bootstrap some initial data, but the final version should always involve some amount of human review to ensure proper quality. That’s why we’ve added data annotation queue’s to LangSmith and will continue to improve our tooling to help you build your data flywheel.LangChain 早有合成数据集生成工具，可助你启动初始数据。但最终版本，总需一定量的人工审核，以确保质量。正因如此，我等在 LangSmith 中加入了数据标注队列，并将持续改进工具，助你构建数据飞轮。
Once you have an initial dataset, you can use the labeled data as few-shot examples within the seed-generation model to improve the quality of data given to humans for review. This can help reduce the amount of work and changes needed when updating the ground truth.一旦拥有初始数据集，便可将标注数据作为种子生成模型中的少样本示例，以提升提交人工审核之数据质量。此举可减少更新真实答案时所需之工作量与改动。
Extraction Schema提取之架构
We wanted the task to be tractable while still offering a challenge for many common models today. We defined the schema using this linked pydantic model. An example extracted value is below:我等欲使任务可解，同时仍对当今常见模型构成挑战。我们以此链接之 pydantic 模型定义了架构。下方为提取值之示例：
{
"GenerateTicket": {
"question": {
"toxicity": 0,
"sentiment": "Neutral",
"is_off_topic": false,
"question_category": "Function Calling",
"programming_language": "unknown"
},
"response": {
"response_type": "provide guidance",
"confidence_level": 5,
"followup_actions": [
"Check with API provider for function calling support."
]
},
"issue_summary": "Function Calling Format Validation"
}
}Example Extracted Output提取输出之示例
Many of these values could be useful in monitoring an actual production chat bot. We made the schema challenging in a few ways to make the benchmark results more useful in separating model capacity and functionality. Some challenges about this schema include:其中许多值，于监控实际生产环境之聊天机器人时，或有用处。我等在若干方面使架构具有挑战性，以使基准测试结果更能区分模型能力与功能。此架构之挑战包括：
- It includes a couple fairly long Enum values. Even OpenAI's function calling/tool usage API can be imperfect in generating these.包含数个颇长之枚举值。即便 OpenAI 的函数调用/工具使用 API，在生成这些值时亦非完美。
- The object is nested - nesting can make it harder for LLMs to stay coherent if they aren't trained on code.对象为嵌套结构——若大语言模型未经代码训练，嵌套结构易使其难以保持连贯。
- The values in each nested component are meant to be inferred only from the corresponding sections of input (response or question).各嵌套组件中之值，应仅从输入之对应部分（回答或问题）推断得出。
- It combines classification, summarization, and structured output generation in a single task.它将分类、摘要与结构化输出生成，合于单一任务之中。
If "attention is all you need", by splitting the attention of the model, this multi-task objective can be challenging for an LLM to address in a single generation. 若“注意力即一切”，则通过分散模型之注意力，此多任务目标对大语言模型而言，于单次生成中应对，颇具挑战。
Evaluation评估
This benchmark is focused on structure and classification, and as such, we don't need to use any LLM-as-a-judge metrics. Instead, we wrote custom LangSmith evaluators (see the code definition here). Below is what we measured:此基准测试聚焦于结构与分类，故无需使用任何大语言模型作为评判之指标。反之，我等编写了自定义 LangSmith 评估器（代码定义见此）。以下为所测量之内容：
- Structure verification
json_schema: 1 if correct, 0 if not. We validate the parsed output for each model using the task schema. json_schema：正确为 1，否则为 0。我们使用任务架构验证每个模型之解析输出。
- Classification tasks
question_category: classification accuracy over the 25 valid enum values. question_category：针对 25 个有效枚举值之分类准确率。off_topic_similarity: binary classification accuracy of whether the LLM considered the question off-topicoff_topic_similarity：大语言模型是否认为问题离题之二元分类准确率。toxicity_similarity: normalized difference in predicted level of "toxicity" of the user question.toxicity_similarity：用户问题预测“毒性”级别之归一化差异。programming_language_similarity- classification accuracy of the predicted programming language the user's question references. In most cases, this is "unknown".programming_language_similarity：预测用户问题所涉编程语言之分类准确率。多数情况下为“未知”。confidence_level_similaritythe normalized similarity between the predicted "confidence" of the response and the labeled confidence.confidence_level_similarity：预测回答之“置信度”与标注置信度之间之归一化相似度。sentiment_similarity- Normalized difference between the prediction and label. Sentiment is scored as 0/1/2 for negative/neutral/positive. sentiment_similarity：预测与标注之间之归一化差异。情感以 0/1/2 分别表示负面/中性/正面。
- Overall difference
json_edit_distance: this is a bit of a catch-all that first canonicalizes the predicted json and label json and then computes the Damerau-Levenshtein string distance between the two serialized forms.json_edit_distance：此乃一个综合指标，首先将预测 JSON 与标注 JSON 规范化，然后计算两者序列化形式之间之 Damerau-Levenshtein 字符串距离。
Experiments实验
In making this dataset, we wanted to answer a few questions:在制作此数据集时，我等欲回答几个问题：
- How do the most popular closed-source LLMs compare?最流行之闭源大语言模型之间，孰优孰劣？
- How well do off-the-shelf open source LLMs perform relative to the closed-source models?现成之开源大语言模型，与闭源模型相比，表现如何？
- How effective are simple prompting strategies improving extraction performance?简单之提示策略，对提升提取性能，效果几何？
- If we control the LLM grammar to output a valid record, how significant is this for the individual classification metrics? 若控制大语言模型之语法，使其输出有效记录，则对各项分类指标，影响几何？
We evaluated the following LLMs:我等评估了以下大语言模型：
gpt-4-1106-previewthe recent long-context, distilled version of GPT-4.gpt-4-1106-preview：近期之长上下文、蒸馏版 GPT-4。claude-2- an LLM from Anthropic.claude-2：来自 Anthropic 之大语言模型。llama-v2-34b-code-instruct- a 34b parameter variant of Code Llama 2 fine-tuned on an instruction dataset.llama-v2-34b-code-instruct：Code Llama 2 之 34b 参数变体，经指令数据集微调。llama-v2-chat-70b- a 70b parameter variant of Llama 2 fine-tuned for chat.llama-v2-chat-70b：Llama 2 之 70b 参数变体，专为聊天微调。yi-34b-200k-capybara- a 34b parameter model from Nous Research.yi-34b-200k-capybara：来自 Nous Research 之 34b 参数模型。
Experiment 1: GPT vs. Claude实验一：GPT 对阵 Claude
We first compared Claude-2 and GPT-4, both closed-source LLMs. For GPT-4, we used its too-calling API, which lets you provide a JSON schema for it to populate. Since Anthropic has yet to release a similar tool-calling API, we tested two different ways of specifying the schema: 我等首先比较了 Claude-2 与 GPT-4，二者皆为闭源大语言模型。对于 GPT-4，我等使用了其工具调用 API，该 API 允许你提供 JSON 架构供其填充。由于 Anthropic 尚未发布类似之工具调用 API，我等测试了两种指定架构之方式：
- Directly as a Json schema.直接作为 JSON 架构。
- As an XSD (XML schema)作为 XSD（XML 架构）。
You can review the individual predictions side-by-side at the linked tests. You can also check out the summary graph and table below:你可于链接之测试中，并排查看各预测结果。亦可查阅下方之汇总图表与表格：
Comparing GPT-4 and ClaudeGPT-4 与 Claude 之比较
Test测试
confidence_level_similarityconfidence_level_similarity
json_edit_distancejson_edit_distance
json_schemajson_schema
off_topic_similarityoff_topic_similarity
programming_language_similarityprogramming_language_similarity
question_categoryquestion_category
sentiment_similaritysentiment_similarity
toxicity_similaritytoxicity_similarity
claude-2-xsd-to-xml-5689claude-2-xsd-to-xml-5689
0.970.97
0.390.39
0.520.52
0.000.00
0.52
0.370.37
0.910.91
1.01.0
claude-2-json-schema-to-xml-5689claude-2-json-schema-to-xml-5689
0.97
0.37
0.780.78
0.00
0.440.44
0.480.48
0.930.93
1.0
gpt-4-1106-preview-5689gpt-4-1106-preview-5689
0.940.94
0.280.28
1.001.00
0.890.89
0.590.59
0.560.56
1.00
0.00.0
As expected, GPT-4 performs better across almost all metrics, and we were unable to get Claude to perfectly output the desired schema in a single shot. Interestingly enough, the Claude model prompted with a JSON schema does slightly better than the one prompted with the same information provided in an XSD (XML schema), indicating that at least in this case, consistent formatting of the schema isn't that important. 果如所料，GPT-4 在几乎所有指标上表现更佳，且我等未能使 Claude 在单次生成中完美输出所需架构。有趣的是，以 JSON 架构提示之 Claude 模型，略优于以 XSD（XML 架构）提供相同信息之模型，表明至少在此例中，架构之格式一致性并非至关重要。
It's easy to see some common schema issues; for instance, in this run and this run, the model outputs a bullet-point list for the follow-up actions rather than properly tagged elements, which was parsed as a string rather than a list. Below is an example image of this:常见架构问题易见；例如，在此次运行与此次运行中，模型为后续行动输出项目符号列表，而非正确标记之元素，导致被解析为字符串而非列表。下方为此问题之示例图：
While we can fix these parsing errors on a case-by-case basis, the unpredictability hinders the overall development experience. There's more overhead in adapting one extraction chain to another task since the parser and other behavior is less consistent. The XML syntax also increases the overall token usage of Claude relative to GPT. Though "tokens" aren't directly comparable, verbose syntaxes will likely lead to slower response times and higher costs.虽可逐案修复这些解析错误，但不可预测性阻碍了整体开发体验。将一条提取链适配至另一任务时，由于解析器及其他行为不够一致，需更多额外工作。XML 语法亦使 Claude 之令牌用量相对于 GPT 增加。虽“令牌”不可直接比较，但冗长之语法很可能导致更慢之响应时间与更高之成本。
Experiment 2: Open-Source Models实验二：开源模型
We next wanted to benchmark popular open-source models off-the shelf, and started out by comparing the same basic prompt across three models:我等接下来欲对现成之流行开源模型进行基准测试，首先比较了三个模型上相同之基础提示：
llama-v2-34b-code-instruct- a 34b parameter variant of Code Llama 2 fine-tuned on an instruction dataset.llama-v2-chat-70b- a 70b parameter variant of Llama 2 fine-tuned for chat.yi-34b-200k-capybara- a 34b parameter model from Nous Research.
Check out the linked comparisons to see the outputs in LangSmith, or reference the aggregate metrics below:请查阅链接之比较，以在 LangSmith 中查看输出，或参考下方之聚合指标：
Compare Baseline OSS Models基础开源模型之比较
Test
confidence_level_similarity
json_edit_distance
json_schema
off_topic_similarity
programming_language_similarity
question_category
sentiment_similarity
toxicity_similarity
yi-34b-200k-capybara-5d76-v1yi-34b-200k-capybara-5d76-v1
-0.30-0.30
0.160.16
0.37
0.410.41
0.150.15
0.15
0.28
0.41
llama-v2-70b-chat-28a7-v1llama-v2-70b-chat-28a7-v1
0.300.30
0.430.43
0.040.04
0.30
0.15
0.04
0.30
0.00
llama-v2-34b-code-instruct-bcce-v1llama-v2-34b-code-instruct-bcce-v1
0.93
0.41
0.89
0.89
0.44
0.070.07
0.59
1.00
Despite its larger model size, the 70B variant of Llama 2 did not reliably output JSON, since the amount of code included in its pretraining and SFT corpus was low. Yi-34b was more reliable in this regard, but it still only matched the required schema 37% of the time. It also performs better on the hardest of the classification tasks, the question_category classification.尽管模型规模更大，Llama 2 之 70B 变体未能可靠输出 JSON，因其预训练与监督微调语料中代码含量较低。Yi-34b 在此方面更为可靠，但仍仅 37% 之时间符合所需架构。其在最难之分类任务——问题类别分类上，表现亦更佳。
The 34B Code Llama 2 was able to output valid JSON and did a decent job for the other metrics, so we will use it as the baseline for the following prompt experiments.34B Code Llama 2 能够输出有效 JSON，并在其他指标上表现尚可，故我等将以其作为后续提示实验之基准。
Experiment 3: Prompting for Schema Compliance实验三：提示以符合架构
Of the three open model baselines, the 34B Code Llama 2 variant performed the best. Because of this, we selected it to answer the question "how well do simple prompting techniques work in getting the model to output reliably structured JSON" (hint: not very well). You can re-run the experiments using this notebook.在三个开源模型基准中，34B Code Llama 2 变体表现最佳。因此，我等选择它来回答“简单提示技术在使模型输出可靠结构化 JSON 方面效果如何”（提示：并不太好）。你可使用此笔记本重新运行实验。
In the baseline experiments, the most common failure mode was hallucination of invalid Enum values (see for example, this run), as well as poor classification performance for simple things like question sentiment.在基础实验中，最常见之失败模式是幻觉出无效枚举值（例如，参见此次运行），以及对于简单事物如问题情感之分类表现不佳。
We tested three prompting strategies to see how they impact the aggregate performance:我等测试了三种提示策略，以观察其对聚合性能之影响：
- Adding additional task-specific instructions: the schema already has descriptions for each value, but we wanted to see if additional instructions to e.g., carefully select a valid Enum values from the list, would help. We had tested this approach on a couple of playground examples and saw that it could occasionally help.添加额外任务特定指令：架构已为每个值包含描述，但吾等欲知额外指令（例如，仔细从列表中选择有效枚举值）是否有帮助。我等已在若干 playground 示例上测试此方法，并发现其偶尔有效。
- Chain-of-thought: Ask the model to think step by step about the schema structure before generating the final output.思维链：要求模型在生成最终输出前，逐步思考架构结构。
- Few-shot examples: We hand-crafted expected input-output pairs for the model to follow, in addition to the explicit instructions and schema. Sometimes LLMs (like people) learn better by seeing a few examples rather than from instructions.少样本示例：我等手工制作了期望之输入输出对，供模型遵循，此外还有显式指令与架构。有时大语言模型（如人）通过观察几个示例，比从指令中学习得更好。
Below are the results:下方为结果：
Compare Prompt Strategies for OSS Models开源模型提示策略之比较
Test
Prompt提示
confidence_level_similarity
json_edit_distance
json_schema
off_topic_similarity
programming_language_similarity
question_category
sentiment_similarity
toxicity_similarity
llama-v2-34b-code-instruct-bcce-v1
baseline基础
0.93
0.41
0.89
0.89
0.44
0.07
0.59
1.00
llama-v2-34b-code-instruct-e20e-v1llama-v2-34b-code-instruct-e20e-v1
instructions指令
0.950.95
0.380.38
0.960.96
0.89
0.630.63
0.110.11
0.540.54
1.00
llama-v2-34b-code-instruct-34b8-v2llama-v2-34b-code-instruct-34b8-v2
few-shot少样本
0.89
0.38
0.59
0.850.85
0.330.33
0.07
0.85
0.96
llama-v2-34b-code-instruct-d3a3-v2llama-v2-34b-code-instruct-d3a3-v2
CoT思维链
0.97
0.420.42
0.85
0.85
0.44
0.04
0.570.57
0.810.81
None of the prompting strategies demonstrate meaningful improvements on the metrics in question. The few-shot examples technique even decreases performance of the model on the JSON Schema test (see: example). This may be because we are increasing the amount of content in the prompt that distracts from the raw schema. Making the instructions explicit does seem to improve the performance of the programming language classification, since the model is instructed to focus on the question. The contribution is minor, however, and for the sentiment classification metric, the model continues to get distracted by the response sentiment.所有提示策略均未在相关指标上展现出有意义之改进。少样本示例技术甚至降低了模型在 JSON 架构测试上之性能（参见示例）。这可能是因为我们增加了提示中分散原始架构注意力之内容量。使指令显式化确实似乎改善了编程语言分类之性能，因为模型被指示专注于问题。然而，贡献甚微，且对于情感分类指标，模型仍持续被回答情感所干扰。
Experiment 4: Structured Decoding实验四：结构化解码
Since none of the prompting techniques offer a significant boost to the structure of the model output, we wanted to test other ways to reliably generate schema-compliant JSON. Specifically, we wanted to apply structured decoding techniques such as logit biasing / constraint-based sampling. For a survey on guided text generation, check out Lilian Weng's excellent post. 由于所有提示技术均未对模型输出之结构带来显著提升，我等欲测试其他方法以可靠生成符合架构之 JSON。具体而言，我等欲应用结构化解码技术，如 logit 偏置/基于约束之采样。关于引导文本生成之综述，请查阅 Lilian Weng 之精彩文章。
In this experiment, we test Llama 70B using Llama.cpp's grammar-based decoding mechanism to guarantee a valid JSON schema. See the comparison with the baseline here and in the table below.在此实验中，我等使用 Llama.cpp 基于语法之解码机制测试 Llama 70B，以保证有效 JSON 架构。请参见此处与基础之比较，以及下方表格。
Compare Baseline vs. Grammar-based Decoding基础与基于语法解码之比较
Test
Decoding解码
confidence_level_similarity
json_edit_distance
json_schema
off_topic_similarity
programming_language_similarity
question_category
sentiment_similarity
toxicity_similarity
baseline
0.30
0.43
0.04
0.30
0.15
0.04
0.30.3
0.0
llama-gguf-1f95-v2llama-gguf-1f95-v2
structured结构化
0.93
0.44
1.00
0.89
0.37
0.260.26
1.0
1.0
The most noticeable (and expected) improvement is that the json_schema correctness went from almost never correct to 100% validity. This means that the other values also could be reliably parsed, leading to fewer 0's in these fields. Since the base Llama 70B chat model is also larger and more capable than our previous 34B model experiments, we can see improvements in the sentiment similarity and question category as well. However, the absolute performance in these metrics is still low. Grammar-based decoding makes the output structure guaranteed, but it alone is insufficient to guarantee the quality of the values themselves. 最显著（且预期之中）之改进是 json_schema 正确率从几乎从不正确变为 100% 有效。这意味着其他值亦可可靠解析，导致这些字段中零值减少。由于基础 Llama 70B 聊天模型亦比先前之 34B 模型实验更大且能力更强，我等可看到情感相似度与问题类别方面之改进。然而，这些指标之绝对性能仍然较低。基于语法之解码保证了输出结构，但仅此不足以保证值本身之质量。
Full Results完整结果
For the full results for the above experiments, check out the LangSmith test link. You can also run any of these benchmarks against your own model by following the notebook here. 关于上述实验之完整结果，请查阅 LangSmith 测试链接。你亦可按照此笔记本，针对自己之模型运行任何这些基准测试。
