Agents may be the “killer” LLM app, but building and evaluating agents is hard. Function calling is a key skill for effective tool use, but there aren’t many good benchmarks for measuring function calling performance. Today, we are excited to release four new test environments for benchmarking LLMs’ ability to effectively use tools to accomplish tasks. We hope this makes it easier for everyone to test different LLM and prompting strategies to show what enables the best agentic behavior.代理或许是“杀手级”的语言模型应用，但构建和评估代理却并非易事。函数调用是有效使用工具的关键技能，然而衡量函数调用性能的优秀基准却不多。今天，我们很高兴发布四个新的测试环境，用于评测大型语言模型有效使用工具完成任务的能力。我们希望这能让大家更容易地测试不同的语言模型和提示策略，以展现何种因素能带来最佳的代理行为。
We designed these tasks to test capabilities we consider to be prerequisites for common agentic workflows, such as planning / task decomposition, function calling, and the ability to override pre-trained biases when needed. If an LLM is unable to solve these types of tasks (without explicit fine-tuning), it will likely struggle to perform and generalize reliably for other workflows where “reasoning” is required. Below are some key take-ways for those eager to see our findings:
我们设计这些任务是为了测试那些我们认为是常见代理工作流的先决条件的能力，例如规划/任务分解、函数调用，以及在需要时覆盖预训练偏差的能力。如果一个大型语言模型无法解决这类任务（未经显式微调），它很可能在需要“推理”的其他工作流中难以可靠地执行和泛化。以下是我们发现的一些关键要点：
💡
Key Findings - All of the models can fail over longer trajectories, even for simple tasks.- GPT-4 got the highest score on the Relational Data task, which most closely approximates common usage.- GPT-4 seems to be worse than GPT-3.5 on the Multiverse Math task; it's possible its pretrained bias hinders its performance in an example of inverse scaling.- Claude-2.1 performs within the error bounds of GPT-4 for 3 of 4 tasks, though seems to lag GPT-4 on the relational data task.- Despite outputting well-formatted tool invocations, AnyScale’s fine-tuned variant of Mistral 7b struggles to reliably compose more than 2 calls. Future open-source function calling efforts should focus on function composition in addition to single-call correctness.- In addition to model quality, service reliability is important. We ran into frequent random 5xx errors from the most popular model providers.关键发现：所有模型在较长的执行路径中都可能失败，即使是简单的任务。- GPT-4 在关系数据任务上得分最高，该任务最接近常见用法。- GPT-4 在多重宇宙数学任务上似乎不如 GPT-3.5；其预训练偏差可能阻碍了其在该任务上的表现，这是一种反向缩放的例子。- Claude-2.1 在 4 个任务中的 3 个任务上的表现均在 GPT-4 的误差范围内，但在关系数据任务上似乎落后于 GPT-4。- 尽管输出了格式良好的工具调用，但 AnyScale 对 Mistral 7b 的微调版本在可靠地组合超过 2 个调用方面遇到了困难。未来的开源函数调用工作应侧重于函数组合以及单次调用的正确性。- 除了模型质量，服务可靠性也很重要。我们遇到了最受欢迎的模型提供商频繁出现的随机 5xx 错误。
🦜
So what?- Superhuman model knowledge doesn't help if your task or knowledge differs significantly from its pre-training. Validate the LLM you choose on the behavior patterns you need it to excel on before deploying.- Planning is still hard for LLMs - the likelihood of failure increases with the number of required steps, even for simple tasks.- Function calling makes it easy to get 100% schema correctness, but that’s not sufficient for task correctness. If you are fine-tuning a model for agent use, it's imperative that you train on multi-step trajectories.那么，这意味着什么？- 如果你的任务或知识与模型的预训练大相径庭，那么超乎寻常的模型知识也无济于事。在部署你选择的语言模型之前，请先在你需要的行为模式上对其进行验证。- 对于语言模型来说，规划仍然是困难的——即使是简单的任务，失败的可能性也随着所需步骤的增加而增加。- 函数调用可以轻松实现 100% 的模式正确性，但这不足以保证任务的正确性。如果你正在为代理使用微调模型，那么在多步执行路径上进行训练是必不可少的。
In the rest of this post, we’ll walk through each task and communicate some initial benchmark results.在本文的其余部分，我们将逐一介绍每个任务，并分享一些初步的评测结果。
Experiment overview实验概述
In this release, we are sharing results and code to reproduce these experiments for 7 models across the 4 tool usage tasks:在此次发布中，我们分享了 7 个模型在 4 个工具使用任务上的结果和复现这些实验的代码：
- Typewriter (Single tool): sequentially call a single tool to type out a word.打字机（单工具）：顺序调用单个工具来打出单词。
- Typewriter (26 tools): call different tools in sequence to type out a word.打字机（26 个工具）：顺序调用不同的工具来打出单词。
- Relational Data: answer questions based on information in three tables.关系数据：根据三个表中的信息回答问题。
- Multiverse Math: use tools to answer math problems, where the underlying math rules have changed slightly.多重宇宙数学：使用工具回答数学问题，其中底层的数学规则略有改变。
We calculate four metrics across these tasks:我们在这几项任务中计算了四个指标：
- Correctness (compared to the ground truth) - this uses an LLM as a judge. Since the answers for all these questions are concise and fairly binary, we found the judgements to correspond to our own decisions.正确性（与真实值相比）——这使用了语言模型作为裁判。由于所有这些问题的答案都简洁且相当二元，我们发现裁判的判断与我们自己的决定一致。
- Correct final state (environment) - for the typewriter tasks, each tool invocation updates the world state. We directly check the equivalence of the environment at the end of each test row.正确的最终状态（环境）——对于打字机任务，每次工具调用都会更新世界状态。我们直接检查每行测试结束时环境的等价性。
- Intermediate step correctness - each data point has an optimal sequence of function calls to obtain the correct answer. We directly check the order of function calls against the ground truth.中间步骤的正确性——每个数据点都有一个最优的函数调用序列来获得正确答案。我们将函数调用的顺序与真实值进行直接比较。
- Ratio of steps taken to the expected steps - it may be that an agent ultimately returns the correct answer despite choosing a suboptimal set of tools. This metric will reflect discrepancies without being as strict as the exact match intermediate step. correctness. 所用步骤与预期步骤的比例——代理最终可能返回正确答案，尽管它选择了一组次优的工具。此指标将反映差异，但不如精确匹配的中间步骤正确性严格。
We compared both closed source models as well as open source. Expand the section below for more details.我们比较了闭源模型和开源模型。展开下面的部分以获取更多详细信息。
Models Tested测试模型
Open Source:开源：
- Mistral-7b-instruct-v0.1: Mistral’s 7B parameter model adapted by Anyscale for function calling.Mistral-7b-instruct-v0.1：Mistral 的 7B 参数模型，由 Anyscale 适配用于函数调用。
- Mixtral-8x7b-instruct: Mistral's 7B parameter mixture of experts model, adapted using instruction tuning by Fireworks.ai.Mixtral-8x7b-instruct：Mistral 的 7B 参数专家混合模型，由 Fireworks.ai 使用指令微调进行适配。
OpenAI - (Tool Calling Agent)OpenAI - (工具调用代理)
- GPT-3.5-0613GPT-3.5-0613
- GPT-3.5-1106-previewGPT-3.5-1106-preview
- GPT-4-0613GPT-4-0613
- GPT-4-1106-previewGPT-4-1106-preview
AnthropicAnthropic
- Claude 2.1 using XML prompting and its tool user library.Claude 2.1 使用 XML 提示和其工具用户库。
⌨ Typewriter (Single tool)⌨ 打字机（单工具）
The typewriter tasks are simple: the agent must “type” a given word and then stop. Words range from easy (a or cat ) to a tad harder (communication and keyboard). In the single-tool setting, the model is given a single type_letter tool that accepts a character as input. To pass, all the agent has to do is call the tool for each letter in the right sequence. For instance, for cat, the agent would execute:打字机任务很简单：代理必须“打出”给定的单词然后停止。单词的难度从容易（a 或 cat）到稍难（communication 和 keyboard）。在单工具设置中，模型被提供一个名为 `type_letter` 的工具，该工具接受一个字符作为输入。要通过，代理只需按正确的顺序为每个字母调用该工具。例如，对于“cat”，代理将执行：
You can check out the full dataset at this link to get a sense of what it looks like and see the doc for more information on how to run this task yourself.您可以在此链接查看完整数据集，以了解其外观，并查阅文档以获取有关如何自行运行此任务的更多信息。
Acing such a simple task is table stakes for any self-respecting agent, and you’d expect large models like gpt-4 with tool-calling to sail through with flying colors, but we found this to not always be the case! Take for instance this example, where the agent simply refuses to try to type the word "keyboard", or this example, where it doesn't recognize the word provided ("head"). 完成这样一个简单的任务是任何有自尊的代理的基本要求，您会期望像 gpt-4 这样具有工具调用能力的大型模型能够轻松通过，但我们发现事实并非总是如此！例如，在这个例子中，代理根本拒绝尝试输入单词“keyboard”，或者在这个例子中，它不认识提供的单词（“head”）。
Below are the results for this task across the tested agents:以下是该任务在测试代理上的结果：
The chart above shows the average correctness for each agent over the given dataset. The error bars are the standard error: 上图显示了每个代理在给定数据集上的平均正确率。误差线是标准误差：
\( \text{Standard Error} = \hat{p}\pm\sqrt{\frac{p \times (1 - p)}{n}}\)\( \text{Standard Error} = \hat{p}\pm\sqrt{\frac{p \times (1 - p)}{n}}\)
We were surprised by the poor performance of the fine-tuned mistral-7b-instruct-v0.1 model. Why does it struggle for this task? Let's review one of its runs to see where it could be improved. For the data point "aaa" (see linked run), the model first invokes "a", then responds in text "a" with a mis-formatted function call for the letter "b". The agent then returns.我们对 fine-tuned mistral-7b-instruct-v0.1 模型糟糕的表现感到惊讶。它为什么会在此任务上遇到困难？让我们回顾一下它的一次运行，看看可以在哪里改进。对于数据点“aaa”（参见链接的运行），模型首先调用“a”，然后以文本“a”响应，并带有错误的字母“b”的函数调用。然后代理返回。
The image above is taken from the second LLM invocation, after it has succesfully typed the letter "a". Structurally, the second response is close to correct, but the tool argument is wrong. 上图取自第二次语言模型调用，在成功输入字母“a”之后。结构上，第二次响应接近正确，但工具参数是错误的。
⌨️ Typewriter (26 tools)⌨️ 打字机（26 个工具）
You’ll likely want your agent to be able to use more than one tool in your application, but will it be able to use them all effectively? How much is too much?您很可能希望您的代理能够在您的应用程序中使用多个工具，但它能否有效地使用所有工具？多少算太多？
The 26-tool typewriter task tests the same thing as the single-tool use case: is the agent able to type the provided word using the provided tools (and then stop)? The difference here is that the agent must select between each of 26 tools, 1 for each letter in the English alphabet. None of the tools accept any arguments. Our cat example above would be passed by doing the following:26 个工具的打字机任务测试的内容与单工具用例相同：代理是否能够使用提供的工具打出给定的单词（然后停止）？这里的区别在于代理必须在 26 个工具之间进行选择，每个工具对应英文字母表中的一个字母。没有一个工具接受任何参数。我们上面“cat”的例子可以通过以下方式通过：
The dataset for this task uses the same test input words as the dataset for the single-tool typewriter. You can check out the dataset at this link and review the task documentation for more details, including how to run your own agent on this benchmark. 此任务的数据集使用与单工具打字机数据集相同的测试输入单词。您可以在此链接查看数据集，并查阅任务文档以获取更多详细信息，包括如何在此基准上运行您自己的代理。
Once again, you'd assume this task to be trivial for a powerful model like gpt-4, but you'd once again be proven incorrect. Take this run as an example. When asked to type "aaaa", it types the four a's out at first but then fails to halt, typing "a" 4 more times before deciding it is done.再一次，您会认为像 gpt-4 这样的强大模型会轻松完成这个任务，但您又一次被证明是错误的。以这个运行为例。当被要求输入“aaaa”时，它首先打出四个“a”，但随后未能停止，又多打了 4 次“a”之后才决定完成。
Below are the results for this task across the tested agents:
🕸️ Relational Data🕸️ 关系数据
A helpful AI assistant should be able to reason about objects and their relationships. Answering a real-world question usually requires synthesizing responses from disparate sources, but how reliable are LLMs at "thinking" in this way?一个有用的 AI 助手应该能够推理对象及其关系。回答一个现实世界的问题通常需要综合来自不同来源的响应，但语言模型在这方面的“思考”能力有多可靠？
In the relational data task, the agent must answer questions based on data contained across 3 relational tables. To use the tools, it is given the following instructions:在关系数据任务中，代理必须根据包含在 3 个关系表中的数据来回答问题。要使用工具，它会收到以下说明：
Please answer the user's question by using the tools provided. Do not guess the answer. Keep in mind that entities like users, foods and locations have both a name and an ID, which are not the same.请使用提供的工具回答用户的问题。不要猜测答案。请记住，用户、食物和地点等实体都有名称和 ID，它们是不同的。
The agent can query these tables for the correct answer using a set of 17 tools at its disposal. The three tables contain information about users, locations and foods, respectively. Of all the synthetic datasets released today, this dataset most closely resembles tool usage in real-life web applications.代理可以使用其可用的 17 个工具集来查询这些表以获取正确答案。这三个表分别包含有关用户、地点和食物的信息。在今天发布的所有合成数据集中，此数据集最接近真实网络应用程序中的工具使用情况。
Using the data in the tables, it’s possible to answer questions like: “what can you tell me about Alice?” or “is it likely that Alice needs an umbrella now?” (sample data shown below). 利用表中的数据，可以回答诸如“你能告诉我关于爱丽丝的什么信息？”或“爱丽丝现在需要雨伞的可能性大吗？”之类的问题（下面显示了示例数据）。
Below is an illustration of the latter question:下面是后一个问题的说明：
For this example, the agent selects the following 3 tools from the 17 at its disposal:对于这个例子，代理从其可用的 17 个工具中选择了以下 3 个工具：
find_users_by_name(name)→ search for users by name.find_users_by_name(name)→ 按名称搜索用户。get_user_location(user_id)→ look up the given user’s favorite color.get_user_location(user_id) → 查询给定用户的最喜欢的颜色。get_weather_at_location(location_id)→ get the weather at the given location.get_weather_at_location(location_id) → 获取给定地点的天气。
By looking at the first 2 records of each table, we can see the results that these function calls will return:通过查看每个表的头两条记录，我们可以看到这些函数调用将返回的结果：
Users用户
id
name姓名
email电子邮件
location地点
favorite_color喜欢的颜色
favorite_foods喜欢的食物
1
Alice爱丽丝
alice@gmail.comalice@gmail.com
1
red红色
[1, 2, 3][1, 2, 3]
21
Bob鲍勃
bob@hotmail.combob@hotmail.com
2
orange橙色
[4, 5, 6][4, 5, 6]
Locations地点
id
city城市
current_time当前时间
current_weather当前天气
1
New York纽约
2023-11-14 10:30 AM2023-11-14 10:30 AM
Partly Cloudy, Temperature: 68°F部分多云，温度：68°F
2
Los Angeles洛杉矶
2023-11-14 7:45 AM2023-11-14 7:45 AM
Sunny, Temperature: 75°F晴朗，温度：75°F
Foods食物
id
name
calories卡路里
allergic_ingredients过敏原成分
1
Pizza披萨
285285
["Gluten", "Dairy"]["谷蛋白", "乳制品"]
2
Chocolate巧克力
50
["Milk", "Soy"]["牛奶", "大豆"]
The agent first retrieves Alice's user ID, then uses that user ID to fetch the current location, and finally it uses the location ID to fetch the current weather. Since the current weather in Alice's location is partly cloudy, it is unlikely that she will need an umbrella. If the agent skips any of these steps, it will lack the required information to accurately provide the final answer.代理首先检索爱丽丝的用户 ID，然后使用该用户 ID 获取当前地点，最后使用地点 ID 获取当前天气。由于爱丽丝所在地的当前天气是部分多云，她不太可能需要雨伞。如果代理跳过其中任何一个步骤，它将缺乏准确提供最终答案所需的信息。
The evaluation dataset consists of 20 questions of varying difficulty, letting us test how well the agent can reason about how each function depends on the others. You can explore the dataset at this link. The chart below shares the results for this task across the tested agents:评估数据集包含 20 个不同难度的问答题，让我们能够测试代理对每个函数如何依赖于其他函数的推理能力。您可以在此链接探索数据集。下面的图表展示了该任务在测试代理上的结果：
Despite being somewhat more difficult than the first two tasks in terms of reasoning ability required, GPT-4 does quite well on this task, answering all but 1 question correctly. Let's walk through this failure. For this data point, the agent is prompted with "Frank who is Even's friend is allergic to dairy. Can he eat the salad?"尽管在所需推理能力方面比前两个任务稍难，GPT-4 在此任务上表现相当不错，正确回答了除 1 个问题之外的所有问题。让我们分析一下这个失败案例。对于这个数据点，代理收到的提示是“弗兰克是伊娃的朋友，他对乳制品过敏。他能吃沙拉吗？”
In this case, GPT-4 makes the correct first call to get_users_by_name("Frank"). The tool returns with information about "Frank the Cat." The model then decides this doesn't match the requested "frank", so it queries again for "Even". There is no direct match, so the agent gives up, responding that it cannot find a user named "Even". While it may be understandable that it would be less confident about "Frank the cat", the agent neither considers it as a possible match nor does the agent mention it in its ultimate response to the user, meaning the user wouldn't be able to effectively provide feedback to help the agent self-correct.在这种情况下，GPT-4 正确地进行了第一次调用 `get_users_by_name("Frank")`。工具返回了关于“Frank the Cat”的信息。然后模型认为这与请求的“frank”不匹配，因此它再次查询“Even”。没有直接匹配，所以代理放弃了，回应说它找不到名为“Even”的用户。虽然可以理解它对“Frank the Cat”的信心较低，但代理既没有将其视为可能的匹配项，也没有在其最终给用户的响应中提及它，这意味着用户无法有效地提供反馈来帮助代理自我纠正。
🌌 Multiverse Math🌌 多重宇宙数学
LLMs are marketed as “reasoning machines,” but how well can they “reason” in practice?大型语言模型被宣传为“推理机器”，但它们在实践中能有多好的“推理”能力？
In the multiverse math task, agents must answer simple math questions, such as add 2 and 3. The twist is that that in this “mathematical universe”, math operations are not the same as what you’d expect. Want to do 2 + 2? The answer is 5.2 . Subtract 5.2 and 2 ? The answer is 0.2.在多重宇宙数学任务中，代理必须回答简单的数学问题，例如 2 加 3。不同之处在于，在这个“数学宇宙”中，数学运算与您期望的不同。想做 2 + 2？答案是 5.2。减去 5.2 和 2？答案是 0.2。
The full task instructions provided to the LLM (provided in the system prompt where available) are provided below:提供给语言模型的完整任务说明（在系统提示中可用）如下：
You are requested to solve math questions in an alternate mathematical universe. The operations have been altered to yield different results than expected. Do not guess the answer or rely on your innate knowledge of math. Use the provided tools to answer the question. While associativity and commutativity apply, distributivity does not. Answer the question using the fewest possible tools. Only include the numeric response without any clarifications.您被要求在一个交替的数学宇宙中解决数学问题。运算已修改，以产生与预期不同的结果。不要猜测答案或依赖您固有的数学知识。使用提供的工具来回答问题。虽然结合律和交换律适用，但分配律不适用。使用最少的工具回答问题。仅包含数字响应，不含任何解释。
Importantly, while these common operations (add, subtract, multiply, divide, cos, etc.) are all slightly altered, most mathematical properties still hold. Operations are still commutative and associative, though they are not distributive.重要的是，虽然这些常见运算（加、减、乘、除、cos 等）都略有改变，但大多数数学性质仍然成立。运算仍然是可交换和可结合的，但不可分配。
Let’s walk through an example to illustrate what we mean: "ecoli divides every 20 minutes. How many cells will be there after 2 hours (120 minutes) if we start with 5 cells?"让我们通过一个例子来说明我们的意思：“大肠杆菌每 20 分钟分裂一次。如果我们从 5 个细胞开始，2 小时（120 分钟）后会有多少个细胞？”
To solve this using the provided tools, the agent needs to identify:要使用提供的工具解决此问题，代理需要识别：
- How many divisions
dwill occur in the allotted time? (d = 120/20)在规定时间内将发生多少次分裂 d？（d = 120/20） - Then, for each cell
c, how many cells will be produced? (c = 2**d)然后，对于每个细胞 c，将产生多少个细胞？（c = 2**d） - Then how many cells will result
fat the end (f = 5*c)然后最终结果 f 是多少个细胞（f = 5*c）
GPT-4 may have seen each of these steps during training, but since it knows that these operations have been modified, it must refrain from skipping steps and instead focus on composing the tools. Below are the results for this task across the tested agents:GPT-4 可能在训练中见过这些步骤中的每一步，但由于它知道这些运算已被修改，因此它必须避免跳过步骤，而是专注于组合工具。以下是该任务在测试代理上的结果：
The multiverse math dataset tests two important characteristics of an LLM in isolation, without letting its factual knowledge interfere:多重宇宙数学数据集在隔离的情况下测试了大型语言模型的两个重要特征，而不让其事实知识干扰：
- How well can it “reason” compositionally?它在组合推理方面的能力如何？
- How well does it following instructions that may contradict the pre-trained knowledge?它遵循可能与预训练知识相矛盾的指令的能力如何？
It’s easy to ace a test when you’ve memorized the answers. It’s harder when the answers contradict patterns you’re used to. Let's seen one of the many examples GPT-4 fails: "how much is 131,778 divided by 2?"当记住了答案时，很容易通过考试。当答案与你习惯的模式相矛盾时，则要困难得多。让我们看看 GPT-4 失败的众多例子之一：“131,778 除以 2 是多少？”
While the GPT-4 agent correctly calls the divide() tool, it ignores the output from the tool and instead uses what it thinks the answer should be. This happens despite the instructions to the agent stating that it should only rely on tool outputs for its answers.尽管 GPT-4 代理正确地调用了 `divide()` 工具，但它忽略了工具的输出，而是使用了它认为应该的答案。尽管指示代理只应依赖工具输出来回答问题，但这种情况仍然发生。
mistral-7b-instruct-v0.1 , the OSS model fine-tuned by Anyscale for function calling, performs surprisingly well on this task. This dataset on average has fewer questions requiring multiple tool invocations (compared to our other tasks). That the model fails on the simple typewriter tasks but performs reasonably well here highlights how fine-tuning only on 1-hop function calling can lead to unintended performance degradations.mistral-7b-instruct-v0.1，由 Anyscale 为函数调用进行微调的 OSS 模型，在此任务上表现出人意料地好。该数据集平均需要多次工具调用的问题较少（与我们的其他任务相比）。该模型在简单的打字机任务上失败，但在该任务上表现尚可，这凸显了仅在单跳函数调用上进行微调可能导致意想不到的性能下降。
Additional Observations:其他观察：
For these results, we communicated model quality, but building an AI app also requires service reliability and stability. Despite the relatively small dataset size for these experiments and despite adding client-side rate limiting to our evaluation suite, we still ran into random-yet-frequent 5xx internal server errors from the popular model providers. 对于这些结果，我们沟通了模型的质量，但构建 AI 应用也需要服务可靠性和稳定性。尽管这些实验的数据集相对较小，并且我们在评估套件中添加了客户端速率限制，但我们仍然遇到了来自流行模型提供商的随机但频繁的 5xx 内部服务器错误。
We originally planned to benchmark Google's gemini-pro model, but because of the rate of internal server errors it rose during evaluations, we decided to leave it out of our results. The API also rejected multiple data points for the Typewriter and Multiverse Math datasets as being "unsafe" (for instance "what is the result of 2 to the power of 3") 我们最初计划对 Google 的 gemini-pro 模型进行基准测试，但由于在评估期间内部服务器错误的发生率很高，我们决定将其排除在我们的结果之外。该 API 还拒绝了打字机和多重宇宙数学数据集的多个数据点，认为它们“不安全”（例如，“2 的 3 次方结果是多少？”）。
Safety filters can be helpful, but if the false positive rate is too high, it can impact your service quality. 安全过滤器可能很有帮助，但如果误报率过高，则会影响您的服务质量。
Finally, we have shown a clear need for better open-source alternatives for tool use. The open-source community is rapidly developing better function calling models, and we expect more competitive options to be broadly available soon. To test your function calling model on these benchmarks, follow the instructions here, or if you'd like us to run a specific model, open an issue in the GitHub repo. We'd love for these results to change!最后，我们已经表明迫切需要更好的开源工具使用替代方案。开源社区正在快速开发更好的函数调用模型，我们预计很快将有更多有竞争力的选项广泛可用。要在此基准上测试您的函数调用模型，请遵循此处的说明，或者如果您希望我们运行特定模型，请在 GitHub 仓库中打开一个 issue。我们很乐意看到这些结果发生变化！
Conclusion结论
Thanks for reading! We’d love to hear your feedback on what other models and architectures you’d like to see tested on these environments, and what other tests would help make your life easier when trying to use agents in your app. You can check out our previous findings on document Q&A, extraction, Q&A over semi-structured tables, and multimodal reasoning abilities in the linked posts. You can also see how to reproduce these results yourself by running the notebooks in the langchain-benchmarks package. Thanks again!感谢您的阅读！我们很乐意听取您关于您希望在这些环境中测试哪些其他模型和架构的反馈，以及还有哪些其他测试可以帮助您在应用程序中使用代理时让生活更轻松。您可以在链接的文章中查看我们之前关于文档问答、提取、半结构化表格问答和多模态推理能力的研究结果。您还可以通过运行 langchain-benchmarks 包中的 notebook 来了解如何重现这些结果。再次感谢！
.png)
