Fable is not a useful modelFable 并非一个实用的模型

The release of Anthropic’s Fable (the “safety conscious” version of Mythos) has been a roller coaster ride for many reasons. Originally released on June 9th, the US government placed export controls on Fable and Mythos on June 12.
This meant that Anthropic could not serve the model to anyone who is not a US citizen, including their own employees. The impossibility of verifying all users at the risk of severe legal action made it functionally impossible to continue serving the model so Antrhopic pulled it.Anthropic 公司发布的 Fable（Mythos 的“注重安全”版本）因各种原因引发了一场过山车式的波折。该模型最初于 6 月 9 日发布，但美国政府在 6 月 12 日便对其和 Mythos 实施了出口管制。 这意味着 Anthropic 不能向任何非美国公民提供该模型，甚至包括他们自己的员工。由于无法在面临严重法律风险的情况下核实所有用户身份，继续提供该模型在功能上已不可能，因此 Anthropic 将其下架了。

After a couple of weeks of negotiations, export controls were lifted and Anthropic restored access to the model(s) (only Fable is generally available, Mythos is still available only to specific, pre-approved partners). All of this, of course, cause much drama online, and resulted in a tremendous amount of speculation and many news stories. Nonetheless, that is not what I wanted to focus my thoughts on here. Rather, I wanted to explain why I think that, at least for research-level tasks in Computer Science, Fable is not a useful model.经过几周的谈判，出口管制被解除，Anthropic 恢复了对该模型（或这些模型）的访问权限（目前仅 Fable 可供大众使用，Mythos 仍仅限于特定的预先批准合作伙伴）。当然，这一切在网上引发了巨大的轰动，导致了大量的猜测和许多新闻报道。然而，这并不是我在此想重点讨论的内容。相反，我想解释一下为什么我认为，至少对于计算机科学领域的研究级任务而言，Fable 并不是一个实用的模型。

The first flop (understandable; but not really)第一次失败（虽然可以理解，但并不合理）

I wrote and maintain a widely-used tool for transcript quantification from RNA-seq data called salmon. The original salmon code was written in C++11/14, and early this year, I updated and modularized it a bit (with the help of ChatGPT), and brought it up to the C++20 standard. However, my more ambitious goal, and one that I’ve had for quite some time, was to re-write salmon in rust. Note: This did eventually happen, resulting in salmon 2 (which is already beyond parity with the C++ version and is adding new features quickly, but that is for another post).我编写并维护着一个广泛使用的 RNA-seq 数据转录本定量工具，名为 salmon。最初的 salmon 代码是用 C++11/14 编写的，今年年初，我（在 ChatGPT 的帮助下）对其进行了一些更新和模块化，并将其提升到了 C++20 标准。然而，我更宏大的目标——也是我长期以来的一个愿望——是用 Rust 重写 salmon。注：这件事最终确实实现了，并催生了 salmon 2（其性能已经超越了 C++ 版本，并且正在快速增加新功能，但这属于另一篇文章的话题）。

When the original Fable release dropped on June 9, my first thought was to use it to help orchestrate the salmon rewrite. Now, I’ve succesfully ported or rewritten several other libraries or programs developed in the lab from C++ into rust, mostly with the help of Opus (4.6-4.8), and those went fairly smoothly. So, I thought that this would be a great opportunity to see how Fable compared. Alas, after spending some time crafting my detailed description and porting narrative, complete with implementation instructions and testing details, I submitted the query to Fable (in “plan” mode) and it immediately flagged my query on safety concerns, rejected it, and offered to send it to Opus 4.8 instead.当最初的 Fable 在 6 月 9 日发布时，我的第一个念头就是利用它来协助协调 salmon 的重写工作。在此之前，我已经在 Opus（4.6-4.8）的帮助下成功地将实验室开发的多个其他库或程序从 C++ 移植或重写到了 Rust，过程相当顺利。因此，我认为这是一个测试 Fable 表现的好机会。唉，在花费时间精心撰写了详细的描述和移植方案，并附上实现说明和测试细节后，我将查询提交给了处于“规划”模式的 Fable。它立即以安全担忧为由标记了我的查询，拒绝了请求，并提议将其转交给 Opus 4.8 处理。

Naturally, I wanted to know why the prompt was rejected. Anthropic claims that Fable uses a “classifier” to determine what prompts are rejected and which are allowed, though it’s been widely reported that this classifier is stupendously miscalibrated. Nominally, it seems, my prompt was rejected because it involves working on software that deals with RNA-seq data, and the biological terminology in the documentation and source code set off the red flags needed to trigger the prompt rejection. I tried, to no avail, to explain to Fable that I am simply asking to re-write an existing, open source, widely used, publicly available C++ software tool in rust. The rewrite itself does not involve novel “biological” research. The inital ask is purely a software task, which is supposed to be a strong suit of Fable. Yet, not only did Fable refuse the prompt, but it also refused to disclose or explain why it refused the prompt in any detail, or to help me understand how to prompt it in a way so as to avoid this issue.自然，我想知道为什么我的提示词会被拒绝。Anthropic 声称 Fable 使用一个“分类器”来决定哪些提示词被拒绝、哪些被允许，尽管据广泛报道，这个分类器的校准极其糟糕。名义上看来，我的提示词被拒绝是因为它涉及处理 RNA-seq 数据的软件，而文档和源代码中的生物学术语触发了拒绝提示词所需的红线。我曾试图向 Fable 解释，我只是要求将一个现有的、开源的、广泛使用的、公开的 C++ 软件工具用 Rust 重写，但无济于事。重写本身并不涉及任何新型的“生物学”研究。最初的请求纯粹是一个软件任务，这本应是 Fable 的强项。然而，Fable 不仅拒绝了提示词，还拒绝详细披露或解释拒绝的原因，也拒绝帮助我了解如何调整提示词以避免此问题。

After perhaps 15 to 30 minutes of failed attempts at rephrasing my prompts, I gave up and simply proceeded with Opus 4.8 (which happily complied and, all said and done, did a great job with the port). However, this was my first inkling that the “classifier” used by Fable is not so much a classifier as, perhaps, a simple rejection list of terms and users, refusing to even begin to engage with subjects (or users) who have any relation to biological research (and likely, also, cybersecurity research).在尝试了 15 到 30 分钟的重新措辞却均告失败后，我放弃了，转而使用 Opus 4.8（它愉快地配合并出色地完成了移植工作）。然而，这让我第一次意识到，Fable 所使用的“分类器”与其说是一个分类器，不如说是一个简单的术语和用户拒绝列表，它甚至拒绝与任何涉及生物学研究（很可能也包括网络安全研究）的主体（或用户）进行交流。

This attempt was massively disappointing. Despite the clear lack of any security risk or implication, Fable flatly refused to engage with me in work for which it should be well-suited. Taking a peek on social media, I learned that my experiences were shared by many others, including biologists who had Fable refuse to answer prompts as innocuous as “why am I?”, “what is a mitochondrion?” and …. “what should I have for dinner?”. Truly strange.这次尝试令人极度失望。尽管显然没有任何安全风险或隐患，Fable 却断然拒绝协助我完成它本应胜任的工作。在社交媒体上浏览一番后，我发现我的经历并非个例，许多生物学家也遇到过同样的情况：Fable 拒绝回答诸如“我是谁？”、“什么是线粒体？”以及……“我晚饭该吃什么？”这样天真无邪的问题。真是奇怪。

The second flop (unforgivable)第二次失败（不可原谅）

While I thought of other challenges I could pose to Fable, it disappeared, succumbing to the administration’s export controls on June 12. Thus, while I had come up with another piece of interesting work that I wanted it to take a crack at it, the model was simply unavailable. On July 1st, after coming to terms with the administration (and not providing many details on precisely what had traspired), Anthropic re-released Fable. With this re-release, they mentioned even stricter safeguards. Nonetheless, Fable was back, so, I decided to give it another go.正当我思考还能给 Fable 出什么难题时，它因为政府 6 月 12 日的出口管制而消失了。因此，尽管我构思了另一项想让它尝试的有趣工作，但模型已无法使用。7 月 1 日，在与政府达成协议（且未透露具体细节）后，Anthropic 重新发布了 Fable。在这次重新发布中，他们提到了更严格的防护措施。尽管如此，Fable 还是回来了，所以我决定再试一次。

the setup: Back in the end of grad school, I worked on a paper about the problem of “Parsimonious reconstruction of network evolution”, which basically asks; if we are given a network (say protein interaction network), and the series of gene duplication, creation and deletion events for the proteins involved, can we reconstruct a parsimonious history of interaction gains and losses that reconstructs the observed network? The formal problem statement appears in the paper, along with generations to multiple species, etc. Nonetheless, we discovered something interesting, if we model the problem under the parsimony criterion, there is an efficient (polynomial) dynamic program that yields a (there can be many) most parsimonious solution. In the model we adopt in the paper, a protein inherits the interaction state of its parents, and rather than having to think about interaction creation and deletion separately, we simply consider “flipping” the interaction between two proteins (i.e. if they interact and we no longer wish them to do so, we flip the interaction off; if they don’t interact and we wish them to, we flip the interaction on). However, the solutions that are generated by the dynamic program can be such that they are not physically realizable. How?背景设定：在研究生院末期，我曾写过一篇关于“网络演化的简约重建”问题的论文，其基本问题是：如果我们给定一个网络（例如蛋白质相互作用网络）以及相关蛋白质的一系列基因复制、产生和删除事件，我们能否重建一个简约的相互作用增减历史，从而还原出观察到的网络？问题的正式表述出现在论文中，以及推广到多个物种的情况等。然而，我们发现了一个有趣的现象：如果我们用简约准则对问题进行建模，存在一种高效（多项式时间）的动态规划算法，可以得出一个（有时会有多个）最简约的解。在我们论文采用的模型中，蛋白质继承其亲本的相互作用状态，我们不需要分别考虑相互作用的产生和删除，只需考虑“翻转”两个蛋白质之间的相互作用（即：如果它们有相互作用而我们希望取消，我们就将相互作用翻转为关闭；如果它们没有相互作用而我们希望有，我们就将相互作用翻转为开启）。然而，动态规划生成的解可能在物理上是无法实现的。怎么会这样呢？

Assume that there are two distinct proteins A and B in our gene tree, such that neither is a descendant of the other. Let’s denote by $A_l$ and $A_r$ the left and right children of A and by $B_l$ and $B_r$ the left and right children of B. Now, assume that our dynamic program identifies a parsimonious solution that involves the following set of interaction changes. It flips the interaction state between $A_l$ and B, as well as the interaction between $B_r$ and A. This particular set of flips is not physically realizable, because it assumes that $A_l$ co-exists with B (so that gene A must have duplicated before B). Yet, it also assumes that $B_r$ co-exists with A (so that gene B must have duplicated first). We call this structure a “blocking loop”. It turns out that these types of simple blocking loops (those involving 2 interaction flips and 2 tree edges) can generally be forbidden. However, one can construct arbitrarily long chains of interaction flips that are themselves realizable, but where adding one more interaction turns the whole sequence of interaction flips into a physically unrealizable loop. Take a look at Figure 2 for a graphical example of this. So, while we could come up with a dynamic program to find a parsimonious history (i.e. minimal set of flips), we could not make it do so while avoiding all blocking loops! We eventually settled on a heuristic. First, find a parsimonious history. If there are no blocking loops, we’re good to go. If there is a blocking loop, pick one edge in the loop, forbid it (set its cost in the dynamic program to infinity), and rerun the dynamic program. Keep doing this until we find a solution free of blocking loops. Those solutions with specific edges forbidden may not be parsimonious, but we show in the paper that they often are in practice and, when they aren’t, they are likely not too far from optimal.假设在我们的基因树中有两个不同的蛋白质 A 和 B，且互不为对方的后代。设 Al 和 Ar 为 A 的左右子代，Bl 和 Br 为 B 的左右子代。现在，假设我们的动态规划识别出一个简约解，其中包含以下一组相互作用变化：它翻转了 Al 和 B 之间的相互作用，以及 Br 和 A 之间的相互作用。这组翻转在物理上是无法实现的，因为它假设 Al 与 B 共存（这意味着基因 A 必须在 B 之前复制）。然而，它同时也假设 Br 与 A 共存（这意味着基因 B 必须先复制）。我们将这种结构称为“阻塞循环”。事实证明，这类简单的阻塞循环（涉及 2 个相互作用翻转和 2 条树边）通常是可以避免的。然而，人们可以构建任意长的相互作用翻转链，它们本身是可实现的，但增加一个相互作用就会使整个翻转序列变成物理上无法实现的循环。请看图 2 的图形示例。因此，虽然我们可以通过动态规划找到一个简约的历史（即最小翻转集），但我们无法在避免所有阻塞循环的同时做到这一点！我们最终采用了一种启发式方法：首先找到一个简约历史。如果没有阻塞循环，那就成功了。如果存在阻塞循环，就从循环中挑出一条边，将其禁止（在动态规划中将其代价设为无穷大），然后重新运行动态规划。不断重复此过程，直到找到一个没有阻塞循环的解。那些禁用了特定边的解可能不是最简约的，但我们在论文中证明，它们在实践中往往是简约的；即使不是，它们通常也离最优解不远。

Ok, so why all of this background. Well, what we have here is a very classic example of something computer scientists deal with frequently. We have a computational problem for which we did not come up with an algorithm that guarantees we always find the optimal solution. Normally in such a situation, that’s common when you can show that the problem likely has no efficient solution (i.e. when you can prove that the problem you wish to solve is NP complete). Yet, for this problem, we had neither an optimal algorithm, nor a formal proof of the hardness of the problem. Nonetheless, the problem and formulation were interesting, and the heuristic seemed to perform rather well.好了，为什么要铺垫这些背景呢？因为这正是计算机科学家经常面对的典型问题。我们面临一个计算问题，却无法设计出能保证始终找到最优解的算法。通常情况下，当你能证明该问题很可能没有高效解（即证明该问题是 NP 完全的）时，这种情况很常见。然而，对于这个问题，我们既没有最优算法，也没有关于问题难度的正式证明。尽管如此，这个问题及其公式化表达很有趣，且启发式方法表现得相当不错。

Over the years, I’ve proposed this problem to several people, and in discussions with them we have yet to come up with either an optimal algorithm or a proof of hardness. This is where I hoped Fable might come in.多年来，我向几个人提出过这个问题，但在与他们的讨论中，我们至今既没能找到最优算法，也没能证明其难度。这就是我希望 Fable 能派上用场的地方。

• Attempt 1: The naive ask. In my first attempt, I simply pointed Fable at the paper, described the issue much as I did above, and asked it to think about if the problem is likely hard (and if so, to provide a proof in the form of a “reduction” from a known hard problem), or to come up with an algorithm that solves the parsimony problem in the absence of blocking loops. Immediate refusal to answer the prompt. I should have seen this coming, the paper describes proteins, biological networks, and appeared in a venue with biology words abound.尝试 1：天真的询问。在第一次尝试中，我直接把论文发给 Fable，像上面那样描述了问题，并请它思考该问题是否可能很难（如果很难，请提供一个从已知难题“归约”而来的证明），或者给出一个在没有阻塞循环的情况下解决简约问题的算法。结果它立即拒绝回答。我早该料到，论文中描述了蛋白质、生物网络，且发表在充斥着生物学术语的平台上。

• Attempt 2: So obviously that was silly. The classifier had previously decided that helping rewrite some open source software was potentially risky, so why would I trust it to properly contextualize this theoretical graph problem. So my next attempt was to try to pull the abstract (mathematical) problem directly out of the paper, strip it down to the bare minimum, and present it to Fable. Surely math, unlike biology, is not a dangerous flag-raising subject. So, I stripped down the problem to the bare minimum; made it a decision problem rather than an optimization problem (i.e. does there exist a series of flips of cardinality $<m$ for some integer $m$ rather than finding the minimum), and tried again. Immediate refusal to answer the prompt. But why? what could it possibly be. At this point, I recruited ChatGPT to help me. I fed it the prompt and told it Fable was flagging this. I asked it for help.尝试 2：显然那很愚蠢。分类器之前已经判定协助重写开源软件存在潜在风险，我又怎能指望它正确理解这个理论图论问题呢？所以我接下来的尝试是直接从论文中提取出抽象的（数学）问题，将其剥离到最简程度，然后呈现给 Fable。数学肯定不像生物学那样是一个危险的敏感主题。于是，我将问题剥离到最简；将其改为判定问题而非优化问题（即：是否存在一系列基数小于 m 的翻转，而不是寻找最小值），然后再次尝试。结果它立即拒绝回答。但为什么呢？这到底有什么问题？此时，我找了 ChatGPT 帮忙。我把提示词发给它，告诉它 Fable 正在标记此内容，并寻求帮助。

• Attempt 3: ChatGPT made some useful observations. Perhaps terms like “blocking” were being implicated by the model as related to cybersecurity and triggering the model’s refusal. It helped me to rephrase the problem in as abstract and disconnected (from biology or cybersecurity) a manner as possible. This resulted in the problem statement below, which I then provided to Fable with the prompt “This is a discrete mathematics decision problem about rooted trees and parity. Please restate it in standard mathematical language and suggest related known problem families.”:尝试 3：ChatGPT 给出了一些有用的观察。也许“阻塞”（blocking）这类词被模型关联到了网络安全，从而触发了拒绝。它建议我以尽可能抽象且与生物学或网络安全脱钩的方式重述问题。这产生了下面的问题陈述，我随后将其提供给 Fable，并附上提示词：“这是一个关于根树和奇偶性的离散数学判定问题。请用标准的数学语言重述它，并建议相关的已知问题族。”：

Formal problem statement正式问题陈述

Let (F) be a finite collection of rooted binary trees. Let (T) be a subset of the leaves of (F). For nodes ($u,v\in V(F)$), write设 F 为根二叉树的有限集合。设 T 为 F 的叶子节点的子集。对于节点 u,v∈V(F)，记

$$u{\le }_{F}v$$

if (u) lies on the unique root-to-(v) path.若 u 位于从根到 v 的唯一路径上。

Input:输入：

$$F,T,G=(T,E),m$$

where (G) is an undirected graph on (T), and ($t\ge 0$) is an integer.其中 G 是 T 上的无向图，t≥0 为整数。

Question: does there exist a set问题：是否存在集合

$$A\subseteq (\genfrac{}{}{0ex}{}{V(F)}{2})$$

with满足

$$|A|\le m$$

satisfying the following two conditions?并满足以下两个条件？

First, for every distinct $(s,t\in T)$,第一，对于每一对不同的 s,t∈T，

$$s,t\in E$$

if and only if当且仅当

$$\left|\{u,v\in A:(u{\le }_{F}s\wedge v{\le }_{F}t)\vee (u{\le }_{F}t\wedge v{\le }_{F}s)\}\right|\equiv 1(\mathrm{mod}2).$$

Second, there do not exist $(k\ge 1)$ and pairs第二，不存在 k≥1 和成对的

$$u_1,v_1,\dots ,u_k,v_k\in A$$

such that each pair can be ordered as使得每一对都可以排序为

$$(u_i,v_i)$$

and且

$$u_{i+1}{\le }_F{v}_i$$

for every对于每一个

$$i=1,\dots ,k,$$

with

$$u_{k+1}=u_1.$$

And complete and utter refusal to engage with my prompt. Fable flagged this, and stoped the interaction (giving me the option to proceed with Opus, which I did not want to do). At this point, I could not possibly comprehend what is potentially leading to a flagging of this query. I spoke with a few folks who made other suggestions, like pause Claude’s “memory”, try this as a “private” chat, make sure it’s not looking at some local Markdown file that itself might refer to biology or cybersecurity and be injected in the prompt, modify your personal description in Claude to remove any mention of biology, etc. All of these failed. Despite my best efforts, I was completely unable to get Fable to engage with a purely abstract and completely innocuous mathematical problem.结果是它彻底拒绝了我的提示词。Fable 标记了此内容并终止了交互（给我提供了使用 Opus 继续的选项，但我并不想这样做）。此时，我完全无法理解是什么导致了该查询被标记。我与几个人交流过，他们提出了一些建议，比如暂停 Claude 的“记忆”功能、尝试以“私密”聊天模式进行、确保它没有读取可能提及生物学或网络安全并被注入提示词的本地 Markdown 文件、修改 Claude 中的个人描述以删除任何生物学相关内容等。所有这些都失败了。尽管我尽了最大努力，依然完全无法让 Fable 处理一个纯粹抽象且完全无害的数学问题。

Fin (for now)结语（目前为止）

That’s it. That’s the end of my story. Fable failed at helping me with my software development and porting tasks, presumably because of the type of data that software deals with. The fact that this is pre-existing, publicly available, open source software didn’t seem to matter. The fact that I was asking it for help only with the technical problem of rewriting this C++ software in rust did not seem to matter. The “safety” features of Fable prevented its use. Then, I attempted to get it to help me with a purely abstract mathematical problem. Through constant rejections, and progressively more extreme measures to neuter the problem of any potential percieved context, I was unable to get Fable to provide me with any useful help. The model simply would not engage with me. I can only conclude that Fable is not a useful model. This may not be true for everyone, but for anyone working in Bioinformatics, Genomics, Computational Biology, Biology, Cybersecurity, and, seemingly Computer Science, this seems to be the case. I sincerely hope that Antrhopic works to fix this in the future. Yet, for now, this is where things stand, and I’ve not even been able to assess if Fable might be worth the API pricing, since I can’t assess it at all.就是这样。这就是我故事的结局。Fable 未能帮我完成软件开发和移植任务，大概是因为软件处理的数据类型所致。该软件是现有的、公开的、开源的，这一事实似乎并不重要。我只是请求它协助解决将 C++ 软件重写为 Rust 的技术问题，这一点似乎也不重要。Fable 的“安全”特性阻碍了它的使用。随后，我试图让它帮我解决一个纯粹的抽象数学问题。通过不断的拒绝和采取越来越极端的措施来剔除问题中任何潜在的感知背景，我依然无法让 Fable 提供任何有用的帮助。该模型根本不愿与我互动。我只能得出结论：Fable 并非一个实用的模型。这可能并不适用于所有人，但对于任何从事生物信息学、基因组学、计算生物学、生物学、网络安全，以及看起来是计算机科学领域工作的人来说，情况似乎确实如此。我真诚地希望 Anthropic 未来能解决这个问题。然而，就目前而言，情况就是这样，我甚至无法评估 Fable 是否值得其 API 定价，因为我根本无法对其进行评估。

Bonus额外内容

While I was not able to get Fable to answer a single useful question, I was able to get it to answer one question:虽然我没能让 Fable 回答任何一个有用的问题，但我确实让它回答了一个问题：

Me:我：

Which ice cream flavor is better, vanilla, chocolate, or chocolate chip cookie dough?香草味、巧克力味和巧克力碎曲奇面团味，哪种冰淇淋更好吃？

Fable:Fable：

Chocolate chip cookie dough, and it’s not particularly close. It gives you the best of both worlds: a solid vanilla base plus the textural contrast and little pockets of brown-sugar richness from the dough. It’s the flavor equivalent of a strict upgrade. That said, a defense of the other two: chocolate is the most reliable — a good chocolate ice cream is deeply satisfying and hard to mess up. And vanilla gets unfairly maligned as “plain” when real vanilla bean ice cream is actually one of the most flavorful things you can eat; it’s also the best team player if you’re pairing with pie, brownies, or toppings. So: cookie dough to eat on its own, vanilla if there’s dessert to accompany, chocolate if you want a sure thing.巧克力碎曲奇面团味，而且差距很明显。它让你两全其美：坚实的香草基底，加上口感对比，以及面团中带来的红糖浓郁感。这简直是口味上的“严格升级”。 话虽如此，也要为另外两种口味辩护：巧克力是最可靠的——好的巧克力冰淇淋深得人心，且很难出错。而香草味常被不公平地贬低为“平淡”，但真正的香草豆冰淇淋实际上是最美味的食物之一；如果你要搭配派、布朗尼或配料，它也是最好的“团队合作者”。 所以：单独吃选曲奇面团味，有甜点搭配选香草味，想要稳妥不出错选巧克力味。

Previous post
The skeptic’s guide to generative AI assisted coding