The demo to production Death Valley从演示到生产的死亡谷
If you’ve worked on an AI feature, you know the feeling. You start building something that you are excited about, set launch timelines. The model spits out a perfect response, the prototype works magically, and everybody in the room is mentally calculating how big this product will be when we launch. I’ve been in that room a lot many times and it’s fun.如果你曾经开发过 AI 功能，你一定懂这种感觉。你开始构建一个让人兴奋的东西，设定上线时间表。模型输出完美的回复，原型神奇地工作，房间里的每个人都在心算这款产品上线后会有多大。我曾多次站在那间屋子里，这很有趣。
Then you try to test before you ship.然后你尝试在发布前进行测试。
Latency spikes to 10 seconds on mobile. The model starts hallucinating on edge cases that happen to represent 15% of actual user queries. Your A/B test shows no statistically significant engagement lift because the variance in AI outputs makes traditional hypothesis testing basically meaningless. The safety team flags 340 failure cases in the first week, and you’re now debugging nondeterministic cases that fail in creative, novel ways every single day.移动端延迟飙升至 10 秒。模型在占实际用户查询 15% 的边缘案例上出现幻觉。你的 A/B 测试没有显示出统计显著的参与度提升，因为 AI 输出的方差让传统假设检验基本失去意义。安全团队在第一周标记了 340 起失败案例，而你现在每天都在调试以创造性、全新方式失败的非确定性案例。
Most often than not, it’s not a model problem but an engineering discipline problem. Shipping an AI product is very different from traditional software. I’ve figured this out the hard way. This playbook shares my learnings.大多数情况下，这不是模型的问题，而是工程纪律的问题。交付 AI 产品与传统软件截然不同。我是吃了苦头才领悟到的。此手册分享了我的经验。
Latency budgets延迟预算
Every AI feature comes with a latency tax. Large language model inference takes time. We’re talking 500 milliseconds to 5 or even 50 seconds depending on model size, input length, and infrastructure setup. For consumer products where people expect sub-200-millisecond interactions, this is a hard constraint you have to design around.每个 AI 功能都伴随延迟成本。大语言模型推理需要时间。根据模型大小、输入长度和基础设施配置，可能需要 500 毫秒到 5 秒甚至 50 秒不等。对于用户期望子 200 毫秒交互的消费产品来说，这是一项必须围绕其设计的硬约束。
The mistake I see most often is teams measuring only p50 latency. A feature with 800 milliseconds p50 sounds fine until you discover the p90 is 15 seconds. That means 10 in every 100 users sit there waiting for 15+ seconds. At scale, that’s thousands of terrible experiences per day.我最常看到的错误是团队只测 p50 延迟。800 毫秒的 p50 听起来还行，直到你发现 p90 是 15 秒。这意味着每 100 位用户中就有 10 位要等 15 秒以上。规模化后，这意味着每天成千上万的糟糕体验。
The way I think about it is you define your latency budget by interaction type, not globally: Synchronous interactions, where the user is staring at a spinner, need to resolve under 1 second. Progressive interactions, where output streams token by token, need first token in under 500 milliseconds and full response under 5 seconds. Asynchronous interactions, where the user keeps doing other stuff, can take up to 20 seconds with a progress indicator.我的思路是按交互类型而不是全局定义延迟预算：同步交互（用户盯着加载图标）需在 1 秒内完成。渐进式交互（输出逐 token 流式）首 token 必须在 500 毫秒内返回，完整响应在 5 秒内完成。异步交互（用户可以做其他事）可在 20 秒内完成，并配以进度指示。
You also need to measure cold starts separately. The first request after a model loads into memory can be 10 times slower than subsequent requests, and if your traffic is bursty, cold starts will disproportionately punish your most engaged users arriving during peak hours.还需要单独测量冷启动。模型加载到内存后的首次请求可能比后续请求慢 10 倍，如果你的流量呈突发式，冷启动会对高峰时段最活跃的用户造成不成比例的惩罚。
Besides, you also need to budget for the full pipeline, not just inference. A typical AI feature pipeline including input preprocessing (tokenization, context assembly, and prompt construction), model inference, output postprocessing (parsing, formatting, safety filtering, etc.), and a full response delivery adds up. Optimizing inference while ignoring the rest is like tuning your engine while driving on flat tires.此外，还要为完整流水线预算，而不仅仅是推理。典型的 AI 功能流水线包括输入预处理（分词、上下文组装、提示构造）、模型推理、输出后处理（解析、格式化、安全过滤等）以及完整响应交付。只优化推理而忽视其余环节，就像在瘪胎上调发动机。
Lastly, use streaming aggressively for generative features. Pushing tokens to the user as they’re generated instead of waiting for the full response changes how users perceive latency. A four-second response that starts appearing at 300 milliseconds feels dramatically faster than one that pops in all at once. Perception is reality when it comes to user experience.最后，对生成式功能要积极使用流式传输。将 token 在生成时即时推送给用户，而不是等完整响应返回，这会改变用户对延迟的感知。四秒的响应如果在 300 毫秒时就开始出现，会比一次性全部弹出感觉快得多。感知即现实，尤其在用户体验上。
Designing fallbacks设计回退方案
Traditional software fails in boring, predictable ways. AI features fail in novel, unpredictable, and occasionally creative ways. I once saw a model respond to a product recommendation query with a poem about loneliness. Your fallback strategy needs to be considerably more sophisticated than a try/catch block.传统软件的失败往往是乏味且可预测的。AI 功能的失败则新颖、不可预测，甚至有时富有创意。我曾见过模型在回答产品推荐查询时回了一首关于孤独的诗。你的回退策略必须比普通的 try/catch 块复杂得多。
I think about fallbacks as a hierarchy. First, model fallback: When your primary model fails, drop to a simpler, faster, and more reliable model. Most failure cases get handled without the user ever knowing. Second, cache fallback: For queries similar to stuff you’ve seen before, serve a cached response. Third, template fallback: When generation fails completely, fall back to prewritten templates. Degraded beats dead every time. Fourth, graceful omission: Sometimes the best fallback is to simply not show the AI feature at all rather than showing a broken version.我把回退视为层级结构。第一，模型回退：当主模型失效时，切换到更简单、更快、更可靠的模型。大多数失败在用户不知情的情况下得到处理。第二，缓存回退：对相似的查询返回缓存结果。第三，模板回退：当生成完全失败时，使用预写模板。降级总比死机好。第四，优雅省略：有时最好的回退是根本不展示 AI 功能，而不是展示一个破损的版本。
The design principle underneath all of this is that users should never encounter an unhandled AI failure. Every failure mode maps to a specific level, and transitions between levels should be invisible whenever you can manage it.所有这些背后的设计原则是：用户绝不能遇到未处理的 AI 失效。每种失效模式对应一个具体层级，层级之间的切换应在可能的情况下保持不可见。
Quality measurement质量衡量
Quality in traditional software is binary. The button works or it doesn’t. AI feature quality is continuous and subjective, and it changes depending on context. I’ve landed on a four-layer quality pyramid.传统软件的质量是二元的：按钮能用或不能用。AI 功能的质量是连续且主观的，并且会随上下文变化。我总结出一个四层质量金字塔。
The foundation is safety, and it’s nonnegotiable. Does the output contain harmful content, PII, or made-up facts? This layer is binary, and you measure it with automated classifiers running against 100% of outputs.底层是安全，且不可妥协。输出是否包含有害内容、个人身份信息或捏造事实？这一层是二元的，需使用自动分类器对 100% 输出进行检测。
The second layer is factual correctness, which is domain specific. Is the output actually right? For a coding assistant that means generated code compiles and passes tests. For a writing tool it means grammatical, stylistically appropriate output. You measure this with domain specific evaluation suites.第二层是事实正确性，属于领域特定。输出是否真的正确？对编码助手而言，意味着生成的代码能够编译并通过测试。对写作工具而言，则意味着语法、风格恰当。使用领域特定的评估套件来衡量。
The third layer is usefulness, and it’s user centered. Did the person actually benefit? Track acceptance rate, edit distance, time to task completion, and repeat usage. This is where traditional product metrics meet AI specific ones.第三层是有用性，面向用户。用户是否真的受益？跟踪接受率、编辑距离、任务完成时间和重复使用率。这是传统产品指标与 AI 专属指标的交汇点。
The fourth layer is delight, which is experimental. Does the output feel good? Hardest to measure but often most important for adoption. Sometimes the numbers say the feature works but users’ guts say it doesn’t. This layer catches that gap.第四层是愉悦感，属于实验性。输出是否让人感觉良好？这是最难衡量却往往最决定采纳的层面。有时数据表明功能有效，但用户直觉却说不行。这一层捕捉到这种差距。
A/B testing AI featuresAI 功能的 A/B 测试
A/B testing AI features is fundamentally harder than traditional features because AI outputs are nondeterministic. The same user doing the same thing twice might get different outputs, introducing variance that traditional frameworks weren’t built to handle.AI 功能的 A/B 测试本质上比传统功能更难，因为 AI 输出是非确定性的。同一用户在相同操作下两次可能得到不同的输出，这会引入传统框架难以处理的方差。
The core challenge is that intratreatment variance inflates the sample size you need for statistical significance, often by three to five times. If you’re running your AI experiment with normal sample size assumptions, you’re probably looking at noise and calling it signal.核心挑战在于内部方差会显著放大达到统计显著性所需的样本量，通常是原来的三到五倍。如果你按常规样本量假设进行 AI 实验，可能只是在把噪声误认为信号。
Then there’s the metric selection problem. A chatbot generating entertaining but factually wrong responses might show amazing engagement numbers while actively misleading users. You have to measure engagement and quality together. “Engaged interactions where quality score exceeds threshold” is more meaningful than raw engagement alone.还有指标选择的问题。一个聊天机器人生成有趣但事实错误的回复可能会显示惊人的参与度数据，却在误导用户。必须同时衡量参与度和质量。比如“质量分数超过阈值的互动”比单纯的参与度更有意义。
The temporal problem matters too. AI feature value changes over time as users learn how to work with it. Short experiments will underestimate long-term value if there’s a learning curve, or overestimate it if there’s a novelty bump.时间因素也很重要。AI 功能的价值会随时间变化，因为用户会学习如何使用它。短期实验会低估有学习曲线的长期价值，或在新奇效应下高估价值。
My practical guidance: budget two to three times more time and traffic for AI experiments than traditional ones. Lean on Bayesian methods as they handle high variance better. And always pair quantitative tests with qualitative research. Ten user interviews will surface failure modes that no amount of statistical analysis will catch.我的实用建议：为 AI 实验预留两到三倍于传统实验的时间和流量。倾向使用贝叶斯方法，因为它们更好地处理高方差。并且始终将定量测试与定性研究相结合。十次用户访谈能发现统计分析捕捉不到的失效模式。
Model drift monitoring模型漂移监控
Model drift is the slow, invisible rot of AI output quality over time, and there are multiple culprits.模型漂移是 AI 输出质量随时间缓慢、隐蔽的衰退，背后有多种原因。
Data drift happens because the world changes and user behavior evolves. A model trained on 2024 data performs worse on 2026 queries referencing new concepts, slang, and cultural moments.数据漂移源于世界变化和用户行为演进。用 2024 年数据训练的模型在 2026 年的查询中表现更差，因为新概念、俚语和文化事件出现。
Provider drift happens because third-party APIs change without your consent. OpenAI acknowledged that GPT-4’s behavior shifted measurably between March and June 2023, and Stanford researchers documented significant performance swings. The fix: Pin your model versions so updates happen on your schedule, after your testing.提供商漂移是因为第三方 API 在未征得同意的情况下变更。OpenAI 曾承认 GPT-4 的行为在 2023 年 3 月至 6 月之间出现可测量的变化，斯坦福研究者也记录了显著的性能波动。解决办法：固定模型版本，让更新在你安排的时间进行，并在更新前完成测试。
Evaluation drift is the subtlest form. Even your quality metrics can become inadequate and the evaluation criteria that made sense at launch might become inadequate as usage patterns shift and user expectations change. Quarterly reviews of your evaluation suites are essential.评估漂移是最微妙的形式。即使是你的质量指标也可能变得不再适用，最初的评估标准在使用模式和用户期望变化后可能失效。每季度审查评估套件是必需的。
At minimum you need daily automated quality evaluations on 1% to 5% of production traffic, weekly analysis of input distribution characteristics, and monthly human evaluation of 100 to 500 examples. Shipping an AI feature without drift monitoring is like deploying a service without alerting. You won’t know it’s broken until your users tell you, and by then they’re angry.最低要求是对 1%~5% 的生产流量进行每日自动质量评估，每周分析输入分布特征，并每月对 100~500 条样本进行人工评估。没有漂移监控就发布 AI 功能，就像部署一个没有告警的服务。你不会知道它坏了，直到用户抱怨，而那时他们已经很生气。
Evaluation frameworks评估框架
How do you know if your AI feature is good enough? You need two fundamentally different approaches, and you genuinely need both.如何判断你的 AI 功能是否足够好？你需要两种根本不同的方法，而且两者都必须具备。
Automated evaluation gives you speed. Build a golden dataset of 500 to 2,000 labeled examples, train a classifier or use a capable model as judge, and validate against human judgment quarterly targeting 85% agreement. Automated evals chew through thousands of examples per hour, making them essential for velocity. The pitfall: They miss novel failure modes not in the training data.自动评估提供速度。构建一个包含 500~2000 条标注样本的金标准数据集，训练分类器或使用强大的模型作为评审，每季度验证与人工判断的匹配度目标为 85%。自动评估每小时可以处理数千条样本，是提升速度的关键。缺点是它们会漏掉训练数据中未出现的全新失效模式。
Human evaluation catches what automation misses. Structure it with five to seven evaluators mixing domain experts and representative users. Use a consistent rubric covering accuracy, helpfulness, tone, completeness, and safety. Run weekly during development, monthly in production. The trade-offs: expensive at $15 to $30 per example, slow with 24 to 72 hour turnaround, and subject to human biases. Manage by rotating evaluators and capping sessions at two hours.人工评估捕捉自动化遗漏。组织 5~7 名评审员，混合领域专家和代表性用户。使用统一的评分标准，覆盖准确性、帮助性、语气、完整性和安全性。开发阶段每周进行，生产阶段每月进行。权衡在于成本（每条样本 15~30 美元）、速度（24~72 小时交付）以及人为偏见。通过轮换评审员并将评审时长限制在两小时内来管理。
The model as judge approach is an increasingly viable middle ground. Judging quality is often easier than generating it, which means a model can reliably evaluate outputs even for tasks where it couldn’t produce them itself. Use it for high-volume evaluation but always validate against human judgment.模型作为评审者的方式是日益可行的中间方案。评估质量往往比生成更容易，这意味着模型可以可靠地评估输出，即使它本身无法完成该任务。可用于大批量评估，但始终要与人工判断进行校验。
Graceful degradation and prompt engineering优雅降级与提示工程
Graceful degradation means when capabilities decrease, the experience gets worse smoothly instead of falling off a cliff. Design for capability levels, not binary states. Define four to five levels with specific behaviors at each. For example, for an AI writing assistant: Level 5 is full capability with real-time suggestions, tone adjustment, and structure recommendations. Level 4 is delayed suggestions appearing after a two- to three-second pause because latency is up. Level 3 is basic suggestions only like grammar and spelling with no style feedback. Each level is a deliberate design decision, not an accident.优雅降级指的是当能力下降时，体验平滑变差，而不是骤然崩溃。设计时要针对能力层级，而不是二元状态。定义四到五个层级，每个层级对应具体行为。例如，对 AI 写作助手：层级 5 为完整能力，实时建议、语调调整和结构推荐；层级 4 为延迟 2~3 秒后出现的建议，因为延迟升高；层级 3 为仅提供语法和拼写建议，无风格反馈。每个层级都是有意的设计决策，而非意外。
Make degradation invisible when possible. Users shouldn’t see a “broken” experience. They see a less detailed one. That’s a huge difference psychologically. However, when the degradation is significant enough that users will notice, proactive communication like “AI suggestions are temporarily limited” builds trust infinitely more than silently pushing poor-quality outputs.在可能的情况下让降级不可感知。用户不应看到“破损”的体验，而是看到细节少一些的版本。这在心理上差别巨大。然而，当降级足够明显以致用户会注意到时，主动沟通如“AI 建议暂时受限”比悄悄推送低质量输出更能建立信任。
Prompt engineering in production is software engineering. In production, prompts are code, and they need version control, testing, monitoring, and maintenance. Version controls every prompt. Parameterize prompts, don’t hardcode context. Production prompts should be templates with clearly defined injection points for user context, system state, and dynamic instructions. This makes them testable because you can inject known inputs and verify outputs, and it makes them maintainable because changing how you handle context shouldn’t require rewriting the entire prompt from scratch.生产环境中的提示工程就是软件工程。提示在生产中是代码，需要版本控制、测试、监控和维护。版本控制每个提示。提示要参数化，避免硬编码上下文。生产提示应为模板，明确划分用户上下文、系统状态和动态指令的注入点。这使得提示可测试，因为可以注入已知输入并验证输出，也便于维护，因为更改上下文处理方式时无需重写整个提示。
Test prompts against regression suites. Maintain 200 to 500 test cases covering the full distribution of expected inputs, including edge cases and adversarial inputs. Run the suite against every prompt change before deployment.对提示进行回归测试。维护 200~500 条测试用例，覆盖预期输入的完整分布，包括边缘案例和对抗性输入。每次提示变更前都运行该套件。
Monitor prompt performance in production. Track output quality metrics like acceptance rate, user edits, and regeneration requests, segmented by prompt version. When you deploy a new version, compare its production metrics against the previous one for at least 72 hours before calling it stable. This is basically canary deployment for prompts.监控生产环境中的提示表现。跟踪输出质量指标，如接受率、用户编辑次数和重新生成请求，按提示版本进行细分。当部署新版本时，至少观察 72 小时，将其生产指标与前一版本对比后再认定为稳定。这本质上是提示的金丝雀部署。
Ship it right正确发布
These systems aren’t optional add ons you can bolt on after launch. Every feature I’ve seen fail was built first with plans to “add production hardening later.” Later never comes.这些系统不是可以在上线后随意添加的可选插件。我见过的每一次功能失败，都源于最初只计划“以后再做生产硬化”。而“以后”永远不会到来。
AI features are probabilistic and nondeterministic, and they change over time without anyone touching them. Build these systems, staff them properly, and treat them with the same seriousness you’d give your core infrastructure. The gap between demo and production is wide, but it’s absolutely crossable if you build the right bridge.AI 功能是概率性和非确定性的，并且会随时间自行变化。要构建这些系统，配备合适的团队，并以与你对核心基础设施同等的严肃态度对待。演示与生产之间的差距很大，但只要搭建好桥梁，完全可以跨越。
Note: The research work pertaining to this article was done in a personal capacity. Views are of my own and do not reflect my employer’s views in any way.注：本文涉及的研究工作是以个人身份完成的。观点仅代表本人，不代表雇主立场。
