I'd been waiting for more than 30 minutes. The terminal just sat there, blinking, without returning a single word. I'd launched Gemma2 in its 9-billion-parameter version on my laptop (a regular Mac, the kind any professor or student would use) and the model simply wasn't responding.我已经等了半个多小时了。终端界面静止在那里，光标闪烁，没有返回哪怕一个字。我在笔记本电脑（一台普通的 Mac，教授或学生常用的那种）上运行了 90 亿参数版本的 Gemma2，但模型毫无响应。

It wasn't a bug. It was the most honest answer the experiment could have given me.这并非程序错误，而是实验能给出的最诚实的答案。

That frustrating wait ended up being, without exaggeration, the most interesting finding of the whole process. Because the question that brought me there wasn't "how big can a model get?" — it was a much more practical one: what actually happens when an agent you built in a tutorial has to survive in production?毫不夸张地说，这段令人沮丧的等待时间最终成为了整个过程中最有趣的发现。因为促使我进行这项实验的问题并非“模型能有多大？”，而是一个更实际的问题：当你按照教程构建的智能体真正投入生产环境时，到底会发生什么？

I've been working with Gemma as a case study to understand that jump — from an educational prototype to something that can hold up under long conversations, limited hardware, and real users. This post is the honest summary of that process: what worked convincingly, what didn't work the way I expected, and why that "didn't work" turned out to be more useful than a clean result would have been.我一直以 Gemma 为案例来研究这一跨越——从教育原型到能够支撑长对话、适应有限硬件并应对真实用户的系统。这篇文章是对该过程的诚实总结：哪些方法行之有效，哪些方法不如预期，以及为什么那些“行不通”的结果反而比理想的结果更有价值。

The real problem: why tutorials are a little dishonest 真正的问题：为什么教程往往有些“不诚实”

Almost every conversational agent tutorial does the same thing, without saying so out loud: on every turn, it sends the model the entire previous history, all over again.几乎每一个关于对话智能体的教程都在做同一件事，尽管它们从未明说：在每一轮对话中，它都会把之前的全部历史记录再次发送给模型。

Imagine that every time you added a sentence to a conversation, you had to repeat everything said before it — every message, every reply — before you could say the new one. At first you don't notice. But if the conversation runs 30 or 50 turns, you're repeating an entire novel just to add one sentence.想象一下，每次你在对话中增加一句话时，都必须重复之前说过的一切——每一条消息、每一次回复——然后才能说出新的一句。起初你可能不会在意。但如果对话进行了 30 或 50 轮，你为了补充一句话，就得重复整本小说。

This pattern is called linear context stacking, and it causes three concrete problems:这种模式被称为线性上下文堆叠（linear context stacking），它会导致三个具体问题：

1. Memory saturation — every call to the model processes an increasingly large context.内存饱和——每次调用模型时，处理的上下文都在不断增大。
2. Risk of hitting the token limit — every model has a maximum context window; sooner or later, you hit it.触及 Token 上限的风险——每个模型都有最大上下文窗口；迟早会达到这个极限。
3. Quality degradation — there's a documented phenomenon in NLP literature called "lost in the middle": when context gets very long, models pay less attention to information sitting in the middle of it, versus the beginning or end. In other words, it's not just slower — it gets worse.质量下降——自然语言处理（NLP）文献中有一个公认的现象叫“迷失在中间”（lost in the middle）：当上下文变得过长时，模型对中间信息的关注度会低于开头或结尾。换句话说，它不仅变慢了，而且效果也变差了。

This problem isn't unique to any one model, but it weighs differently depending on context. If you're using a closed API with a massive context window and pay-per-token billing, the cost of this problem is financial — you just pay more. But if you're running an open model locally, as is common in universities and research labs across Latin America, the cost is infrastructure: limited RAM, no dedicated GPU, no room to "just pay for more compute." An unbounded context isn't a minor optimization detail there — it's the difference between the agent working at all or not.这个问题并非某个模型独有，但根据环境的不同，其影响也不同。如果你使用的是拥有超大上下文窗口且按 Token 付费的闭源 API，那么这个问题带来的代价是经济上的——你只需支付更多费用。但如果你像拉丁美洲的大学和实验室那样，在本地运行开源模型，代价就是基础设施：内存有限、没有专用 GPU、没有“加钱买算力”的空间。在这种情况下，不受限制的上下文不仅仅是一个微小的优化细节，更是智能体能否正常工作的关键。

The experiment: design and decisions 实验：设计与决策

To avoid staying purely theoretical, I ran a simple but controlled comparative experiment using Gemma 2 (2B), running locally with Ollama — no dependency on any paid external API.为了避免纸上谈兵，我使用 Ollama 在本地运行 Gemma 2 (2B) 进行了一次简单但受控的对比实验——不依赖任何付费外部 API。

The idea: simulate a typical technical conversation (a microservice troubleshooting case, where each turn adds new information) and run it against two different architectures:思路是：模拟一次典型的技术对话（微服务故障排除案例，每一轮对话都会增加新信息），并对比两种不同的架构：

• Pipeline A (Naive): accumulates the entire history with no compression at all. This is, literally, what a tutorial-style agent looks like.流水线 A（朴素型）：不进行任何压缩，累积全部历史记录。这正是教程类智能体的标准写法。
• Pipeline B (Optimized): applies history pruning — instead of sending the whole conversation, it sends a compact summary of the latest state. 流水线 B（优化型）：应用历史剪枝——不发送整个对话，而是发送最新状态的紧凑摘要。

# Pipeline A — accumulates everything, no pruning
conversation_history += f"\nPrevious text {i+1}: {chunk}\n"
full_prompt = f"{conversation_history}\n{TASK_PROMPT}\n{chunk}"

# Pipeline B — only a compact summary of the latest state
full_prompt = f"Previous compact context: {compact_context}\n{TASK_PROMPT}\n{chunk}"

Three methodological decisions I almost overlooked, and which turned out to be key to making the results trustworthy:三个我差点忽略、但对确保结果可信度至关重要的方法论决策：

1. The "cold start" nearly ruined everything.
In my first run, the first step of each pipeline came out suspiciously slower than the ones after it — several seconds off. It wasn't the prompt size: it was the cost of loading the model into memory the first time it's called. The fix was adding a throwaway "warm-up" call before starting to measure each pipeline, so both started on equal footing.1. “冷启动”差点毁了一切。 在第一次运行中，每条流水线的第一步都比后续步骤慢得离谱——慢了好几秒。原因不在于 Prompt 的大小，而在于模型首次加载到内存时的开销。解决方法是在开始测量每条流水线之前，先进行一次无用的“预热”调用，确保两者在同一起跑线上。

2. Real tokens, not estimated ones.
At first I was estimating tokens by counting words and applying an approximate conversion factor — a completely avoidable loss of precision. Ollama returns the real, exact count in every response (prompt_eval_count). Switching to that number made the charts far more defensible.2. 使用真实 Token，而非估算值。 起初我通过计算字数并应用近似转换系数来估算 Token 数量——这完全是一种可以避免的精度损失。Ollama 在每次响应中都会返回真实、精确的计数（prompt_eval_count）。改用该数据后，图表变得更有说服力了。

3. A single run isn't enough.
I ran each pipeline 3 times and averaged the results, with error bars included in the charts. This is what honestly revealed that one of my early results wasn't as solid as it first looked — more on that below.3. 单次运行是不够的。 我将每条流水线运行了 3 次并取平均值，同时在图表中加入了误差线。这诚实地揭示了我早期的一个结果并不像最初看起来那么稳固——下文会详细说明。

Results: what held up cleanly, and what didn't 结果：哪些经得起推敲，哪些没有

Tokens: the result that actually holds Token：经得起考验的结果

The token pattern was consistent across all 3 runs, with no ambiguity. The naive pipeline grows linearly — from 107 to 266 tokens in just 4 steps, nearly tripling. The optimized pipeline flattens into a plateau, around 104 tokens.Token 的模式在所有 3 次运行中都保持一致，没有任何歧义。朴素型流水线呈线性增长——在短短 4 步内从 107 个 Token 增加到 266 个，几乎翻了三倍。而优化型流水线则趋于平稳，维持在 104 个 Token 左右。

That's a 61% reduction in input tokens by the final step. Active context management delivers exactly what it promises: it keeps the conversation's memory footprint from growing unchecked.在最后一步，输入 Token 减少了 61%。主动的上下文管理正如其承诺的那样：防止了对话内存占用的无序增长。

Latency: the result that forced me to rethink the hypothesis 延迟：迫使我重新思考假设的结果

This is where the experiment got genuinely interesting. The intuition says: fewer input tokens, faster response. The real data didn't back that up — at least not clearly. The error bars for the naive and optimized pipelines overlap in almost every step.这是实验中最有趣的部分。直觉告诉我们：输入 Token 越少，响应速度越快。但真实数据并没有支持这一结论——至少不明显。朴素型和优化型流水线在几乎每一步的误差线都有重叠。

Why? Because with a 2B model, on relatively short conversations, total response time is dominated by how much the model has to generate as output — not by how much it has to read as input. Shrinking the context doesn't automatically speed up the generation of the response.为什么？因为对于 2B 模型，在相对较短的对话中，总响应时间主要取决于模型需要生成多少输出，而不是需要读取多少输入。缩减上下文并不能自动加快响应生成的速度。

It's a "negative" result in the sense that it doesn't confirm the initial hypothesis, but it's honestly the most valuable finding of the whole experiment: context management and latency are related problems, but they're not the same problem, and optimizing one doesn't guarantee improving the other.从没能证实初始假设的角度来看，这是一个“负面”结果，但它确实是整个实验中最有价值的发现：上下文管理和延迟是相关的问题，但它们并非同一个问题，优化前者并不保证能改善后者。

The failed attempt with Gemma2 9B (and why I'm not hiding it) Gemma2 9B 的失败尝试（以及我为什么不隐瞒它）

I wanted to push one step further and repeat the comparison with Gemma2's 9B version, to see whether a larger model would show a clearer latency advantage — the hypothesis being that processing a long prompt weighs more when the model itself is bigger.我本想更进一步，用 Gemma2 的 9B 版本重复对比，看看更大的模型是否会表现出更明显的延迟优势——我的假设是，当模型本身更大时，处理长 Prompt 的开销占比会更高。

I never got that data. Over 30 minutes running on my laptop, without a single complete response. I had to cancel it.我最终没能得到数据。在我的笔记本电脑上运行了 30 多分钟，连一次完整的响应都没得到。我不得不将其取消。

I could have left this out of the post. But it's a relevant data point in its own right, and honestly the one closest to my reality as a researcher in the region of Latin America: the barrier to experimenting with larger models isn't just a software optimization problem, it's a hardware access problem. If I, with intent and dedicated time, struggle to run a 9B model on a consumer laptop, that's exactly why this kind of work — optimizing efficient agents with small, accessible models — matters for universities, labs, and teams in the region that don't have dedicated GPUs on hand.我本可以不在文章中提及此事。但它本身就是一个相关的数据点，而且老实说，它最贴近我作为拉丁美洲研究者的现实：尝试更大模型的障碍不仅仅是软件优化问题，更是硬件获取问题。如果我投入了专门的时间和精力，却依然难以在消费级笔记本上运行 9B 模型，那么这正是为什么这类工作——用小型、易获取的模型优化高效智能体——对那些没有专用 GPU 的大学、实验室和团队如此重要的原因。

What this means in practice 这对实践意味着什么

If you're building, or thinking about building, an agent on a local open model, here's what I'm taking away from this experiment:如果你正在或打算在本地开源模型上构建智能体，以下是我从这次实验中得出的结论：

• Measure before you optimize. My initial intuition about latency was not the correct one, and I only found out because I measured rigorously (3 runs, warm-up, real tokens) instead of trusting a single run.先测量，再优化。我对延迟的最初直觉是不正确的，我之所以发现这一点，是因为我进行了严格的测量（3 次运行、预热、真实 Token），而不是轻信单次运行的结果。
• Saving tokens doesn't automatically buy you latency. Depending on model size and conversation length, the real bottleneck might be somewhere else entirely.节省 Token 并不自动等同于降低延迟。根据模型大小和对话长度，真正的瓶颈可能完全在别处。
• Context pruning has trade-offs — it's not magic. My current implementation trims by length, not semantic relevance, which means there's real risk of losing important historical information. That's a limitation I'm naming, not hiding.上下文剪枝有其权衡——它不是魔法。我目前的实现是按长度剪枝，而非语义相关性，这意味着存在丢失重要历史信息的风险。这是我指出的局限性，而非隐瞒。
• A failed experiment on real hardware is data, not a failure. I couldn't run 9B on my laptop. That data point ends up being as useful to the argument of this work as any chart.在真实硬件上的失败实验也是数据，而非失败。我无法在笔记本上运行 9B 模型。这个数据点对于这项工作的论证价值，丝毫不亚于任何图表。

Wrap-up 总结

This experiment started from a simple question — how do you take a tutorial-style agent and make it survive production? — and ended up giving me a more nuanced answer than I expected: context management matters, a lot, but it doesn't solve every performance problem on its own, and hardware constraints are a legitimate part of the technical conversation, not just a logistics footnote.这次实验始于一个简单的问题——如何让教程类的智能体在生产环境中生存？——最终给出了一个比预期更细致的答案：上下文管理非常重要，但它不能单独解决所有性能问题，硬件限制是技术讨论中不可或缺的一部分，而不仅仅是一个物流脚注。

All the code is available in the repository for anyone who wants to reproduce or adapt it — including both the successful results with Gemma2 (2B) and the documented limitation with the 9B model, because I believe transparency about what didn't work is as valuable as what did.所有代码均已在仓库中公开，供任何想要复现或调整的人使用——包括 Gemma2 (2B) 的成功结果以及关于 9B 模型局限性的记录，因为我相信，对失败之处的透明度与对成功之处的总结同样有价值。

If you're working with open models in the region, I'd genuinely love to hear about your experience — what hardware you're running, what you've hit, what context management strategies have worked for you. Reach out on LinkedIn.如果你在该地区使用开源模型，我非常希望能听到你的经验——你使用的是什么硬件、遇到了什么问题、哪些上下文管理策略对你有效。欢迎在 LinkedIn 上与我联系。

This work was also presented as a poster at the Second South American NLP School (Buenos Aires, August 2026).这项工作也曾在第二届南美 NLP 学校（2026 年 8 月，布宜诺斯艾利斯）上作为海报展示。