Prompt Compression and Cache Tuning: Cut Your LLM API Costs by 60%

LLM cost optimization is fundamentally a token economics problem. This tutorial covers four distinct techniques — prompt compression, semantic caching, chain-of-thought pruning, and output length constraints — that when combined can reduce LLM API costs by up to 63%.LLM 成本优化本质上是一个 Token 经济学问题。本教程涵盖了四种不同的技术——提示词压缩、语义缓存、思维链剪枝和输出长度限制——结合使用这些技术可将 LLM API 成本降低高达 63%。

How to Reduce LLM API Costs如何降低 LLM API 成本

Instrument token logging on every API call to establish a cost baseline before optimizing.在每次 API 调用中植入 Token 日志记录，以便在优化前建立成本基准。
Compress system prompts by eliminating hedge language, consolidating instructions into structured formats, and using tools like LLMLingua.通过消除模糊语言、将指令整合为结构化格式以及使用 LLMLingua 等工具来压缩系统提示词。
Constrain output length with max_completion_tokens or max_tokens and enforce structured JSON schemas.使用 max_completion_tokens 或 max_tokens 限制输出长度，并强制执行结构化 JSON 模式。
Prune chain-of-thought reasoning in production by instructing the model to return only the final answer.在生产环境中剪枝思维链推理，指示模型仅返回最终答案。
Implement semantic caching using embedding similarity to skip redundant API calls entirely.利用嵌入相似度实现语义缓存，从而完全跳过冗余的 API 调用。
Leverage provider-native prompt caching from OpenAI, Anthropic, or Google for automatic input token discounts.利用 OpenAI、Anthropic 或 Google 提供的原生提示词缓存功能，自动获得输入 Token 折扣。
Validate output quality against your evaluation set after each optimization to ensure accuracy holds.每次优化后，根据您的评估集验证输出质量，以确保准确性不受影响。

Table of Contents目录

Why Standard Prompting Is Burning Your Budget为什么标准提示词正在烧掉您的预算
Understanding Token Economics Across Providers了解各提供商的 Token 经济学
Technique 1: Prompt Compression技术 1：提示词压缩
Technique 2: Semantic Caching技术 2：语义缓存
Technique 3: Chain-of-Thought Pruning for Production技术 3：生产环境中的思维链剪枝
Technique 4: Output Length Constraints技术 4：输出长度限制
Cost Comparison Table: Before and After Across 5 Models成本对比表：5 种模型优化前后的对比
Combining All Four Techniques: A Real-World Optimization Pipeline结合所有四种技术：一个真实的优化流水线
Start With the Lowest-Hanging Fruit从最容易实现的目标开始

Why Standard Prompting Is Burning Your Budget

LLM cost optimization is fundamentally a token economics problem. Every API call to OpenAI, Anthropic, or Google Gemini bills by the token, and most production systems send far more tokens than the task actually requires. Verbose system prompts padded with hedge language, repeated context across conversation turns, unconstrained output lengths, and chain-of-thought reasoning left enabled in production all contribute to bills that run two to three times higher than necessary.LLM 成本优化本质上是一个 Token 经济学问题。每次对 OpenAI、Anthropic 或 Google Gemini 的 API 调用都是按 Token 收费的，而大多数生产系统发送的 Token 远超任务实际所需。充斥着模糊语言的冗长系统提示词、对话轮次中重复的上下文、未受限的输出长度以及在生产环境中启用的思维链推理，所有这些都导致账单比必要水平高出两到三倍。

This tutorial covers four distinct techniques for reducing that waste: prompt compression, semantic caching, chain-of-thought pruning, and output length constraints. When combined, these methods can reduce LLM API costs by up to 63%, though the exact figure depends on use case, model selection, and traffic patterns. The techniques are not theoretical. Each section includes working code examples in Python and Node.js that target the OpenAI and Anthropic APIs directly, with measured token counts showing the before and after.本教程涵盖了四种减少此类浪费的独特技术：提示词压缩、语义缓存、思维链剪枝和输出长度限制。结合使用这些方法，可以将 LLM API 成本降低高达 63%，尽管具体数字取决于用例、模型选择和流量模式。这些技术并非理论。每一节都包含了针对 OpenAI 和 Anthropic API 的 Python 和 Node.js 代码示例，并展示了优化前后的 Token 计数对比。

The audience here is developers already calling LLM APIs in production or at scale, not those experimenting with chat completions for the first time.本文的目标读者是已经在生产环境或大规模调用 LLM API 的开发人员，而非初次尝试聊天补全功能的实验者。

Understanding Token Economics Across Providers

How OpenAI, Anthropic, and Google Gemini Price TokensOpenAI、Anthropic 和 Google Gemini 如何对 Token 定价

All three major providers split billing into input tokens and output tokens, but the ratio between them varies significantly. Output tokens cost more than input tokens, by a factor of 2x to 5x depending on the model. For GPT-4o, OpenAI charges $2.50 per million input tokens and $10.00 per million output tokens, a 4x ratio. Anthropic's Claude 3.5 Sonnet prices at $3.00 per million input and $15.00 per million output, a 5x ratio. Google's Gemini 1.5 Flash costs roughly 33x less than GPT-4o on both input ($0.075 per million) and output ($0.30 per million) for prompts under 128K tokens.这三家主要提供商都将计费分为输入 Token 和输出 Token，但两者之间的比例差异很大。输出 Token 的成本高于输入 Token，根据模型不同，倍数在 2 倍到 5 倍之间。对于 GPT-4o，OpenAI 的收费标准是每百万输入 Token 2.50 美元，每百万输出 Token 10.00 美元，比例为 4 倍。Anthropic 的 Claude 3.5 Sonnet 定价为每百万输入 3.00 美元，每百万输出 15.00 美元，比例为 5 倍。Google 的 Gemini 1.5 Flash 在 128K Token 以内的提示词中，输入（每百万 0.075 美元）和输出（每百万 0.30 美元）的成本大约比 GPT-4o 低 33 倍。

Note: All pricing figures in this article are as of the time of writing. Verify current pricing at openai.com/pricing, anthropic.com/pricing, and Google's Generative AI pricing page before running cost projections.注意：本文中的所有定价数据均为撰写时的数据。在进行成本预测之前，请在 openai.com/pricing、anthropic.com/pricing 以及 Google 的生成式 AI 定价页面核实当前价格。

This asymmetry has a direct consequence for optimization priority: reducing output tokens yields disproportionately larger cost savings per token eliminated.这种不对称性对优化优先级有直接影响：减少输出 Token 所带来的单位 Token 成本节约效果更为显著。

Reducing output tokens yields disproportionately larger cost savings per token eliminated.减少输出 Token 所带来的单位 Token 成本节约效果更为显著。

Each provider also offers cached token discounts. OpenAI's automatic prompt caching provides a 50% discount on cached input tokens. Anthropic's explicit prompt caching offers a 90% discount on cache reads (though cache writes cost 25% more than base input). Google Gemini's context caching charges at about 25% of the standard input rate for cached content.每家提供商还提供缓存 Token 折扣。OpenAI 的自动提示词缓存对缓存的输入 Token 提供 50% 的折扣。Anthropic 的显式提示词缓存对缓存读取提供 90% 的折扣（尽管缓存写入成本比基础输入高 25%）。Google Gemini 的上下文缓存对缓存内容收取标准输入费率的约 25%。

Where Tokens Are Wasted in a Typical API Call典型 API 调用中 Token 在哪里被浪费了

Four categories account for the bulk of unnecessary token spend:四大类原因导致了大部分不必要的 Token 支出：

System prompt bloat. Instructions contain filler phrases, excessive examples, and redundant guardrails that often double the prompt length without improving output quality.系统提示词臃肿。指令中包含填充短语、过多的示例以及冗余的护栏，这些往往使提示词长度翻倍，却并未提高输出质量。
Repeated context across conversation turns. Multi-turn flows resend the same background information with every request.对话轮次中重复的上下文。多轮对话流程在每次请求时都会重新发送相同的背景信息。
Uncontrolled output verbosity. Models generate explanations, caveats, and preambles that the consuming application immediately discards when you don't cap output length.不受控制的输出冗长。模型生成的解释、注意事项和前言，在您未限制输出长度时，会被消费应用程序立即丢弃。
Chain-of-thought reasoning left active in production. Lengthy intermediate reasoning steps that served their purpose during development add no value in a deployed pipeline.在生产环境中保持激活的思维链推理。在开发过程中起到作用的冗长中间推理步骤，在已部署的流水线中毫无价值。

Technique 1: Prompt Compression

What Prompt Compression Means in Practice提示词压缩在实践中意味着什么

Prompt compression reduces the token count of a prompt while preserving the information the model needs to produce an accurate response. There are two categories. Lossy compression removes content entirely, such as dropping optional examples or eliminating edge case instructions that apply to a small fraction of requests. Lossless compression rephrases the same content more concisely, such as converting prose instructions into structured YAML or JSON format, or replacing multi-sentence explanations with terse directives.提示词压缩在保留模型产生准确响应所需信息的同时，减少了提示词的 Token 数量。分为两类：有损压缩会完全删除内容，例如丢弃可选示例或删除仅适用于极少数请求的边缘情况指令。无损压缩则以更简洁的方式重写相同内容，例如将散文指令转换为结构化的 YAML 或 JSON 格式，或用简练的指令替换多句解释。

Compression hurts quality when it removes disambiguation that the model genuinely needs. For tasks with narrow, well-defined outputs like entity extraction or classification, aggressive compression is safe. For tasks requiring nuanced judgment, such as open-ended writing or complex reasoning, over-compression can degrade results. Track output quality metrics (F1 score for extraction, human evaluation scores for generation) alongside token counts; if quality drops more than 2-3% on your eval set, you've compressed too far.当压缩删除了模型真正需要的消歧信息时，会损害质量。对于实体提取或分类等输出定义明确的任务，激进的压缩是安全的。对于需要细致判断的任务，如开放式写作或复杂推理，过度压缩可能会降低结果。请在跟踪 Token 数量的同时跟踪输出质量指标（提取的 F1 分数，生成的评估分数）；如果您的评估集质量下降超过 2-3%，说明压缩过度了。

Manual Prompt Compression Strategies手动提示词压缩策略

Three manual strategies yield the largest gains with the least risk:三种手动策略以最小的风险获得最大的收益：

Eliminate hedge language and politeness tokens. Phrases like "Please kindly ensure that you carefully consider" become "Ensure."消除模糊语言和礼貌性 Token。诸如“请您务必仔细考虑”之类的短语应改为“确保”。
Consolidate multi-sentence instructions into structured formats. A five-sentence paragraph explaining a desired JSON output shape becomes the JSON schema itself, which is both shorter and more precise.将多句指令整合为结构化格式。解释所需 JSON 输出形状的五句段落应直接变为 JSON 模式本身，这既简短又精确。
Use reference tokens instead of repeating context. Rather than restating a product description in both the system prompt and the user message, define it once and refer to it by label.使用引用 Token 而非重复上下文。与其在系统提示词和用户消息中重复产品描述，不如定义一次并按标签引用它。

Programmatic Prompt Compression with LLMLingua使用 LLMLingua 进行程序化提示词压缩

Microsoft Research's LLMLingua approach uses a small language model to identify and remove tokens from a prompt that contribute least to the model's ability to produce correct outputs. The library evaluates token-level perplexity and prunes low-information tokens while preserving semantic integrity.微软研究院的 LLMLingua 方法使用小型语言模型来识别并删除提示词中对模型产生正确输出贡献最小的 Token。该库评估 Token 级别的困惑度，并在保持语义完整性的同时剪枝低信息量的 Token。

Install the required dependencies first:首先安装所需的依赖项：

pip install openai "llmlingua>=0.2.2" numpy

Note: The first run will download a transformer model checkpoint (~500MB) from Hugging Face. Ensure sufficient disk space and allow several minutes for the download.注意：首次运行将从 Hugging Face 下载一个 Transformer 模型检查点（约 500MB）。请确保磁盘空间充足，并预留几分钟下载时间。

Note: The checkpoint microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank used below is optimized for meeting transcripts (MeetingBank dataset). Validate compressed output quality on your domain before production use. For other text types, evaluate alternative LLMLingua-2 checkpoints and compare entity extraction accuracy before and after compression.注意：下文使用的检查点 microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank 针对会议记录（MeetingBank 数据集）进行了优化。在生产使用前，请在您的领域验证压缩输出的质量。对于其他类型的文本，请评估其他 LLMLingua-2 检查点，并比较压缩前后的实体提取准确率。

import time
from llmlingua import PromptCompressor
from openai import OpenAI, RateLimitError, APIError

client = OpenAI()

# Original ~500-token system prompt (token count is approximate;
# use tiktoken for exact measurement)
original_prompt = """You are an expert product review analyst. Your job is to carefully
read product reviews submitted by users and extract structured information from them.
You should identify the key entities mentioned in the review, including product names,
brand names, and specific features that the reviewer discusses. Please make sure to
consider both positive and negative sentiments expressed about each entity. When you
find an entity, classify it into one of the following categories: product, brand, or
feature. Also determine the sentiment as positive, negative, or neutral. Return your
analysis as a JSON object with an array called 'entities', where each entity has the
fields 'name', 'type', and 'sentiment'. Be thorough but concise in your extraction.
Do not include entities that are only mentioned in passing without any opinion expressed.
Focus on entities where the reviewer has expressed a clear opinion or evaluation.
Make sure your JSON is valid and properly formatted. Do not include any explanation
or commentary outside the JSON object. Only return the JSON.
You should handle reviews in English. If the review contains multiple products being
compared, extract entities for all of them. If a feature is mentioned for multiple
products, create separate entity entries for each product-feature combination.
Ensure that entity names are normalized — for example, use the full brand name rather
than abbreviations when possible. If the reviewer uses slang or informal language,
interpret it to the best of your ability and use standard terminology in your output."""

# Compress using LLMLingua-2
compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True
)

compressed = compressor.compress_prompt(
    original_prompt,
    rate=0.4,  # Target 40% of original length
    force_tokens=["JSON", "entities", "name", "type", "sentiment"]
)

compressed_prompt = compressed["compressed_prompt"]

# Introspect available keys at runtime to guard against version differences
origin_tokens     = compressed.get("origin_tokens",          "UNVERIFIED")
compressed_tokens = compressed.get("compressed_tokens",      "UNVERIFIED")
ratio             = compressed.get("compressed_tokens_ratio", "UNVERIFIED")

print(f"Available keys: {list(compressed.keys())}")
print(f"Original tokens:   {origin_tokens}")
print(f"Compressed tokens: {compressed_tokens}")
print(f"Compression ratio: {ratio}")

# Send compressed prompt to OpenAI with retry logic
max_retries = 3
response = None
for attempt in range(max_retries):
    try:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": compressed_prompt},
                {"role": "user", "content": "The new Sony WH-1000XM5 headphones have amazing noise cancellation but the build quality feels cheaper than the XM4. Battery life is stellar though."}
            ]
        )
        break
    except RateLimitError:
        wait = 2 ** attempt
        print(f"Rate limited. Retrying in {wait}s (attempt {attempt + 1}/{max_retries})")
        time.sleep(wait)
    except APIError as e:
        print(f"API error on attempt {attempt + 1}: {e}")
        if attempt == max_retries - 1:
            raise

if response is None:
    raise RuntimeError("Exceeded max retries for OpenAI API call")

if response.usage is None:
    raise ValueError("response.usage is None — streaming mode is not supported here")

print(f"Prompt tokens used: {response.usage.prompt_tokens}")
print(f"Completion tokens used: {response.usage.completion_tokens}")
print(response.choices[0].message.content)

The force_tokens parameter ensures that critical terms survive the compression pass. With a rate of 0.4, the compressed prompt retains about 200 tokens from the original ~500 while preserving the extraction instructions and output format requirements.force_tokens 参数确保关键术语在压缩过程中得以保留。以 0.4 的比率，压缩后的提示词保留了原始约 500 个 Token 中的约 200 个，同时保留了提取指令和输出格式要求。

Measuring Compression Impact衡量压缩影响

Systematic measurement requires logging token usage on every call and comparing against a known baseline.系统性衡量需要记录每次调用的 Token 使用情况，并与已知基准进行比较。

Note: These JavaScript examples use top-level await and require Node.js 14.8+ with ES modules. Add "type": "module" to your package.json or wrap the code in (async () => { ... })();.注意：这些 JavaScript 示例使用顶层 await，并需要 Node.js 14.8+ 及 ES 模块。请在 package.json 中添加 "type": "module"，或将代码包装在 (async () => { ... })(); 中。

npm install openai @anthropic-ai/sdk

import OpenAI from "openai";

const openai = new OpenAI();

// Pricing per million tokens (verify current pricing at openai.com/pricing)
const PRICING = {
  "gpt-4o": { input: 2.5, output: 10.0 },
  "gpt-4o-mini": { input: 0.15, output: 0.6 },
};

async function trackedCompletion(model, messages, label = "default") {
  const pricing = PRICING[model];
  if (!pricing) {
    throw new Error(
      `Model "${model}" not found in PRICING table. ` +
      `Add it or verify the model name. Known models: ${Object.keys(PRICING).join(", ")}`
    );
  }

  let response;
  const MAX_RETRIES = 3;
  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    try {
      response = await openai.chat.completions.create({ model, messages });
      break;
    } catch (err) {
      if (err?.status === 429 && attempt < MAX_RETRIES - 1) {
        const wait = Math.pow(2, attempt) * 1000;
        console.warn(`[${label}] Rate limited. Retrying in ${wait}ms`);
        await new Promise(r => setTimeout(r, wait));
      } else {
        throw err;
      }
    }
  }

  if (!response?.usage) {
    throw new Error(`[${label}] response.usage is null — check for streaming mode`);
  }

  const { prompt_tokens, completion_tokens } = response.usage;

  const inputCost = (prompt_tokens / 1_000_000) * pricing.input;
  const outputCost = (completion_tokens / 1_000_000) * pricing.output;
  const totalCost = inputCost + outputCost;

  console.log(`[${label}] Model: ${model}`);
  console.log(`  Prompt tokens: ${prompt_tokens}`);
  console.log(`  Completion tokens: ${completion_tokens}`);
  console.log(`  Input cost:  $${inputCost.toFixed(6)}`);
  console.log(`  Output cost: $${outputCost.toFixed(6)}`);
  console.log(`  Total cost:  $${totalCost.toFixed(6)}`);

  return { response, prompt_tokens, completion_tokens, totalCost };
}

// Compare baseline vs compressed
const baseline = await trackedCompletion(
  "gpt-4o",
  [
    { role: "system", content: "Your original 500-token system prompt here..." },
    { role: "user", content: "Review text here..." },
  ],
  "baseline"
);

const compressed = await trackedCompletion(
  "gpt-4o",
  [
    { role: "system", content: "Your compressed 200-token prompt here..." },
    { role: "user", content: "Review text here..." },
  ],
  "compressed"
);

const savings = ((baseline.totalCost - compressed.totalCost) / baseline.totalCost) * 100;
console.log(`
Cost reduction: ${savings.toFixed(1)}%`);

You can drop this wrapper into any production pipeline to continuously monitor token spend and validate that compression delivers expected savings.您可以将此包装器放入任何生产流水线中，以持续监控 Token 支出并验证压缩是否带来了预期的节省。

Technique 2: Semantic Caching

What Semantic Caching Is and How It Differs from Exact-Match Caching什么是语义缓存及其与精确匹配缓存的区别

Exact-match caching only returns a stored result when the incoming request is identical, character for character, to a previously seen request. Semantic caching uses embedding-based similarity to recognize that "What is the capital of France?" and "Tell me France's capital city" should return the same cached response. This increases cache hit rates significantly for applications where users phrase similar questions in different ways.精确匹配缓存仅在传入请求与之前见过的请求逐字完全相同时才返回存储的结果。语义缓存使用基于嵌入的相似度来识别“法国的首都是哪里？”和“告诉我法国的首都城市”应该返回相同的缓存响应。这对于用户以不同方式表达类似问题的应用程序，显著提高了缓存命中率。

Provider-native caching and application-layer semantic caching solve different problems. OpenAI and Anthropic's prompt caching discount the cost of resending identical prompt prefixes. Application-layer semantic caching avoids the API call entirely when a sufficiently similar query has already been answered.提供商原生缓存和应用层语义缓存解决的是不同的问题。OpenAI 和 Anthropic 的提示词缓存对重新发送相同的提示词前缀进行折扣。应用层语义缓存则在已回答过足够相似的查询时，完全避免了 API 调用。

Implementing Application-Layer Semantic Caching实现应用层语义缓存

Note: The in-memory cache below is for demonstration only and is not production-safe. It has no TTL and uses a simple size cap for eviction, meaning it will not handle expiration or sophisticated eviction strategies. For production use, replace with Redis (using RediSearch for vector similarity) or a dedicated vector database with TTL and eviction configured.注意：下方的内存缓存仅供演示，不适用于生产环境。它没有 TTL，并使用简单的容量上限进行驱逐，这意味着它无法处理过期或复杂的驱逐策略。在生产环境中使用时，请替换为 Redis（使用 RediSearch 进行向量相似度搜索）或配置了 TTL 和驱逐策略的专用向量数据库。

import threading
import time
import numpy as np
from openai import OpenAI, RateLimitError, APIError

client = OpenAI()

# Thread-safe bounded in-memory vector cache — DEMONSTRATION ONLY (see note above)
_cache_lock = threading.Lock()
_cache: list[dict] = []  # List of {"embedding": np.ndarray, "query": str, "response": str}
CACHE_MAX_SIZE = 10_000       # evict oldest when exceeded
SIMILARITY_THRESHOLD = 0.95


def get_embedding(text: str) -> np.ndarray:
    result = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(result.data[0].embedding)


def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    if norm_a == 0.0 or norm_b == 0.0:
        return 0.0
    return float(np.dot(a, b) / (norm_a * norm_b))


def cached_completion(user_query: str, system_prompt: str, model: str = "gpt-4o") -> str:
    query_embedding = get_embedding(user_query)

    # Search cache for similar queries (thread-safe read)
    with _cache_lock:
        for entry in _cache:
            similarity = cosine_similarity(query_embedding, entry["embedding"])
            if similarity >= SIMILARITY_THRESHOLD:
                print(f"Cache HIT (similarity: {similarity:.4f})")
                return entry["response"]

    # Cache miss — call the API (outside the lock to avoid blocking other threads)
    print("Cache MISS — calling API")
    response = None
    max_retries = 3
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": user_query},
                ]
            )
            break
        except RateLimitError:
            wait = 2 ** attempt
            print(f"Rate limited. Retrying in {wait}s (attempt {attempt + 1}/{max_retries})")
            time.sleep(wait)
        except APIError as e:
            print(f"API error on attempt {attempt + 1}: {e}")
            if attempt == max_retries - 1:
                raise

    if response is None:
        raise RuntimeError("Exceeded max retries for API call")
    if response.usage is None:
        raise ValueError("response.usage is None — streaming mode is not supported here")

    result = response.choices[0].message.content

    # Store in cache (thread-safe write with eviction)
    with _cache_lock:
        if len(_cache) >= CACHE_MAX_SIZE:
            _cache.pop(0)   # evict oldest; use collections.deque for O(1)
        _cache.append({
            "embedding": query_embedding,
            "query": user_query,
            "response": result
        })

    return result


# First call — cache miss
result1 = cached_completion(
    "What are the main features of the iPhone 15 Pro?",
    "You are a product expert. Answer concisely."
)

# Second call — semantically similar, should hit cache
result2 = cached_completion(
    "Tell me the key features of Apple's iPhone 15 Pro",
    "You are a product expert. Answer concisely."
)

For production use, replacing the in-memory list with Redis using its vector search capability (RediSearch) or a dedicated vector database provides persistence and scalability. The embedding call itself is very cheap: OpenAI's text-embedding-3-small costs $0.02 per million tokens (as of the time of writing — verify current pricing at openai.com/pricing before projecting costs).在生产环境中使用时，将内存列表替换为具有向量搜索能力的 Redis (RediSearch) 或专用向量数据库可提供持久性和可扩展性。嵌入调用本身非常便宜：OpenAI 的 text-embedding-3-small 每百万 Token 收费 0.02 美元（截至撰写时——在预测成本前请在 openai.com/pricing 核实当前价格）。

Using Provider-Native Prompt Caching使用提供商原生提示词缓存

OpenAI's prompt caching is automatic. When the first 1,024 or more tokens of a prompt match a previous request exactly, cached tokens are billed at a 50% discount. No code changes are required, but structuring prompts so that the static system instructions appear first and variable content appears last maximizes cache hit rates.OpenAI 的提示词缓存是自动的。当提示词的前 1,024 个或更多 Token 与之前的请求完全匹配时，缓存的 Token 将享受 50% 的折扣。无需更改代码，但将静态系统指令放在前面、变量内容放在最后，可以最大化缓存命中率。

Note: OpenAI's automatic prompt caching only activates when the matching prompt prefix is at least 1,024 tokens. Prompts shorter than this threshold will not benefit from caching.注意：OpenAI 的自动提示词缓存仅在匹配的提示词前缀至少为 1,024 个 Token 时才会激活。短于此阈值的提示词将无法从缓存中受益。

Anthropic's prompt caching is explicit and offers steeper discounts. Cache reads cost 90% less than base input pricing. Cache writes cost 25% more, which is worth noting as a cost factor for low-traffic deployments where cache writes may outnumber reads. The developer places cache_control breakpoints to mark which prompt segments should be cached.Anthropic 的提示词缓存是显式的，并提供更大幅度的折扣。缓存读取成本比基础输入定价低 90%。缓存写入成本高出 25%，对于低流量部署而言，这值得注意，因为缓存写入次数可能超过读取次数。开发人员放置 cache_control 断点来标记哪些提示词片段应该被缓存。

Note: Anthropic requires the cached segment to be at least 1,024 tokens for cache_control to take effect. The example below uses a shortened prompt for readability; in practice, expand or combine segments to meet the ≥1,024 token threshold. Confirm caching activated by checking cache_creation_input_tokens > 0 in the response.注意：Anthropic 要求缓存片段至少为 1,024 个 Token，cache_control 才会生效。下方的示例为了可读性使用了缩短的提示词；在实践中，请扩展或组合片段以满足 ≥1,024 个 Token 的阈值。通过检查响应中的 cache_creation_input_tokens > 0 来确认缓存已激活。

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

// This prompt is shortened for readability. In practice, the cached segment
// must be at least 1,024 tokens for cache_control to activate.
const systemPrompt = `You are an expert product review analyst. Extract entities 
from reviews as JSON with fields: name, type (product/brand/feature), sentiment 
(positive/negative/neutral). Return only valid JSON. Handle comparisons by creating 
separate entries. Normalize entity names to full brand names.`;

async function analyzeReview(reviewText) {
  let response;
  try {
    response = await anthropic.messages.create({
      model: "claude-3-5-sonnet-20241022", // Verify current model ID at docs.anthropic.com/en/docs/about-claude/models
      max_tokens: 1024,
      system: [
        {
          type: "text",
          text: systemPrompt,
          cache_control: { type: "ephemeral" },
        },
      ],
      messages: [{ role: "user", content: reviewText }],
    });
  } catch (err) {
    if (err?.status === 429) {
      console.warn("Rate limited by Anthropic. Implement retry logic for production use.");
    }
    throw err;
  }

  console.log("Input tokens:", response.usage.input_tokens);
  console.log("Cache creation tokens:", response.usage.cache_creation_input_tokens || 0);
  console.log("Cache read tokens:", response.usage.cache_read_input_tokens || 0);

  if (!response.content || response.content.length === 0 || response.content[0].type !== "text") {
    throw new Error("Unexpected response content format from Anthropic API");
  }

  return response.content[0].text;
}

// First call — cache write (25% premium on system prompt tokens)
await analyzeReview("The Sony WH-1000XM5 has great ANC but feels flimsy.");

// Subsequent calls — cache read (90% discount on system prompt tokens)
await analyzeReview("Samsung Galaxy S24 Ultra camera is incredible, battery is mediocre.");
await analyzeReview("MacBook Pro M3 performance is outstanding but it runs hot.");

Anthropic's cached prompt content has a minimum length requirement of 1,024 tokens and a time-to-live of 5 minutes from the last cache write; cache reads do not extend the TTL. For high-throughput applications making multiple calls per minute with the same system prompt, the 90% read discount accumulates rapidly. In low-traffic scenarios, be aware that cache writes cost 25% more than standard input pricing, so infrequent usage patterns may not see net savings from caching.Anthropic 的缓存提示词内容有 1,024 个 Token 的最小长度要求，且自上次缓存写入起有 5 分钟的生存时间（TTL）；缓存读取不会延长 TTL。对于每分钟进行多次相同系统提示词调用的高吞吐量应用程序，90% 的读取折扣会迅速累积。在低流量场景中，请注意缓存写入成本比标准输入定价高 25%，因此不频繁的使用模式可能无法从缓存中获得净节省。

Cache Invalidation and Freshness缓存失效与新鲜度

Set TTLs based on how frequently the underlying data or instructions change. For static system prompts, long TTLs or no expiration are appropriate. For queries against rapidly changing data, such as real-time pricing or inventory, semantic caching introduces stale response risk. User-specific dynamic queries with personal context should bypass the cache entirely.根据底层数据或指令的变化频率设置 TTL。对于静态系统提示词，较长的 TTL 或不过期是合适的。对于针对快速变化数据（如实时定价或库存）的查询，语义缓存会引入陈旧响应的风险。带有个人上下文的用户特定动态查询应完全绕过缓存。

Technique 3: Chain-of-Thought Pruning for Production

Why CoT Reasoning Inflates Output Costs为什么思维链（CoT）推理会增加输出成本

Chain-of-thought prompting is valuable during development and evaluation because it makes the model's reasoning auditable. In production, however, downstream systems consume only the final answer. CoT reasoning can inflate output length by 3x to 5x (this is a commonly observed range and varies by task), and since output tokens carry the highest per-token cost, this represents a 3x to 5x increase in output cost that adds no value to the deployed system.思维链提示词在开发和评估期间很有价值，因为它使模型的推理过程可审计。然而，在生产环境中，下游系统仅消耗最终答案。CoT 推理可以将输出长度增加 3 倍到 5 倍（这是一个常见的观察范围，具体取决于任务），由于输出 Token 的单位成本最高，这代表了 3 倍到 5 倍的输出成本增加，而这对已部署的系统没有任何价值。

CoT reasoning can inflate output length by 3x to 5x, and since output tokens carry the highest per-token cost, this represents a 3x to 5x increase in output cost that adds no value to the deployed system.CoT 推理可以将输出长度增加 3 倍到 5 倍，由于输出 Token 的单位成本最高，这代表了 3 倍到 5 倍的输出成本增加，而这对已部署的系统没有任何价值。

Strategies for Pruning CoT in Production生产环境中剪枝 CoT 的策略

The most direct approach: instruct the model to return only the final answer. Combining this with structured output mode (JSON) constrains the response shape and eliminates explanatory prose.最直接的方法：指示模型仅返回最终答案。将其与结构化输出模式（JSON）结合使用，可以限制响应形状并消除解释性散文。

Anthropic's extended thinking feature (available on Claude 3.7 Sonnet and later compatible models) provides a budget_tokens parameter that caps the number of tokens the model can spend on internal reasoning. Verify model support in Anthropic's extended thinking documentation before use. This allows controlled reasoning depth without unlimited output expansion.Anthropic 的扩展思考功能（适用于 Claude 3.7 Sonnet 及后续兼容模型）提供了一个 budget_tokens 参数，用于限制模型在内部推理上可花费的 Token 数量。使用前请在 Anthropic 的扩展思考文档中验证模型支持。这允许在不无限制扩展输出的情况下控制推理深度。

import time
from openai import OpenAI, RateLimitError, APIError

client = OpenAI()

review = """The Bose QuietComfort Ultra earbuds deliver exceptional sound quality 
with deep bass and clear highs. The noise cancellation is top-tier, rivaling 
over-ear headphones. However, the fit can be uncomfortable during long sessions, 
and the case is unnecessarily bulky. Battery life of 6 hours is decent but not 
class-leading. At $299, they're expensive but justified for audiophiles."""

max_retries = 3

# WITH chain-of-thought
cot_response = None
for attempt in range(max_retries):
    try:
        cot_response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Extract product entities with sentiment. Think step by step."},
                {"role": "user", "content": review}
            ]
        )
        break
    except RateLimitError:
        wait = 2 ** attempt
        print(f"Rate limited. Retrying in {wait}s (attempt {attempt + 1}/{max_retries})")
        time.sleep(wait)
    except APIError as e:
        print(f"API error on attempt {attempt + 1}: {e}")
        if attempt == max_retries - 1:
            raise

if cot_response is None:
    raise RuntimeError("Exceeded max retries for CoT API call")
if cot_response.usage is None:
    raise ValueError("cot_response.usage is None — streaming mode is not supported here")

# WITHOUT chain-of-thought — constrained to JSON only
direct_response = None
for attempt in range(max_retries):
    try:
        direct_response = client.chat.completions.create(
            model="gpt-4o",
            max_completion_tokens=256,  # Cap output length to prevent runaway generation
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": "Extract entities as JSON: {\"entities\": [{\"name\": str, \"type\": str, \"sentiment\": str}]}. Return ONLY the JSON."},
                {"role": "user", "content": review}
            ]
        )
        break
    except RateLimitError:
        wait = 2 ** attempt
        print(f"Rate limited. Retrying in {wait}s (attempt {attempt + 1}/{max_retries})")
        time.sleep(wait)
    except APIError as e:
        print(f"API error on attempt {attempt + 1}: {e}")
        if attempt == max_retries - 1:
            raise

if direct_response is None:
    raise RuntimeError("Exceeded max retries for direct API call")
if direct_response.usage is None:
    raise ValueError("direct_response.usage is None — streaming mode is not supported here")

print(f"CoT output tokens: {cot_response.usage.completion_tokens}")
print(f"Direct output tokens: {direct_response.usage.completion_tokens}")

# Pricing per million output tokens for GPT-4o (verify at openai.com/pricing)
OUTPUT_PRICE_PER_MILLION = 10.0

cot_cost = (cot_response.usage.completion_tokens / 1_000_000) * OUTPUT_PRICE_PER_MILLION
direct_cost = (direct_response.usage.completion_tokens / 1_000_000) * OUTPUT_PRICE_PER_MILLION
print(f"CoT output cost:    ${cot_cost:.6f}")
print(f"Direct output cost: ${direct_cost:.6f}")

The CoT version returns several paragraphs of reasoning followed by the extraction, while the direct version returns only the JSON object. On a task like this, expect a 3x or greater difference in output token count.CoT 版本返回几段推理，随后是提取结果，而直接版本仅返回 JSON 对象。在这样的任务中，预计输出 Token 数量会有 3 倍或以上的差异。

Keeping CoT for Debugging Without Paying for It在不付费的情况下保留 CoT 用于调试

A practical pattern: gate CoT behind an environment variable or feature flag. Enable CoT during development and in error-analysis pipelines. Disable it in production. When production errors surface for investigation, replay the specific failing input with CoT enabled, generating the reasoning trace on demand rather than on every request.一种实用的模式：将 CoT 放在环境变量或功能标志之后。在开发和错误分析流水线中启用 CoT。在生产环境中禁用它。当生产环境出现错误需要调查时，使用启用了 CoT 的特定失败输入进行重放，按需生成推理跟踪，而不是在每次请求时都生成。

Technique 4: Output Length Constraints

Using `max_tokens` / `max_completion_tokens` Strategically策略性地使用 max_tokens / max_completion_tokens

Most developers leave the maximum output length unset, allowing the model to generate as many tokens as it deems appropriate. This is expensive. For tasks with predictable output shapes, such as classification, extraction, or short-answer responses, setting a ceiling prevents runaway generation.大多数开发人员不设置最大输出长度，允许模型生成其认为合适的任意数量的 Token。这很昂贵。对于具有可预测输出形状的任务（如分类、提取或简短回答），设置上限可以防止失控生成。

The parameter names differ by provider: OpenAI uses max_completion_tokens, Anthropic uses max_tokens, and Google Gemini uses maxOutputTokens. To find the right ceiling, sample outputs from representative inputs during development and set the limit at 1.5x to 2x the observed p95 (the 95th percentile — i.e., the length exceeded by only 5% of outputs in your sample) output length.参数名称因提供商而异：OpenAI 使用 max_completion_tokens，Anthropic 使用 max_tokens，Google Gemini 使用 maxOutputTokens。要找到合适的上限，请在开发期间对代表性输入进行采样，并将限制设置为观察到的 p95（第 95 百分位——即只有 5% 的输出超过该长度）输出长度的 1.5 倍到 2 倍。

Structured Output as a Cost Control Mechanism结构化输出作为成本控制机制

Function calling and tool use schemas act as implicit output constraints. When the model must conform to a defined schema, it cannot generate preambles, explanations, or unnecessary fields. Note that when using tool_choice to force a function call, the model's response content will be null — the actual payload is in tool_calls[0].function.arguments, which must be parsed as JSON.函数调用和工具使用模式充当了隐式的输出约束。当模型必须符合定义的模式时，它无法生成前言、解释或不必要的字段。请注意，当使用 tool_choice 强制进行函数调用时，模型响应的内容将为空——实际载荷位于 tool_calls[0].function.arguments 中，必须将其解析为 JSON。

import OpenAI from "openai";

const openai = new OpenAI();

// Pricing per million output tokens for GPT-4o (verify at openai.com/pricing)
const OUTPUT_PRICE_PER_MILLION = 10.0;

const review = `The Dyson V15 Detect has incredible suction power and the laser dust 
detection is genuinely useful. But at $750 it's overpriced, and the battery only 
lasts 25 minutes on max power. The attachments are well-designed.`;

// Unconstrained prose response
let proseResponse;
try {
  proseResponse = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "Extract product entities with sentiment from this review." },
      { role: "user", content: review },
    ],
  });
} catch (err) {
  if (err?.status === 429) {
    console.warn("Rate limited. Implement retry logic for production use.");
  }
  throw err;
}

if (!proseResponse?.usage) {
  throw new Error("proseResponse.usage is null — check for streaming mode");
}

// Structured function calling response
let structuredResponse;
try {
  structuredResponse = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "Extract product entities with sentiment." },
      { role: "user", content: review },
    ],
    tools: [
      {
        type: "function",
        function: {
          name: "extract_entities",
          description: "Extract entities from a product review",
          parameters: {
            type: "object",
            properties: {
              entities: {
                type: "array",
                items: {
                  type: "object",
                  properties: {
                    name: { type: "string" },
                    type: { type: "string", enum: ["product", "brand", "feature"] },
                    sentiment: { type: "string", enum: ["positive", "negative", "neutral"] },
                  },
                  required: ["name", "type", "sentiment"],
                },
              },
            },
            required: ["entities"],
          },
        },
      },
    ],
    tool_choice: { type: "function", function: { name: "extract_entities" } },
  });
} catch (err) {
  if (err?.status === 429) {
    console.warn("Rate limited. Implement retry logic for production use.");
  }
  throw err;
}

if (!structuredResponse?.usage) {
  throw new Error("structuredResponse.usage is null — check for streaming mode");
}

// Extract the tool call payload (content is null for tool_choice responses)
const message = structuredResponse.choices[0].message;

if (!message.tool_calls || message.tool_calls.length === 0) {
  throw new Error("No tool_calls returned. Check tool_choice config.");
}

const rawArgs = message.tool_calls[0].function.arguments;

let entities;
try {
  entities = JSON.parse(rawArgs).entities;
} catch (e) {
  throw new Error(`Failed to parse tool arguments as JSON: ${rawArgs}`);
}

console.log(`Prose completion tokens: ${proseResponse.usage.completion_tokens}`);
console.log(`Structured completion tokens: ${structuredResponse.usage.completion_tokens}`);
console.log("Extracted entities:", entities);

const proseCost = (proseResponse.usage.completion_tokens / 1_000_000) * OUTPUT_PRICE_PER_MILLION;
const structuredCost = (structuredResponse.usage.completion_tokens / 1_000_000) * OUTPUT_PRICE_PER_MILLION;
console.log(`Prose output cost:      $${proseCost.toFixed(6)}`);
console.log(`Structured output cost: $${structuredCost.toFixed(6)}`);

The structured response constrains the model to populating only the defined fields, while the prose response includes introductory text, explanations of each entity, and a closing summary. In practice, structured output produces 2x to 4x fewer tokens than unconstrained prose for extraction tasks. Run the code above on your own inputs and log the difference.结构化响应将模型限制为仅填充定义的字段，而散文响应则包含介绍性文本、每个实体的解释以及总结。在实践中，对于提取任务，结构化输出产生的 Token 比不受限的散文少 2 倍到 4 倍。在您自己的输入上运行上述代码并记录差异。

Cost Comparison Table: Before and After Across 5 Models

The following table shows estimated costs for a standardized task, extracting three entities from a two-paragraph product review, run 1,000 times. Baseline uses a verbose 500-token system prompt with unconstrained output. Optimized uses a compressed 200-token prompt with structured JSON output.下表显示了针对标准化任务（从两段产品评论中提取三个实体，运行 1,000 次）的预估成本。基准测试使用冗长的 500 Token 系统提示词和不受限的输出。优化版本使用压缩的 200 Token 提示词和结构化 JSON 输出。

Note on pricing: GPT-4o: $2.50/$10.00 per million input/output tokens. GPT-4o mini: $0.15/$0.60. Claude 3.5 Sonnet: $3.00/$15.00. Claude 3.5 Haiku (Anthropic's lower-cost model tier): $0.80/$4.00. Gemini 1.5 Flash: $0.075/$0.30 (under 128K tokens). All prices are as of the time of writing — verify at each provider's pricing page before projecting costs.定价说明：GPT-4o：每百万输入/输出 Token 2.50/10.00 美元。GPT-4o mini：0.15/0.60 美元。Claude 3.5 Sonnet：3.00/15.00 美元。Claude 3.5 Haiku（Anthropic 的低成本模型层级）：0.80/4.00 美元。Gemini 1.5 Flash：0.075/0.30 美元（128K Token 以内）。所有价格均为撰写时的数据——在预测成本前请在各提供商的定价页面核实。

Model	Baseline Input	Compressed Input	Baseline Output	Constrained Output	Baseline Cost/1K	Optimized Cost/1K	Savings
GPT-4o	580	280	350	120	$4.95	$1.90	62%
GPT-4o mini	580	280	350	120	$0.30	$0.11	63%
Claude 3.5 Sonnet	580	280	350	120	$6.99	$2.64	62%
Claude 3.5 Haiku	580	280	350	120	$1.86	$0.70	62%
Gemini 1.5 Flash	580	280	350	120	$0.15	$0.06	60%

The savings percentages are consistent by construction, since token reductions are fixed and pricing scales linearly. Models with higher output-to-input price ratios, like Claude 3.5 Sonnet at 5x, show slightly higher absolute dollar savings. The Gemini 1.5 Flash savings, while proportionally similar, represent a much smaller absolute dollar figure because the base pricing is already very low. These figures do not include additional savings from semantic caching, which would further reduce costs proportional to cache hit rate.节省百分比在结构上是一致的，因为 Token 的减少是固定的，且定价是线性缩放的。输出与输入价格比率更高的模型（如 Claude 3.5 Sonnet 的 5 倍）显示出略高的绝对美元节省。Gemini 1.5 Flash 的节省虽然比例相似，但代表的绝对美元数字要小得多，因为其基础定价已经非常低。这些数字不包括语义缓存带来的额外节省，如果包含，成本将根据缓存命中率进一步降低。

Combining All Four Techniques: A Real-World Optimization Pipeline

Recommended Order of Operations推荐的操作顺序

Apply the techniques in order of effort-to-impact ratio:按努力与影响比率的顺序应用这些技术：

Compress prompts. This delivers the largest input savings and takes the least effort — you only rewrite prompts.压缩提示词。这能带来最大的输入节省，且工作量最小——您只需重写提示词。
Constrain outputs using max_completion_tokens (OpenAI) or max_tokens (Anthropic) and structured output schemas. This targets the most expensive token category with minimal code changes.使用 max_completion_tokens (OpenAI) 或 max_tokens (Anthropic) 以及结构化输出模式限制输出。这以最小的代码更改针对最昂贵的 Token 类别。
Prune chain-of-thought for production. This requires a conditional flag but yields 3x to 5x output token reductions.在生产环境中剪枝思维链。这需要一个条件标志，但能带来 3 倍到 5 倍的输出 Token 减少。
Add semantic caching. This demands the most infrastructure (embedding generation, a vector store) but delivers the highest long-term savings at scale because it eliminates API calls entirely.添加语义缓存。这需要最多的基础设施（嵌入生成、向量存储），但由于它完全消除了 API 调用，因此在大规模应用中能带来最高的长期节省。

Estimating Your Savings估算您的节省

The savings formula: (baseline_cost - optimized_cost) / baseline_cost. As an estimate based on the token reductions demonstrated above, prompt compression saves 20% to 40% on input tokens. Output constraints save 30% to 50% on output tokens. Caching saves proportionally to hit rate — even a 30% hit rate eliminates nearly a third of all API calls.节省公式：(基准成本 - 优化成本) / 基准成本。基于上述展示的 Token 减少量，提示词压缩可节省 20% 到 40% 的输入 Token。输出限制可节省 30% 到 50% 的输出 Token。缓存的节省与命中率成正比——即使是 30% 的命中率也能消除近三分之一的 API 调用。

The 60%+ aggregate figure is realistic when at least three of the four techniques target a workload with repeated query patterns and predictable output shapes. Workloads with highly unique queries and variable-length outputs will see lower caching benefits but can still achieve 40% to 50% savings from compression and output constraints alone.当至少四种技术中的三种针对具有重复查询模式和可预测输出形状的工作负载时，60% 以上的总节省比例是现实的。具有高度唯一查询和可变长度输出的工作负载，缓存收益会较低，但仅通过压缩和输出限制仍可实现 40% 到 50% 的节省。

Start With the Lowest-Hanging Fruit

The four techniques covered here — prompt compression, semantic caching, chain-of-thought pruning, and output length constraints — form a practical framework for LLM token optimization that works across providers and models. The highest-priority first step is not implementing any technique but instrumenting token logging on every API call. Without a baseline measurement, savings cannot be quantified or validated.本文涵盖的四种技术——提示词压缩、语义缓存、思维链剪枝和输出长度限制——构成了一个适用于各提供商和模型的实用 LLM Token 优化框架。优先级最高的第一步不是实施任何技术，而是在每次 API 调用中植入 Token 日志记录。没有基准测量，节省就无法量化或验证。

The highest-priority first step is not implementing any technique but instrumenting token logging on every API call. Without a baseline measurement, savings cannot be quantified or validated.优先级最高的第一步不是实施任何技术，而是在每次 API 调用中植入 Token 日志记录。没有基准测量，节省就无法量化或验证。

For implementation details, see the LLMLingua repository, OpenAI's prompt caching guide, Anthropic's prompt caching documentation, and Google's context caching reference. Check current pricing on each provider's pricing page before running cost projections.有关实现细节，请参阅 LLMLingua 存储库、OpenAI 的提示词缓存指南、Anthropic 的提示词缓存文档以及 Google 的上下文缓存参考。在进行成本预测前，请检查各提供商定价页面上的当前价格。

Prompt Compression and Cache Tuning: Cut Your LLM API Costs by 60%提示词压缩与缓存调优：将您的 LLM API 成本降低 60%