SentencePieceSentencePiece

SentencePiece is a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems (such as Large Language Models) where the vocabulary size is fixed prior to training.SentencePiece 是个文本分词工具。快，轻，不用人工标注。给神经网络生成系统用，比如大语言模型。训练前，词表大小就定死了。

It implements subword units—including Byte-Pair-Encoding (BPE) [Sennrich et al.] and the unigram language model [Kudo.]—with the ability to train directly from raw sentences. By treating input text as a raw sequence of Unicode characters, SentencePiece enables a purely end-to-end, language-independent pipeline that completely eliminates the need for language-specific pre- or post-processing.它用子词（subword）切分。BPE、Unigram 都能做。直接拿原始文本训练。把输入看作 Unicode 字符流。端到端，不挑语言。省去了前处理和后处理的麻烦。

This is not an official Google product.这不是 Google 的官方产品。

---

Quick Start (Python)快速上手 (Python)

SentencePiece provides an easy-to-use Python module. Install it via pip:Python 调起来很方便。用 pip 安装：

pip install sentencepiece

Basic Example基础示例

Here is how to train a model, encode text into tokens/IDs, and decode them back to the original string:训练模型、转成 Token/ID、再还原回字符串，看这里：

import sentencepiece as spm

# 1. Train a model directly from a raw text file.
# (No pre-tokenization or language-specific preprocessing required!)
spm.SentencePieceTrainer.train(
    input='data/botchan.txt', 
    model_prefix='m', 
    vocab_size=1000
)

# 2. Load the trained model.
sp = spm.SentencePieceProcessor(model_file='m.model')

# 3. Encode raw text into subword pieces (strings) or vocabulary IDs (integers).
text = "I saw a girl with a telescope."
pieces = sp.encode(text, out_type=str)
ids = sp.encode(text, out_type=int)

print(f"Pieces: {pieces}")
# Output: ['▁I', '▁saw', '▁a', '▁girl', '▁with', '▁a', '▁', 'te', 'le', 's', 'c', 'o', 'pe', '.']

print(f"IDs:    {ids}")
# Output: [9, 459, 11, 939, 44, 11, 4, 142, 82, 8, 28, 21, 132, 6]

# 4. Decode IDs or pieces back into the original text.
# The reconstruction is completely lossless and reversible!
print(sp.decode(ids))
# Output: "I saw a girl with a telescope."

print(sp.decode(pieces))
# Output: "I saw a girl with a telescope."

---

Why SentencePiece?为什么选 SentencePiece？

1. Reversible & Lossless Tokenization (Whitespace as a Basic Symbol)1. 可逆且无损 (空格作为符号)

Traditional tokenizers drop whitespace information (e.g., treating Tokenize("World.") identically to Tokenize("World .")), making detokenization ambiguous and language-dependent.老式分词器爱丢掉空格。比如 "World." 和 "World ."，分出来可能一样。还原回去，就乱了，还得看语言。

SentencePiece treats the input text as a raw sequence of Unicode characters. It escapes whitespaces with a meta-symbol ▁ (U+2581) and includes it in the tokenization. This design ensures that detokenization is a simple, lossless string join operation, entirely independent of the language:SentencePiece 把输入当 Unicode 字符流。空格换成 ▁ (U+2581) 这个符号。分词时带上它。还原就是简单的字符串拼接。不丢信息，什么语言都通用。

# Lossless detokenization
original_text = "".join(pieces).replace("▁", " ")

2. Purely Data-Driven & Language-Independent2. 数据驱动，不挑语言

SentencePiece trains tokenization and detokenization models directly from raw sentences. It does not require language-specific pre-tokenizers (such as Moses, MeCab, or KyTea). This makes it highly effective for languages without explicit word boundaries, such as Chinese, Japanese, and Korean.直接拿原始句子训练。不用 Moses、MeCab 这些专门的分词器。中、日、韩这些没空格的语言，用起来也顺手。

3. Subword Regularization & BPE-Dropout3. 子词正则化与 BPE-Dropout

To improve the robustness and accuracy of translation and language models, SentencePiece supports on-the-fly subword sampling during training. By sampling different segmentations for the same input text (Subword Regularization for Unigram, BPE-Dropout for BPE), it virtually augments your training data and makes the model more resilient to spelling variations and noise.训练时，它能随机采样切分方式（Unigram 用正则化，BPE 用 Dropout）。相当于扩充了训练数据。模型见多了切法，拼写变一变，也不容易乱。

# Sample different segmentations on-the-fly
for _ in range(3):
    print(sp.encode('New York', out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1))
# May output:
# ['▁', 'N', 'e', 'w', '▁York']
# ['▁New', '▁York']
# ['▁New', '▁Y', 'o', 'r', 'k']

4. Fast, Lightweight, and Self-Contained4. 快，轻，自包含

• Performance: Written in highly optimized C++. Segmentation speed is around 50,000 sentences per second, with a memory footprint of only ~6MB.性能：C++ 写成。一秒能切五万句。内存只要 6MB。
• Self-Contained: The generated .model file contains the entire normalization rules, vocabulary mapping, and segmentation model. You are guaranteed to get the exact same tokenization results in any environment (C++, Python, Go, etc.) as long as you use the same model file.自包含：生成的 .model 文件，存了归一化规则、词表、模型参数。不管你在 C++、Python 还是 Go 里用，只要模型文件一样，结果就一样。

---

Performance Benchmark (SentencePiece vs. Hugging Face Fast)性能对比 (SentencePiece 对比 Hugging Face Fast)

Benchmark Setup测试环境

• Environment: 24-core CPU, Python 3.13.环境：24 核 CPU，Python 3.13。
• Dataset: Balanced raw multilingual text from FLORES-200 (parallel sentences in English, Chinese, Japanese, and Thai; 11.29 MB, 60,720 lines). CJK and Thai texts are raw and do not contain artificial space delimiters.数据：FLORES-200 多语言集（英、中、日、泰语，11.29 MB，60,720 行）。中日泰语为原始文本，没加人工空格。
• Batch Request Size: The entire dataset (60,720 sentences) is fed as a single batch request (a single Python list[str]) in one call.批处理：一次性塞入 60,720 句（Python list[str]）。
• Metric: Encoding throughput in MB/s (higher is better).指标：编码吞吐量 (MB/s)，越高越好。

1. Unigram Model: T5-base (32k vocab)1. Unigram 模型：T5-base (32k 词表)

Tokenizer	1 Thread	2 Threads	4 Threads	8 Threads	16 Threads	24 Threads
SentencePiece	27.41	43.83	71.62	102.08	123.33	127.60
Hugging Face Fast	3.78	7.15	12.45	20.33	27.00	31.49

2. BPE Model: Gemma 3 (256k vocab)2. BPE 模型：Gemma 3 (256k 词表)

Tokenizer	1 Thread	2 Threads	4 Threads	8 Threads	16 Threads	24 Threads
SentencePiece	7.44	12.82	23.03	36.66	48.65	52.43
Hugging Face Fast	3.66	6.37	10.45	15.54	21.05	20.48

Why performance does not scale linearly:为什么性能不线性增长：

While the core tokenization (C++ or Rust) runs in parallel, the final step of converting the native results (C++ vector of vectors or Rust vector of vectors) into Python objects (list[list[int]] or list[Encoding]) is sequential and must be done on Python's main thread (GIL-locked). At high thread counts, this single-threaded serialization step becomes the dominant bottleneck, capping the scaling performance.核心分词（C++ 或 Rust）能并行。但最后把结果转成 Python 对象（list[list[int]]）这一步，得在 Python 主线程跑，受 GIL 锁限制。线程开多了，全卡在这一步，上不去。

For the detailed analysis and single-thread reference comparison, see Performance Benchmark Details.详细分析和单线程对比，看“性能测试详情”。

To run these benchmarks yourself, see the reproduction instructions and scripts.想自己跑测试，看复现说明和脚本。

---

Documentation & Resources文档与资源

For detailed guides, API references, and advanced usage, please refer to the following resources:指南、API 参考、进阶用法，看这些：

• Command Line Interface (CLI) & Build Guide命令行 (CLI) 与构建指南
• C++ API ReferenceC++ API 参考
• Python API Reference & Python Module DirectoryPython API 参考与模块目录
• Python Tokenizer Comparison Cheat SheetPython 分词器对比速查表
• Performance Benchmark Details性能测试详情
• Performance Benchmark Code & Reproduction Guide性能测试代码与复现指南
• Training Options Reference训练选项参考
• Text Normalization & Custom Rules文本归一化与自定义规则
• Special Symbols & Control Tokens特殊符号与控制 Token
• Vocabulary Piece Constraints词表限制
• Model Protobuf Schema模型 Protobuf 结构
• Docker Deployment GuideDocker 部署指南
• NLCodec BPE Trainer (Contrib)NLCodec BPE 训练器 (贡献)

---

License许可

SentencePiece is licensed under the Apache 2.0 License.SentencePiece 使用 Apache 2.0 协议。