Beyond LoRA: Can you beat the most popular fine-tuning technique? 超越 LoRA：你能击败最流行的微调技术吗？
When you plan to fine-tune a model in a parameter-efficient way, think beyond LoRA 当你计划以参数高效的方式微调模型时，请考虑超越 LoRA
If you want to fine-tune an open model on your own data, you are probably interested in so-called parameter-efficient fine-tuning, in short PEFT. This term describes techniques that significantly reduce the memory requirement to fine-tune a model. Although there are dozens of these techniques, almost everyone chooses one called “LoRA”. In this blog post, we explore whether LoRA is really the best choice, what tools are available to make an informed decision, and how you can benefit from extending your horizon beyond LoRA.如果你想在自己的数据上微调开源模型，你可能对所谓的参数高效微调（PEFT）感兴趣。该术语描述了显著降低微调模型内存需求的技术。虽然已有数十种此类技术，但几乎所有人都会选择一种叫做“LoRA”的方法。在本博客中，我们将探讨 LoRA 是否真的最佳选择、有哪些工具可以帮助你做出明智决定，以及如何通过超越 LoRA 来获益。
What is PEFT and when do you need it 什么是 PEFT 以及何时需要它
There are countless open models available, but they often aren't quite good enough for your use case. Prompting may help, but it usually isn't enough. Rather than training a new model from scratch, you should consider fine-tuning an existing one.可供选择的开源模型数量众多，但它们往往并不能完全满足你的使用场景。提示工程或许有帮助，但通常不足以解决问题。与其从头训练新模型，不如考虑对已有模型进行微调。
Fine-tuning, however, is memory-hungry: you generally need enough memory to fit the whole model several times over. Quantization reduces a model's memory footprint, but quantized models can't be fine-tuned directly. So a set of techniques emerged to cut the memory needed for fine-tuning, called "parameter-efficient fine-tuning", or PEFT.然而，微调非常消耗内存：通常需要足够的显存来容纳整个模型的多份拷贝。量化可以降低模型的内存占用，但量化模型无法直接微调。因此出现了一系列技术，用于削减微调所需的内存，这类技术被称为“参数高效微调”（PEFT）。
With PEFT, you can fine-tune a model using only a fraction of that memory and even fine-tune quantized models. It offers other advantages, such as tiny checkpoint sizes, greater resistance to catastrophic forgetting, and the ability to serve multiple fine-tunes from the same base model.使用 PEFT，你只需极少的显存即可微调模型，甚至可以微调量化模型。它还具备其他优势，例如检查点体积极小、对灾难性遗忘的抵抗力更强，以及能够从同一基模型服务多个微调版本。
At Hugging Face, we develop the PEFT library, which implements many PEFT techniques behind a unified API and integrates well with the ecosystem, for example Transformers and Diffusers. It also supports multiple quantization methods, enabling further accessibility in parameter-efficient fine-tuning. PEFT provides a good starting point, whether you want to fine-tune on your own data or you're researching a new PEFT method.在 Hugging Face，我们开发了 PEFT 库，实现了众多 PEFT 技术的统一 API，并能很好地与生态系统（如 Transformers 和 Diffusers）集成。它还支持多种量化方法，进一步提升了参数高效微调的可及性。无论是想在自己的数据上微调，还是研究新的 PEFT 方法，PEFT 都是一个很好的起点。
LoRA: The queen of fine-tuning techniques 👑 LoRA：微调技术之王 👑
One parameter-efficient fine-tuning technique that emerged early and proved to be quite effective is called “Low Rank Adaptation”, or short “LoRA”. It works by adding a handful of parameters on top of the base model, freezing the base model weights, and only training those few parameters.一种最早出现且被证明相当有效的参数高效微调技术叫做“低秩适配”（Low Rank Adaptation），简称 LoRA。它的做法是：在基模型之上添加少量参数，冻结基模型权重，仅训练这少量新增参数。
Among all PEFT techniques, LoRA is by far the most popular. Here are a few estimates:在所有 PEFT 技术中，LoRA 远远是最受欢迎的。以下是一些估计数据：
- Of a sample of 20,834 model cards on Hugging Face Hub that mention exactly one PEFT technique, 20,509 mention LoRA (98.4%).在 Hugging Face Hub 上抽取的 20,834 条仅提及一种 PEFT 技术的模型卡中，20,509 条（98.4%）提到了 LoRA。
- We checked which PEFT techniques are popular for image generation on an external site, too. Using a sample of 10,000 checkpoints, we found 7,111 to be LoRAs. The other identified PEFT techniques are LoCon (363) and DoRA (11, arguably a LoRA variant). That means 95.0% of PEFT checkpoints are LoRAs.我们还在外部站点上检查了图像生成领域的 PEFT 使用情况。对 10,000 个检查点的抽样中，发现 7,111 个是 LoRA。其他被识别的 PEFT 技术有 LoCon（363）和 DoRA（11，算作 LoRA 的变体）。这意味着 95.0% 的 PEFT 检查点是 LoRA。
- Searching for the code snippet
from peft import <PEFT CONFIG>on GitHub (example GH query), 71.3% of results are for LoRA. The runners-up are LoHa (3.7%) and AdaLoRA (3.5%).在 GitHub 上搜索 `from peft import <PEFT CONFIG>` 的代码片段（示例 GH 查询），71.3% 的结果对应 LoRA。紧随其后的是 LoHa（3.7%）和 AdaLoRA（3.5%）。
Although these estimates are not perfect, the conclusion is nonetheless that LoRA is almost certainly by far the most common PEFT technique.虽然这些估计并不完美，但结论仍然是：LoRA 几乎可以肯定是最常见的 PEFT 技术。
This could just mean that LoRA works best for everyone, and this fact is reflected in its usage statistics. There is, however, another possibility: LoRA was one of the earlier, popular PEFT techniques. So maybe its usage became self-reinforcing: LoRA has the highest visibility, the highest number of tutorials/examples, and it has the best support in downstream packages. Thus LoRA's popularity feeds on itself.这可能仅仅说明 LoRA 对大多数人来说效果最好，使用统计也反映了这一点。不过，也有另一种可能：LoRA 是较早出现且流行的 PEFT 技术之一。因此它的使用可能形成了自我强化的循环：LoRA 的可见度最高、教程/示例最多，并且在下游库中的支持最完善，从而进一步提升了其人气。
This all leads to the question: Are we all leaving performance on the table by shunning better techniques? After all, there are countless researchers whose papers claim their technique beats LoRA. Isn't that sufficient proof that we should go beyond LoRA in favor of newer techniques?这引出了一个问题：我们是否因为排斥更好的技术而在性能上留下了遗憾？毕竟有无数研究者的论文声称他们的技术击败了 LoRA。这难道不是足够的证据，说明我们应该超越 LoRA，采用更新的技术吗？
Choosing the right PEFT technique based on paper results is problematic 仅凭论文结果选择合适的 PEFT 技术是有问题的
There are dozens of papers that investigate fine-tuning techniques other than LoRA. Just in the PEFT library, there are more than 40 distinct PEFT techniques at the time of writing (and numerous more when counting variations of PEFT techniques). For almost all of them, you will find researchers claiming that their technique beats LoRA according to their benchmarks.已有数十篇论文研究了除 LoRA 之外的微调技术。仅在 PEFT 库中，就已经实现了 40 多种不同的 PEFT 方法（若计入各种变体则更多）。几乎所有方法的作者都会声称自己的技术在基准测试中优于 LoRA。
The trouble with these claims is that researchers are under pressure to provide results that beat the existing benchmark. Even without ill intent, this can bias the results, e.g. by spending less time tuning the alternative techniques compared to the one proposed by the researchers. One study found, for instance, that LoRA can match supposedly better PEFT techniques by tuning the learning rate.这些声称的问题在于，研究者往往面临必须超越已有基准的压力。即使没有恶意，这也可能导致结果偏向，例如对替代技术的调参时间少于对自己提出的技术的调参时间。已有研究发现，仅通过调节学习率，LoRA 就能匹配所谓更好的 PEFT 技术。
Another complication is that each paper chooses a different set of PEFT techniques to compare to, and a different set of benchmarks to run. And even if the same technique is compared on the same benchmark, the code is often not available or not easy to run yourself, which makes results hard to reproduce.另一个复杂因素是，每篇论文选择的对比 PEFT 技术和基准测试都不相同。即使在相同基准上比较相同技术，代码也常常不可用或难以自行运行，导致结果难以复现。
Overall, it's difficult to figure out the PEFT technique that works best for you by only checking paper results. Therefore, you might be tempted to just go with the default, LoRA.总体而言，仅凭论文结果很难判断哪种 PEFT 技术最适合你。因此，你可能会倾向于直接使用默认的 LoRA。
How we approach benchmarking in PEFT
我们在 PEFT 中的基准测试方法
At Hugging Face, we thought about how we can help users make informed decisions about which PEFT technique to use. With the PEFT library, we already provide a package that implements many PEFT techniques and exposes them with the same API. The next step is to provide benchmarks that can shed more light on the discussed issue.在 Hugging Face，我们思考如何帮助用户对使用哪种 PEFT 技术做出明智决定。PEFT 库已经提供了实现众多 PEFT 技术的统一包装。下一步是提供能够进一步阐明上述问题的基准测试。
We already had a benchmark that checks fine-tuning of LLMs on a math dataset for some time. This benchmark takes an LLM and fine-tunes it on chain-of-thought reasoning to produce the result to a mathematical question using a base model that is not instruction fine-tuned. The benchmark thus checks if the model can learn to perform mathematical reasoning and also to adjust the generated output to the expected format.我们已经有一个针对 LLM 在数学数据集上微调的基准。该基准让模型在未进行指令微调的基础模型上，通过链式思考（chain‑of‑thought）微调，以生成数学问题的答案，并检查模型是否能够学习数学推理以及将生成的输出调整为期望的格式。
To extend our findings on another modality, we also added an image generation benchmark. This one tests whether the model can be fine-tuned to learn a new concept, a cat plushy, and generate it in new contexts without forgetting existing concepts.为了在另一种模态上扩展我们的发现，我们还加入了图像生成基准。该基准测试模型是否能够微调学习一个新概念（如猫玩偶），并在新场景中生成该概念而不遗忘已有概念。
 |
 |
| Left: Sample question and answer from the MetaMathQA dataset. Right: Sample image from the cat plushy dataset. | |
All PEFT techniques are evaluated according to the exact same conditions: same base model, same dataset, same training and evaluation code, same hardware. As different users have different needs, we track more than just test performance. Besides VRAM usage, we track metrics like forgetting/drift, runtime, and checkpoint size. The results are designed to run on consumer hardware, and adding a new experiment only requires adding a new PEFT config and running a script.所有 PEFT 技术都在完全相同的条件下评估：相同的基模型、相同的数据集、相同的训练与评估代码、相同的硬件。考虑到不同用户有不同需求，我们追踪的不仅是测试性能，还包括显存占用、遗忘/漂移、运行时长以及检查点大小等指标。结果设计为可在消费级硬件上运行，新增实验只需添加一个 PEFT 配置并运行脚本。
Since we compare all PEFT techniques on equal footing and have no horse in the race, we believe that these benchmarks can draw an objective picture of how well different PEFT techniques work. We argue that if you have your own dataset, you can take a similar approach and take advantage of the PEFT library to evaluate multiple PEFT techniques.由于我们在同等条件下比较所有 PEFT 技术且没有立场偏向，我们相信这些基准能够客观呈现不同 PEFT 方法的表现。如果你有自己的数据集，也可以采用类似方法，利用 PEFT 库评估多种 PEFT 技术。
Our findings: LoRA works well but is not necessarily the best choice 我们的发现：LoRA 表现良好，但不一定是最佳选择
After finishing the benchmark runs, we found that although LoRA works well, other PEFT methods can beat it on one or multiple axes and should thus be considered. Check the image below that compares the performance of LoRA and five other PEFT techniques.完成基准运行后，我们发现虽然 LoRA 表现不错，其他 PEFT 方法在一个或多个维度上能够超越它，值得考虑。请查看下图，对比 LoRA 与另外五种 PEFT 技术的表现。
 |
| Some results from the benchmark. When it comes to test performance and memory usage, LoRA is not necessarily the best choice. Left: MetaMathQA benchmark; right: image generation benchmark. Consult this Space for the most up-to-date results. |
One way to interpret the results above is to think in terms of tradeoffs, for example: How well does the model perform on the test set vs how much memory is needed to train it? If a PEFT technique cannot be beaten on both of these metrics at the same time by any other technique, it is on the Pareto Frontier. In other words: If you want better test accuracy, you need more memory, and if you want more memory efficiency, you have to give up on accuracy.解读上述结果的一种方式是从权衡角度考虑，例如：模型在测试集上的表现 vs. 训练所需显存。如果某个 PEFT 技术在这两个指标上没有被其他技术同时超越，则它位于帕累托前沿。换句话说：想要更高的测试准确率就需要更多显存，想要更高的显存效率则必须在准确率上做出让步。
Let's take a closer look at the results for the LLM Math dataset benchmark. When it comes to test accuracy vs memory, we find that LoRA is indeed on the Pareto frontier. It achieves 53.2% test accuracy and requires 22.6 GB of VRAM at the peak. There are, however, other PEFT techniques on the Pareto Frontier. For instance, BEFT achieves 32.9% test accuracy and requires only 20.2 GB of memory at max. On the other end, we have Lily, which achieves 54.9% test accuracy but requires 25.6 GB of memory. Depending on what's more important to you, you may conclude that LoRA does not present the best tradeoff for you.让我们仔细看看 LLM 数学数据集基准的结果。在测试准确率 vs. 显存方面，LoRA 确实位于帕累托前沿。它的测试准确率为 53.2%，峰值显存需求为 22.6 GB。与此同时，其他技术也在前沿上。例如，BEFT 的测试准确率为 32.9%，最大显存仅需 20.2 GB；而 Lily 的测试准确率为 54.9%，但显存需求上升至 25.6 GB。根据你的侧重点，可能会认为 LoRA 并非最佳权衡。
 |
Test accuracy vs memory usage tradeoff of fine-tuning meta-llama/Llama-3.2-3B and evaluating it on GSM8K. LoRA does well but so do other PEFT techniques. |
It is also worth noting that even though LoRA does well on this task, we're not talking about vanilla LoRA. On one side, we have LoRA with rank stabilized initialization, which is a technique to scale the LoRA contribution differently from the default initialization and provides very good test accuracy (53.2%). On the other end, we have LoRA-FA, which uses an optimizer specialized for LoRA that freezes part of the LoRA weights and is thus more memory efficient (20.2 GB). Normal LoRA only achieves an accuracy of 48.1% at 22.5 GB memory and should thus be avoided in favor of the alternatives.值得注意的是，即使 LoRA 在该任务上表现不错，我们讨论的并非原始 LoRA。一方面是使用了秩稳定初始化的 LoRA，该技术通过不同于默认初始化的方式缩放 LoRA 的贡献，取得了 53.2% 的优秀测试准确率。另一方面是 LoRA‑FA，它使用专为 LoRA 设计的优化器，冻结部分 LoRA 权重，从而在显存上更高效（仅 20.2 GB）。普通 LoRA 的准确率只有 48.1%，显存需求 22.5 GB，因而应优先考虑上述变体。
Next let's take a look at the image generation benchmark. In the Hugging Face Space, choose “image-gen” in the “Select Task” dropdown to show the results. The goal of the task is to learn a new concept, namely a cat plushy, and generalize it to new prompts.接下来看看图像生成基准。在 Hugging Face Space 中，选择 “Select Task” 下拉框里的 “image‑gen” 即可查看结果。任务目标是学习一个新概念——猫玩偶，并在新提示下进行生成。
 |
Cat plushy image created with LoRA fine-tuned on FLUX.2-klein-base-4B. |
For this task, the main metric is “dino similarity”, which measures how much a generated image resembles the picture from a holdout test dataset, with higher values being better. As always, we also want to keep an eye on memory usage. When plotting the Pareto Frontier of these two metrics, we find that LoRA is below that frontier. Let's get concrete numbers: LoRA achieves a similarity score of 0.697 whereas OFT achieves 0.708; in terms of memory, LoRA requires 9.97 GB, and OFT requires 9.01 GB. Therefore, OFT strictly dominates LoRA on these metrics.该任务的主要指标是 “dino similarity”，用于衡量生成图像与保留测试数据集中图片的相似度，数值越高越好。我们同样关注显存使用情况。绘制这两个指标的帕累托前沿后发现 LoRA 位于前沿之下。具体数值为：LoRA 的相似度得分为 0.697，而 OFT 为 0.708；显存方面 LoRA 需要 9.97 GB，OFT 只需 9.01 GB。因此，在这两个指标上 OFT 完全优于 LoRA。
 |
Test accuracy vs memory usage tradeoff of fine-tuning FLUX.2-klein-base-4B and evaluating it on the test set. Other PEFT techniques like OFT beat LoRA in terms of test score and lower memory usage. |
Of course, you should also check the other PEFT methods that are close to the Pareto frontier, as metrics can be subject to small variations due to randomness. Also, you should explore other metrics: is runtime performance important to you or do you care about the size of the checkpoints? Choose the relevant metric from the dropdown and the picture can change considerably. For the image generation benchmark, do inspect the generated sample images to get a vibe of the fine-tuned model's capability.当然，你也应检查其他接近帕累托前沿的 PEFT 方法，因为指标可能因随机性出现细微波动。还可以探索其他维度：运行时性能是否重要？检查点大小是否关键？在下拉框中选择相应指标，图形可能会有显著变化。对于图像生成基准，建议查看生成的示例图片，以直观感受微调模型的能力。
Limitations 局限性
Objection: But the benchmarks favor one method over another!异议：基准测试偏向某一种方法！
One criticism that could be leveled at the PEFT benchmarks is that the choice of hyper-parameters may favor one technique over another. This is true, doing an exhaustive and fair hyper-parameter sweep with this many techniques is difficult. It is, however, very easy for everyone to contribute their own experiments to PEFT: If you believe that a specific PEFT technique can be improved by choosing different hyper-parameters, create a PR! We added instructions on how to do that. In a similar vein, if you want to contribute a completely new benchmark, reach out to us to discuss your idea.对 PEFT 基准的一个批评是超参数的选择可能偏向某些技术。的确，要对如此多的技术进行全面且公平的超参数搜索非常困难。不过，任何人都可以很容易地为 PEFT 贡献自己的实验：如果你认为某个 PEFT 技术可以通过不同的超参数得到改进，请提交 PR！我们已经在仓库中添加了相应的操作指南。同理，如果你想贡献全新的基准，也可以联系我们讨论想法。
Another problem with the benchmarks is that they may not fully reflect the capabilities of a specific PEFT technique. We make it possible to compare the techniques along many different dimensions and discover the best ones according to these tradeoffs. But it's impossible to capture all facets this way. For instance, one PEFT technique called Cartridges was developed to compress long prompts, which is not measured in the benchmarks. Other factors can also influence the choice, for instance:另一个问题是基准可能无法完整体现某些 PEFT 技术的能力。我们提供了多维度的比较，以帮助发现最适合的技术，但不可能覆盖所有方面。例如，有一种叫做 Cartridges 的 PEFT 技术专门用于压缩长提示，这在我们的基准中并未衡量。还有其他因素也会影响选择，例如：
- Depending on the PEFT technique, only certain layer types can be modified.不同的 PEFT 技术只能修改特定类型的层。
- Not all PEFT techniques support quantized base models (but we actively expand the support in
PEFT).并非所有 PEFT 技术都支持量化的基模型（但我们正在积极扩展 PEFT 对量化的支持）。 - Some PEFT techniques allow merging of the adapter to reduce runtime overhead but others don't.有些 PEFT 技术允许合并适配器以降低运行时开销，而有些则不支持。
The benchmarks cannot fully lift the responsibility to do your research, but they can be reasonable pointers.基准测试并不能完全替代你的研究工作，但可以作为有价值的参考。
 |
| Click on the image to peruse the PEFT shop to find the best PEFT technique for you. It allows you to browse not only by benchmark metrics but also by capabilities, like quantization support. |
Objection: But llama.cpp/vLLM/... only supports LoRA异议：llama.cpp / vLLM / … 只支持 LoRA
A limitation of using a PEFT technique other than LoRA is that they don't get the broad support in downstream packages that LoRA sees. For example, if you want to serve the model using vLLM, only LoRA checkpoints can be loaded. Thankfully, PEFT now supports converting other adapters into LoRA. That way, you can convert a non-LoRA checkpoint into LoRA and use it in vLLM or other downstream packages.使用非 LoRA 的 PEFT 技术的一个限制是，它们没有 LoRA 那样广泛的下游支持。例如，在使用 vLLM 部署模型时，仅能加载 LoRA 检查点。幸运的是，PEFT 现在支持将其他适配器转换为 LoRA，这样你就可以把非 LoRA 检查点转换为 LoRA 并在 vLLM 或其他下游库中使用。
To test this, we converted an image adapter using the GraLoRA technique into a LoRA checkpoint. The test scores were virtually identical after conversion (similarity 0.702 → 0.694, 0.260 → 0.269). Below are test images for the prompt “sks cat at the beach”:为验证这一点，我们将使用 GraLoRA 技术的图像适配器转换为 LoRA 检查点。转换前后的测试分数几乎相同（相似度 0.702 → 0.694，0.260 → 0.269）。以下是提示 “sks cat at the beach” 的测试图片：
 |
 |
| Left: Image generated by GraLoRA. Right: Image generated by the same GraLoRA checkpoint converted to a LoRA checkpoint. The images quality is comparable. | |
At the moment, we haven't implemented conversion for all PEFT techniques, but if there is demand, we will expand the support.目前我们尚未实现对所有 PEFT 技术的转换，但如果有需求，我们会继续扩展支持。
Conclusion and what you can do 结论与可采取的行动
While working on the PEFT package, we noticed that LoRA has a lot of momentum behind it, even though other PEFT techniques are potentially better. Therefore, we set out to add benchmarks to PEFT that could paint a more objective picture of how well different PEFT techniques perform on different metrics.在开发 PEFT 包的过程中，我们注意到 LoRA 具有很大的动能，即使其他 PEFT 技术可能更好。因此我们为 PEFT 添加了基准测试，以更客观地展示不同 PEFT 技术在多项指标上的表现。
Given the results we found, we can confidently conclude that LoRA is not a bad choice at all, but there are potentially better choices. Especially when checking the image generation benchmark, LoRA is beaten by other techniques. We discussed that besides metrics, other considerations must be taken into account when choosing the right PEFT technique. However, even then, we are pushing PEFT further to achieve feature parity between LoRA and those other techniques.基于我们的实验结果，可以自信地得出结论：LoRA 并非糟糕的选择，但确实存在潜在的更佳方案。尤其在图像生成基准中，LoRA 被其他技术超越。我们还讨论了除指标之外的其他考量因素，并继续推动 PEFT 向 LoRA 及其他技术的功能对齐。
Our journey is far from finished; we want to extend and improve the existing benchmarks, and we also plan to add more benchmarks in the future. We ensured that it is easy for the community to contribute, so if this is something you would like to do, please open an issue on the PEFT repository and let us know how you would like to contribute.我们的工作远未结束；我们计划扩展并改进现有基准，并在未来加入更多基准。我们已经让社区贡献变得轻松，如果你有兴趣，请在 PEFT 仓库中打开 issue，告诉我们你想如何贡献。
If you take away only one thing from this article, it is that LoRA should not be the automatic default when choosing a PEFT technique for your use case. Given the unified API provided by PEFT, changing from one PEFT technique to another is as easy as switching one config in your code. And even if you stick with LoRA, check out all the variants that are supported in PEFT: DoRA, rs-LoRA, LoRA-FA etc. Give these other techniques a try and you might be pleasantly surprised.如果只能从本文中记住一点，那就是：在为你的使用场景选择 PEFT 技术时，LoRA 不应成为自动默认。得益于 PEFT 提供的统一 API，只需在代码中切换一个配置即可更换 PEFT 方法。即使仍然使用 LoRA，也请尝试 PEFT 中支持的各种变体，如 DoRA、rs‑LoRA、LoRA‑FA 等，可能会有惊喜。
Example: Changing from LoRA to OFT using PEFT:示例：使用 PEFT 将 LoRA 切换为 OFT：
from transformers import AutoModelForCausalLM
-from peft import LoraConfig, get_peft_model
+from peft import OFTConfig, get_peft_model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B", dtype="bfloat16")
-config = LoraConfig(target_modules=["q_proj", "v_proj"])
+config = OFTConfig(target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)
