ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization

compressing a fine-tuned adapter's weight update into sparse signs plus one shared scalar, with no retraining

Prateek Yadav, Leshem Choshen, Colin Raffel, Mohit Bansal · TMLR · 2025

In one sentence

ComPEFT compresses a PEFT module's fine-tuning residual by keeping only the signs of its top-k% largest-magnitude entries and replacing all magnitudes with one tuned multiple of the task vector's standard deviation, giving 8x-50x smaller experts with no retraining.

Abstract

Parameter-efficient fine-tuning (PEFT) techniques make it possible to efficiently adapt a language model to create"expert"models that specialize to new tasks or domains. Recent techniques in model merging and compositional generalization leverage these expert models by dynamically composing modules to improve zero/few-shot generalization. Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU. To address these issues, we present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models. ComPEFT employs sparsification and ternary quantization to reduce the size of the PEFT module without performing any additional retraining while preserving or enhancing model performance. In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x. In particular, we show that ComPEFT improves with scale - stronger models exhibit higher compressibility and better performance. For example, we show that ComPEFT applied to LLaMA outperforms QLoRA by 4.16% on MMLU with a storage size reduction of up to 26x. In addition, we show that the compressed experts produced by ComPEFT maintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare it with other PEFT methods, and test ComPEFT's efficacy for compressing the residual of full-finetuning. Our code is available at https://github.com/prateeky2806/compeft.

Questions this paper answers

How much can a LoRA or QLoRA adapter be shrunk without losing accuracy?
Can fine-tuned adapters be compressed without retraining?
What compression ratio does ComPEFT get on QLoRA adapters?
ComPEFT-compressed QLoRA adapters on LLaMA-65B average 63.45% on the MMLU test set versus 59.29% for the original QLoRA adapters, a 4.16-point gain while shrinking storage 26x from 1.49 GB to about 0.058 GB.
Holds for: 5-shot MMLU with QLoRA checkpoints released by the QLoRA authors for 8 instruction-tuning datasets; density k and scaling alpha selected on a small held-out MMLU subset; storage assumes Golomb coding against 16-bit uncompressed checkpoints.
On T5-Base, T5-Large and T0-3B, ComPEFT compresses (IA)^3 and LoRA modules 12x-25x while changing average performance over 7 GLUE tasks by at most 1.3 points, from -1.3 for (IA)^3 on T5-Base to +0.1 for LoRA on T5-Large.
Holds for: 7 GLUE classification tasks (MNLI, RTE, QNLI, WNLI, SST2, MRPC, QQP), test set; alpha and k chosen on a validation set per task.
At 95% sparsity the ternary ComPEFT update has entropy of about 0.34 bits per parameter plus a 16-bit scalar, down from 16 bits per parameter for a bfloat16 task vector, a 47x reduction in communication and storage cost under a perfect coding scheme.
Holds for: Analytical entropy assuming signs of nonzero entries are uniformly distributed; realised sizes in the experiments use Golomb coding, and a two-binary-mask alternative costs 2 bits per parameter but is cheaper to compute with.
Does compressing an adapter ever make it perform better?
Why would sparsifying a task vector improve accuracy instead of hurting it?
Does ComPEFT beat the uncompressed QLoRA checkpoint?
ComPEFT improves on the original QLoRA checkpoint in 28 of 32 dataset-by-model-size configurations while compressing the LoRA module 10x-50x in storage.
Holds for: 8 instruction-tuning datasets crossed with LLaMA 7B/13B/33B/65B, 5-shot MMLU test; the 4 losses are concentrated at 7B and 13B on Unnatural Instructions, Alpaca, HH-RLHF and Guanaco.
The gain from ComPEFT over uncompressed QLoRA grows with base-model size: +0.54 points on LLaMA-7B, +1.06 on 13B, +3.44 on 33B and +4.16 on 65B MMLU, with compression factors of 16x, 20x, 16x and 26x respectively.
Holds for: LLaMA 7B/13B/33B/65B with QLoRA adapters averaged over 8 instruction-tuning datasets, 5-shot MMLU; alpha and k tuned per configuration on a small held-out MMLU subset.
On LLaMA2-70B, ComPEFT averages 67.53% MMLU at 56 MB, above STC (65.24%, 56 MB), BitDelta without training (64.73%, 99 MB) and DAREx-q at 95% sparsity (64.68%, 395 MB), and matches BitDelta with trained scale (67.46%) which requires backward passes.
Holds for: Rank-64 QLoRA on LLaMA2-70B over 5 instruction-tuning datasets; storage uses Golomb coding for ComPEFT and STC, bitmask for BitDelta and COO sparse matrices for DAREx; DAREx at 99% sparsity collapses to 45.86%.
Does adapter compression work better on bigger base models?
Are task vectors from larger language models more compressible?
How does ComPEFT scale from 7B to 65B parameters?
The gain from ComPEFT over uncompressed QLoRA grows with base-model size: +0.54 points on LLaMA-7B, +1.06 on 13B, +3.44 on 33B and +4.16 on 65B MMLU, with compression factors of 16x, 20x, 16x and 26x respectively.
Holds for: LLaMA 7B/13B/33B/65B with QLoRA adapters averaged over 8 instruction-tuning datasets, 5-shot MMLU; alpha and k tuned per configuration on a small held-out MMLU subset.
ComPEFT-compressed QLoRA adapters on LLaMA-65B average 63.45% on the MMLU test set versus 59.29% for the original QLoRA adapters, a 4.16-point gain while shrinking storage 26x from 1.49 GB to about 0.058 GB.
Holds for: 5-shot MMLU with QLoRA checkpoints released by the QLoRA authors for 8 instruction-tuning datasets; density k and scaling alpha selected on a small held-out MMLU subset; storage assumes Golomb coding against 16-bit uncompressed checkpoints.
How much faster is it to download and load a compressed adapter?
What are the real wall-clock savings from compressing expert modules?
Does adapter compression reduce CPU-to-GPU transfer time?
Downloading a ComPEFT-compressed LLaMA-65B QLoRA checkpoint from a simulated internet server takes 2.59 s versus 83.17 s uncompressed (about 32x faster), and CPU-to-GPU loading takes 18.60 ms versus 475.26 ms (about 25x faster).
Holds for: Wall-clock means over 10 repetitions per configuration, LLaMA 7B-65B QLoRA checkpoints, simulated internet server and a single 48GB A6000 host; ternary-vector compute speedups would need custom kernels not implemented in the paper.
Can full fine-tuning residuals be compressed the same way as LoRA modules?
Does ComPEFT work on fully fine-tuned models, not just PEFT?
Is sign-plus-scalar compression lossless for full fine-tuning?
ComPEFT also compresses full fine-tuning residuals, achieving 12x-19x compression on BERT, RoBERTa, T5-v1.1 and T5 with changes from +1.7 points (T5v1.1-Base) to -4.7 points (T5-Base) on 7 GLUE tasks.
Holds for: Base and Large sizes of 4 architectures, average test performance over 7 GLUE tasks; the largest drops are T5-Base (-4.7) and RoBERTa-Base (-2.2), so full-finetuning compression is not uniformly near-lossless.
Does compressing checkpoints before merging hurt model merging?
Do compressed task vectors merge better or worse with task arithmetic and TIES-Merging?
What happens if I merge ComPEFT checkpoints instead of the originals?
Merging ComPEFT-compressed checkpoints beats merging the uncompressed ones in 9 of 12 settings, and on T0-3B improves merged-model performance by 2.4% on average while being about 15x smaller; (IA)^3 on T5 models is the exception.
Holds for: Task Arithmetic and TIES-Merging applied to (IA)^3 and LoRA modules for 7 GLUE tasks on T5-Base, T5-Large and T0-3B; average test performance of the merged multitask model.
Do compressed LoRA experts still work for few-shot composition on unseen tasks?
Does LoraHub still work if the expert modules are compressed?
Is compositional generalization preserved after adapter compression?
ComPEFT-compressed LoRA experts retain compositional generalization under LoraHub, averaging 30.6 exact match across 27 Big-Bench-Hard tasks versus 30.5 for uncompressed experts.
Holds for: Flan-T5-Large with ~200 LoRA experts, N=20 modules composed per unseen task with the gradient-free Shiwa optimizer, averaged over 5 seeds; best-seed results favour uncompressed LoraHub (37.3 vs 36.4).
How does ComPEFT compare with BitDelta and DAREx for delta compression?
Is there a delta-compression method that beats STC without extra training?
What baselines did ComPEFT beat on LLaMA2-70B?
On LLaMA2-70B, ComPEFT averages 67.53% MMLU at 56 MB, above STC (65.24%, 56 MB), BitDelta without training (64.73%, 99 MB) and DAREx-q at 95% sparsity (64.68%, 395 MB), and matches BitDelta with trained scale (67.46%) which requires backward passes.
Holds for: Rank-64 QLoRA on LLaMA2-70B over 5 instruction-tuning datasets; storage uses Golomb coding for ComPEFT and STC, bitmask for BitDelta and COO sparse matrices for DAREx; DAREx at 99% sparsity collapses to 45.86%.
ComPEFT beats Sparse Ternary Compression and a prune-only ablation at nearly all density levels from 3B to 65B base models, and STC is far worse than the uncompressed model at 3B and 7B, showing the tuned scalar alpha is what recovers the performance lost to sparsification and ternarization.
Holds for: Average validation performance versus density k for LoRA modules on T0-3B and LLaMA 7B/13B/33B/65B; at base sizes of 13B and above all variants match or beat the original checkpoint at all densities tested.
Which part of ComPEFT matters most: sparsification, ternarization or the scaling factor?
Why does Sparse Ternary Compression fail on task vectors where ComPEFT works?
Does sparsifying and ternarizing a task vector need a tuned scaling factor?
ComPEFT beats Sparse Ternary Compression and a prune-only ablation at nearly all density levels from 3B to 65B base models, and STC is far worse than the uncompressed model at 3B and 7B, showing the tuned scalar alpha is what recovers the performance lost to sparsification and ternarization.
Holds for: Average validation performance versus density k for LoRA modules on T0-3B and LLaMA 7B/13B/33B/65B; at base sizes of 13B and above all variants match or beat the original checkpoint at all densities tested.
Do I have to tune the scaling hyperparameter alpha for ComPEFT?
What value of alpha should I use for a 65B model?
How sensitive is sparse ternary adapter compression to its scaling factor?
For base models with 13B or more parameters and density k of 20% or less, performance varies little with the scaling factor alpha, and the paper recommends simply setting alpha=1 rather than tuning it.
Holds for: Sweeps over k in {5,10,20,30,50} and alpha in {0.5,1,2,3,4,5,6,8,10} on T0-3B and LLaMA 7B-65B; at 3B and 7B the optimal alpha still matters and shifts with k (about 5-8 at k=5 versus 2-3 at k=50 for T0-3B).
Is ComPEFT competitive with other parameter-efficient fine-tuning methods on storage versus performance?
Which PEFT method gives the best accuracy per megabyte?
How does compressed (IA)^3 compare to BitFit, Adapters and Prompt Tuning?
ComPEFT applied to (IA)^3 and LoRA is Pareto-optimal in performance versus storage size against 10 PEFT methods including BitFit, Adapters, Compacter, Prompt Tuning, Prefix Tuning and Intrinsic SAID, with Com(IA)^3 matching methods that use 1000x more storage.
Holds for: T0-3B base model, 11 held-out datasets from the T0 evaluation suite, first PromptSource template, 200 training examples used as validation per task.
What should I read about the cost of serving many LoRA experts?
Which paper argues that expert adapter size is a communication bottleneck?
Where does work on compressing fine-tuning task vectors come from?
What is a good starting paper on compressing PEFT modules for multi-expert serving?
ComPEFT frames expert-adapter size as a communication and memory bottleneck for multi-expert serving rather than as a training-cost problem, noting that a QLoRA adapter for LLaMA-65B is 3.2 GB and must be swapped between disk, CPU and GPU per query.
Holds for: Framing as of the paper's 2023 arXiv release and 2025 TMLR version, aimed at PEFT-based expert serving, model merging and MoErging pipelines; the paper measures download and CPU-to-GPU transfer latency but does not build a full serving system.
ComPEFT adapts the sparsification-plus-ternary-quantization idea from federated-learning gradient compression (STC, TernGrad) to post-hoc compression of fine-tuning task vectors, removing the retraining that pruning methods usually need to recover accuracy.
Holds for: Compression of the residual between a released fine-tuned checkpoint and its initialization; assumes access to both, and to a small validation set for choosing the single scaling hyperparameter alpha.
How many bits per parameter does a ternary task vector need?
What is the theoretical storage limit for a 95%-sparse ternary adapter update?
How is the compressed ComPEFT checkpoint actually encoded on disk?
At 95% sparsity the ternary ComPEFT update has entropy of about 0.34 bits per parameter plus a 16-bit scalar, down from 16 bits per parameter for a bfloat16 task vector, a 47x reduction in communication and storage cost under a perfect coding scheme.
Holds for: Analytical entropy assuming signs of nonzero entries are uniformly distributed; realised sizes in the experiments use Golomb coding, and a two-binary-mask alternative costs 2 bits per parameter but is cheaper to compute with.

Claims and scope

Common misreadings

Terminology in this paper

task vector
The difference between a fine-tuned model's parameters and the initialization they were fine-tuned from, treated as the object to be stored and communicated.
density (k)
The fraction of task-vector entries whose signs are kept after sparsification; the remaining 1-k fraction is set to zero, so sparsity equals 1-k.
ternary quantization
Replacing every retained entry of a task vector by its sign in {-1, 0, +1} and one shared 16-bit scalar, so no per-parameter magnitudes are stored.
Com(IA)^3 / ComLoRA
The names for (IA)^3 and LoRA modules after ComPEFT compression, treated as PEFT methods in their own right on the storage-versus-performance frontier.
alpha
The single tuned hyperparameter of ComPEFT, a multiplier on the original task vector's standard deviation that sets the magnitude of every retained ternary entry.

How to cite

@article{
yadav2025compeft,
title={Com{PEFT}: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization},
author={Prateek Yadav and Leshem Choshen and Colin Raffel and Mohit Bansal},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2025},
url={https://openreview.net/forum?id=CovLQwu611},
note={}
}

References

See the full reference list in the paper.