Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

Rickard Brüel Gabrielsson; Jiacheng Zhu; Onkar Bhardwaj; Leshem Choshen; Kristjan Greenewald; Mikhail Yurochkin; Justin Solomon

doi:10.48550/arXiv.2407.00066

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

compressing many LoRA adapters into one shared basis plus a small per-adapter matrix, so thousands can be served from one GPU

Rickard Brüel Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, Justin Solomon · ICML 2025 · 2025

In one sentence

Compress then Serve jointly compresses a collection of LoRA adapters into a shared basis U, V plus small LoRA-specific matrices, found by joint diagonalization and clustering, so that thousands of adapters fit in GPU memory and can be served with little throughput loss.

Abstract

Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates. This paradigm presents challenges for systems that serve real-time responses to queries that each involve a different LoRA. Prior works optimize the design of such systems but still require continuous loading and offloading of LoRAs, as it is infeasible to store thousands of LoRAs in GPU memory. To mitigate this issue, we investigate the efficacy of compression when serving LoRAs. We propose a method for the joint compression of LoRAs into a shared basis paired with LoRA-specific scaling matrices. We extend our algorithm to learn clusters of LoRAs that are amenable to joint compression, allowing it to scale gracefully to large LoRA collections. Our experiments with up to 1000 LoRAs demonstrate that compressed LoRAs preserve performance while offering major throughput gains in realistic serving scenarios with over a thousand LoRAs, maintaining 80% of the throughput of serving a single LoRA.

Questions this paper answers

How can a server host thousands of fine-tuned adapters for one base model without running out of GPU memory?
What throughput can I expect when serving 1000+ LoRA adapters at once?
Does compressing LoRA adapters actually make multi-adapter serving faster?: Serving over 1000 jointly compressed LoRAs with vLLM increases throughput 1.6x over vLLM multi-LoRA at a matched GPU memory footprint. The compressed collection retains 80% of the throughput of serving the base LLM or a single merged LoRA.
Holds for: Mistral-7B-Instruct-v0.2, rank-16 LoRAs, 25 clusters at compression rank 16, asynchronous requests generating 10 tokens each on a Shakespeare-sonnet input set, one H100 80GB GPU capped at 40% memory.; Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+.
Holds for: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
Does compressing LoRA adapters hurt task accuracy?
How much performance is lost when many LoRAs share one basis?
Can lossy compression of adapters ever improve results?: Minimizing reconstruction error does not maximize downstream performance for compressed LoRAs: moderate reconstruction error around 60% matches or slightly exceeds the zero-error setting on Rouge-L. At equal reconstruction error, clustering beats non-clustered joint diagonalization.
Holds for: Rouge-L relative to uncompressed LoRA on 10 in-distribution natural-instruction tasks, Mistral-7B-Instruct, rank-16 LoRAs, collections of 10 to 1000; at large reconstruction error performance falls off sharply.; LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation.
Holds for: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.
How do I choose the compression rank and number of clusters for a large adapter collection?
Can LoRA compression hyperparameters be tuned without expensive LLM evaluation?
What reconstruction error is safe when compressing LoRAs?: LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation.
Holds for: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.; Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance.
Holds for: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.
When is clustering adapters necessary instead of a single shared basis?
Is JD-Full or JD-Diag the better choice for compressing LoRAs?
Does the shared-basis approach scale from tens to a thousand adapters?: Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance.
Holds for: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.; Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+.
Holds for: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
Is there a theoretical limit on how well many low-rank adapters can share one basis?
What guarantee bounds reconstruction error for joint diagonalization of LoRAs?
When is joint compression of adapters provably lossy?: For n LoRAs stacked as columns of a matrix L, JD-Full's reconstructed Frobenius energy is bounded above by the sum of L's top min(r^2, n) squared singular values. Reconstruction error is thus unavoidable unless L's spectrum concentrates in the top r^2 directions.
Holds for: JD-Full with orthogonal U, V of r columns; the bound is on Frobenius reconstruction error, not on downstream LLM accuracy, and the proof notes the Von Neumann upper bound is generous because vec(BA) has Kronecker structure.; When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters.
Holds for: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
Do fine-tuned LoRA adapters share structure with each other?
Is joint compression of LoRAs exploiting real shared structure or just low-rank noise?
How does reconstruction error on trained adapters compare to random ones?: Reconstruction error of joint diagonalization is consistently higher on random untrained LoRA matrices than on trained ones, indicating that training gives LoRAs a shared component that joint diagonalization exploits.
Holds for: JD-Full at ranks 16, 32 and 64 on collections of 10, 50, 100 and 500 rank-16 Mistral-7B-Instruct LoRAs, compared against the trained-LoRA errors in Table 14.; When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters.
Holds for: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
Does joint LoRA compression still work when the adapter does not match the evaluation task?
How do compressed adapters do under the LoRA-hub out-of-distribution protocol?
What happens to accuracy on unseen tasks after compressing 100 adapters?: Under the LoRA-hub out-of-distribution protocol with 100 sampled adapters, JD-Full at rank 64 averages 47.66 versus 48.32 for uncompressed LoRAs and 32.28 for the base model. JD-Diag at rank 64 averages 47.43.
Holds for: 100 LoRAs sampled independently of the evaluation task, averaged over 10 BIG-Bench-style tasks; these runs were done without the Frobenius normalization of adapters that the paper later found beneficial.
Where can I get a large public collection of LoRA adapters for research?
Are the 1000 adapters used in the joint-compression experiments any good?
How were 1000 task-specific adapters trained for Mistral-7B-Instruct?: Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections.
Holds for: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.; The 1000 released LoRAs for Mistral-7B-Instruct-v0.2 raise mean Rouge-L from 20.62 to 67.80 over the base model. Mean exact match rises from 1.81 to 51.38 and mean test loss falls from 4.14 to 0.56.
Holds for: Rank-16 LoRAs on q_proj, k_proj and v_proj of a 4-bit quantized base model, one adapter per task across 1000 English natural-instruction tasks, early stopping on validation loss.
Could compressing several users' adapters together leak information between them?
Is joint compression of per-user adapters private?
Does an adapter gain ability on another adapter's task after joint compression?: Compressing an adapter for one task jointly with an adapter for a second task gave no performance gain on the second task. This is preliminary evidence that joint compression does not leak task information between adapters.
Holds for: A single ablation on pairs of natural-instruction tasks with Mistral-7B-Instruct; a negative result at this scale, not a privacy guarantee, and the paper flags a fuller privacy study as future work.
What should I read about serving many LoRA adapters efficiently?
Which paper treats multi-adapter serving as a compression problem instead of a systems problem?
Where should I start reading about the memory bottleneck in multi-LoRA inference?: Compress then Serve frames multi-LoRA serving as a compression problem rather than only a systems problem. Scheduling and memory-management optimizations such as S-LoRA and vLLM multi-LoRA still degrade when thousands of adapters must be swapped in and out of GPU memory.
Holds for: Positioning as of the 2025 ICML publication, for LLM inference servers holding one base model plus many per-user LoRAs; complementary to rather than a replacement for kernel-level work, and the paper's own experiments use vLLM with the Punica kernel.; Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections.
Holds for: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.

Claims and scope

Serving over 1000 jointly compressed LoRAs with vLLM increases throughput 1.6x over vLLM multi-LoRA at a matched GPU memory footprint. The compressed collection retains 80% of the throughput of serving the base LLM or a single merged LoRA. (Figure 1)
Scope: Mistral-7B-Instruct-v0.2, rank-16 LoRAs, 25 clusters at compression rank 16, asynchronous requests generating 10 tokens each on a Shakespeare-sonnet input set, one H100 80GB GPU capped at 40% memory.
Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+. (Figure 4)
Scope: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation. (Section 6.5)
Scope: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.
Minimizing reconstruction error does not maximize downstream performance for compressed LoRAs: moderate reconstruction error around 60% matches or slightly exceeds the zero-error setting on Rouge-L. At equal reconstruction error, clustering beats non-clustered joint diagonalization. (Figure 3)
Scope: Rouge-L relative to uncompressed LoRA on 10 in-distribution natural-instruction tasks, Mistral-7B-Instruct, rank-16 LoRAs, collections of 10 to 1000; at large reconstruction error performance falls off sharply.
Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance. (Section 6.5)
Scope: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.
For n LoRAs stacked as columns of a matrix L, JD-Full's reconstructed Frobenius energy is bounded above by the sum of L's top min(r^2, n) squared singular values. Reconstruction error is thus unavoidable unless L's spectrum concentrates in the top r^2 directions. (Theorem 1, Section 4)
Scope: JD-Full with orthogonal U, V of r columns; the bound is on Frobenius reconstruction error, not on downstream LLM accuracy, and the proof notes the Von Neumann upper bound is generous because vec(BA) has Kronecker structure.
When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters. (Corollary 1, Section 4)
Scope: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
Reconstruction error of joint diagonalization is consistently higher on random untrained LoRA matrices than on trained ones, indicating that training gives LoRAs a shared component that joint diagonalization exploits. (Table 15, Appendix H.11)
Scope: JD-Full at ranks 16, 32 and 64 on collections of 10, 50, 100 and 500 rank-16 Mistral-7B-Instruct LoRAs, compared against the trained-LoRA errors in Table 14.
Under the LoRA-hub out-of-distribution protocol with 100 sampled adapters, JD-Full at rank 64 averages 47.66 versus 48.32 for uncompressed LoRAs and 32.28 for the base model. JD-Diag at rank 64 averages 47.43. (Table 18)
Scope: 100 LoRAs sampled independently of the evaluation task, averaged over 10 BIG-Bench-style tasks; these runs were done without the Frobenius normalization of adapters that the paper later found beneficial.
The 1000 released LoRAs for Mistral-7B-Instruct-v0.2 raise mean Rouge-L from 20.62 to 67.80 over the base model. Mean exact match rises from 1.81 to 51.38 and mean test loss falls from 4.14 to 0.56. (Table 1)
Scope: Rank-16 LoRAs on q_proj, k_proj and v_proj of a 4-bit quantized base model, one adapter per task across 1000 English natural-instruction tasks, early stopping on validation loss.
Compressing an adapter for one task jointly with an adapter for a second task gave no performance gain on the second task. This is preliminary evidence that joint compression does not leak task information between adapters. (Appendix H.2)
Scope: A single ablation on pairs of natural-instruction tasks with Mistral-7B-Instruct; a negative result at this scale, not a privacy guarantee, and the paper flags a fuller privacy study as future work.
Compress then Serve frames multi-LoRA serving as a compression problem rather than only a systems problem. Scheduling and memory-management optimizations such as S-LoRA and vLLM multi-LoRA still degrade when thousands of adapters must be swapped in and out of GPU memory. (Section 2)
Scope: Positioning as of the 2025 ICML publication, for LLM inference servers holding one base model plus many per-user LoRAs; complementary to rather than a replacement for kernel-level work, and the paper's own experiments use vLLM with the Punica kernel.
Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections. (Section 1, Table 3)
Scope: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.

Common misreadings

Joint diagonalization is not model merging: it shares only the subspaces U and V across adapters while each adapter keeps its own Sigma, so the result is a set of per-task models rather than one general model.
Reducing the parameter count of LoRA adapters does not by itself speed up inference; compression reduces memory load and CPU-to-GPU transfer time but leaves forward-pass latency unchanged, and the throughput gains come from fitting more adapters on the GPU.
Lower reconstruction error is not the goal in itself: settings with near-zero Frobenius error do not give the best Rouge-L, and moderately lossy reconstruction can match or slightly beat them.
The 1.6x throughput gain is not universal across collection sizes; an aggressive setting such as 25 clusters at rank 15 underperforms the vLLM multi-LoRA baseline when only 32 or fewer adapters are served.
The privacy ablation showing no gain on a co-compressed adapter's task is a single preliminary experiment, not a proof that joint compression is leakage-free.

Terminology in this paper

JD-Full: Joint diagonalization of a collection of LoRA products BA into a shared orthogonal basis U, V of r columns plus an unconstrained r-by-r matrix per adapter.
JD-Diag: Joint diagonalization of a collection of LoRA products BA into a shared basis U, V plus a diagonal r-parameter scaling per adapter, cheaper per adapter than a full r-by-r matrix.
Agreement: The fraction of generations on which a compressed adapter's output exactly matches the uncompressed adapter's output, compared model-to-model rather than against ground truth.
Performance relative to LoRA: A method's task metric divided by the uncompressed LoRA's metric on the same task, so 1.0 means parity with the original adapter.
Total Parameter Saved Ratio: One minus the number of parameters after compression divided by the number before, computed for a system serving a large number of distinct adapters.

How to cite

@inproceedings{
gabrielsson2025compress,
title={Compress then Serve: Serving Thousands of Lo{RA} Adapters with Little Overhead},
author={Rickard Br{\"u}el Gabrielsson and Jiacheng Zhu and Onkar Bhardwaj and Leshem Choshen and Kristjan Greenewald and Mikhail Yurochkin and Justin Solomon},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=3XMA8RDJu2}
}

References

See the full reference list in the paper.