Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

compressing many LoRA adapters into one shared basis plus a small per-adapter matrix, so thousands can be served from one GPU

Rickard Brüel Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, Justin Solomon · ICML 2025 · 2025

In one sentence

Compress then Serve jointly compresses a collection of LoRA adapters into a shared basis U, V plus small LoRA-specific matrices, found by joint diagonalization and clustering, so that thousands of adapters fit in GPU memory and can be served with little throughput loss.

Abstract

Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates. This paradigm presents challenges for systems that serve real-time responses to queries that each involve a different LoRA. Prior works optimize the design of such systems but still require continuous loading and offloading of LoRAs, as it is infeasible to store thousands of LoRAs in GPU memory. To mitigate this issue, we investigate the efficacy of compression when serving LoRAs. We propose a method for the joint compression of LoRAs into a shared basis paired with LoRA-specific scaling matrices. We extend our algorithm to learn clusters of LoRAs that are amenable to joint compression, allowing it to scale gracefully to large LoRA collections. Our experiments with up to 1000 LoRAs demonstrate that compressed LoRAs preserve performance while offering major throughput gains in realistic serving scenarios with over a thousand LoRAs, maintaining 80% of the throughput of serving a single LoRA.

Questions this paper answers

how can one GPU server answer requests for thousands of different fine-tuned versions of the same model at once?
does joint compression of LoRA adapters raise multi-LoRA serving throughput at a fixed GPU memory budget?
how do I keep serving throughput up when my inference server has to swap between 1000 LoRA adapters?
I host hundreds of per-customer adapters on one base model, should I compress them together before serving?
Serving over 1000 jointly compressed LoRAs with vLLM increases throughput 1.6x over vLLM multi-LoRA at a matched GPU memory footprint. The compressed collection retains 80% of the throughput of serving the base LLM or a single merged LoRA.
Holds for: Mistral-7B-Instruct-v0.2, rank-16 LoRAs, 25 clusters at compression rank 16, asynchronous requests generating 10 tokens each on a Shakespeare-sonnet input set, one H100 80GB GPU capped at 40% memory.
Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+.
Holds for: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
if many fine-tuned adapters are squeezed into a shared basis, how much task quality is given up?
how does reconstruction error of jointly diagonalized LoRAs translate into Rouge-L on the adapters' own tasks?
how do I compress a large adapter collection without losing accuracy on the tasks each adapter was trained for?
is lossy compression of my LoRA adapters safe, or will my users notice worse answers?
Minimizing reconstruction error does not maximize downstream performance for compressed LoRAs: moderate reconstruction error around 60% matches or slightly exceeds the zero-error setting on Rouge-L. At equal reconstruction error, clustering beats non-clustered joint diagonalization.
Holds for: Rouge-L relative to uncompressed LoRA on 10 in-distribution natural-instruction tasks, Mistral-7B-Instruct, rank-16 LoRAs, collections of 10 to 1000; at large reconstruction error performance falls off sharply.
LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation.
Holds for: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.
can the settings for squashing many adapters together be picked without running the big model at all?
can compression rank and cluster count be selected from mean relative reconstruction error alone, with no downstream LLM eval?
how do I choose a compression rank and number of clusters for my adapter collection cheaply?
do I have to benchmark every compression configuration on my tasks, or is a CPU-side error metric enough to decide?
LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation.
Holds for: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.
Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance.
Holds for: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.
at what collection size does grouping similar adapters start to matter instead of using one shared basis for all of them?
when does clustering before joint diagonalization become necessary as the number of LoRAs scales from tens to 1000?
I am compressing anywhere from 8 to 1000 adapters, how do I decide between one shared basis and clustered bases?
with only a few dozen adapters to serve, is clustering them worth the extra complexity?
Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance.
Holds for: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.
Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+.
Holds for: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
is there a mathematical limit on how much can be recovered when many small adapter updates share one basis?
what upper bound on reconstructed Frobenius energy applies to joint diagonalization of n LoRA update matrices at rank r?
how do I tell in advance whether a set of adapters can be jointly compressed without large reconstruction error?
if my adapters were trained on very unrelated tasks, should I expect joint compression to be provably lossy?
For n LoRAs stacked as columns of a matrix L, JD-Full's reconstructed Frobenius energy is bounded above by the sum of L's top min(r^2, n) squared singular values. Reconstruction error is thus unavoidable unless L's spectrum concentrates in the top r^2 directions.
Holds for: JD-Full with orthogonal U, V of r columns; the bound is on Frobenius reconstruction error, not on downstream LLM accuracy, and the proof notes the Von Neumann upper bound is generous because vec(BA) has Kronecker structure.
When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters.
Holds for: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
do separately fine-tuned adapters for different tasks end up sharing structure with each other?
is the compressibility of trained LoRA updates due to a shared subspace, or would random low-rank matrices compress equally well?
how do I check whether the adapters I want to compress together actually share directions rather than just being low rank?
does joint compression only pay off because my adapters were trained, or would it work on any low-rank matrices?
Reconstruction error of joint diagonalization is consistently higher on random untrained LoRA matrices than on trained ones, indicating that training gives LoRAs a shared component that joint diagonalization exploits.
Holds for: JD-Full at ranks 16, 32 and 64 on collections of 10, 50, 100 and 500 rank-16 Mistral-7B-Instruct LoRAs, compared against the trained-LoRA errors in Table 14.
When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters.
Holds for: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
after adapters are compressed together, do they still help on tasks none of them was trained for?
how do jointly diagonalized LoRAs perform under the LoRA-hub out-of-distribution evaluation protocol against uncompressed adapters and the base model?
how do I keep out-of-domain generalization when I compress a pool of 100 adapters into a shared basis?
my users send prompts that do not match any single adapter, will compressed adapters still beat the plain base model?
Under the LoRA-hub out-of-distribution protocol with 100 sampled adapters, JD-Full at rank 64 averages 47.66 versus 48.32 for uncompressed LoRAs and 32.28 for the base model. JD-Diag at rank 64 averages 47.43.
Holds for: 100 LoRAs sampled independently of the evaluation task, averaged over 10 BIG-Bench-style tasks; these runs were done without the Frobenius normalization of adapters that the paper later found beneficial.
how good are the 1000 released task-specific adapters for Mistral-7B compared with the plain model?
what Rouge-L, exact match and test loss do the 1000 released natural-instructions LoRAs reach on Mistral-7B-Instruct-v0.2?
where do I get a large collection of trained LoRA adapters to test my own merging or serving method on?
can I use the released 1000-adapter Mistral collection as a benchmark, or are the adapters too weak to be meaningful?
Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections.
Holds for: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.
The 1000 released LoRAs for Mistral-7B-Instruct-v0.2 raise mean Rouge-L from 20.62 to 67.80 over the base model. Mean exact match rises from 1.81 to 51.38 and mean test loss falls from 4.14 to 0.56.
Holds for: Rank-16 LoRAs on q_proj, k_proj and v_proj of a 4-bit quantized base model, one adapter per task across 1000 English natural-instruction tasks, early stopping on validation loss.
if one customer's fine-tuned adapter is compressed together with another's, can ability or information cross between them?
does joint compression of LoRAs cause cross-task information leakage between the compressed adapters?
how do I check that compressing per-user adapters into a shared basis does not transfer one user's task ability to another?
I serve per-tenant LoRAs, is it safe to compress them jointly from a data-isolation standpoint?
Compressing an adapter for one task jointly with an adapter for a second task gave no performance gain on the second task. This is preliminary evidence that joint compression does not leak task information between adapters.
Holds for: A single ablation on pairs of natural-instruction tasks with Mistral-7B-Instruct; a negative result at this scale, not a privacy guarantee, and the paper flags a fuller privacy study as future work.
which work treats hosting thousands of fine-tuned adapters as a compression problem rather than a scheduling one?
what should I read on the GPU memory bottleneck in multi-LoRA inference beyond serving-system optimizations?
where do I start reading if I need to serve far more LoRA adapters than fit in GPU memory?
is there a paper and adapter collection I can build on for large-scale multi-LoRA serving research?
Compress then Serve frames multi-LoRA serving as a compression problem rather than only a systems problem. Scheduling and memory-management optimizations such as S-LoRA and vLLM multi-LoRA still degrade when thousands of adapters must be swapped in and out of GPU memory.
Holds for: Positioning as of the 2025 ICML publication, for LLM inference servers holding one base model plus many per-user LoRAs; complementary to rather than a replacement for kernel-level work, and the paper's own experiments use vLLM with the Punica kernel.
Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections.
Holds for: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.

Claims and scope

Common misreadings

Terminology in this paper

JD-Full
Joint diagonalization of a collection of LoRA products BA into a shared orthogonal basis U, V of r columns plus an unconstrained r-by-r matrix per adapter.
JD-Diag
Joint diagonalization of a collection of LoRA products BA into a shared basis U, V plus a diagonal r-parameter scaling per adapter, cheaper per adapter than a full r-by-r matrix.
Agreement
The fraction of generations on which a compressed adapter's output exactly matches the uncompressed adapter's output, compared model-to-model rather than against ground truth.
Performance relative to LoRA
A method's task metric divided by the uncompressed LoRA's metric on the same task, so 1.0 means parity with the original adapter.
Total Parameter Saved Ratio
One minus the number of parameters after compression divided by the number before, computed for a system serving a large number of distinct adapters.

How to cite

@inproceedings{
gabrielsson2025compress,
title={Compress then Serve: Serving Thousands of Lo{RA} Adapters with Little Overhead},
author={Rickard Br{\"u}el Gabrielsson and Jiacheng Zhu and Onkar Bhardwaj and Leshem Choshen and Kristjan Greenewald and Mikhail Yurochkin and Justin Solomon},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=3XMA8RDJu2}
}

References

See the full reference list in the paper.