# Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

compressing many LoRA adapters into one shared basis plus a small per-adapter matrix, so thousands can be served from one GPU

Authors: Rickard Brüel Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, Justin Solomon
Venue: Forty-second International Conference on Machine Learning (2025)

## What this paper shows
Compress then Serve jointly compresses a collection of LoRA adapters into a shared basis U, V plus small LoRA-specific matrices, found by joint diagonalization and clustering, so that thousands of adapters fit in GPU memory and can be served with little throughput loss.

## Claims, with scope
- Serving over 1000 jointly compressed LoRAs with vLLM increases throughput 1.6x over vLLM multi-LoRA at a matched GPU memory footprint. The compressed collection retains 80% of the throughput of serving the base LLM or a single merged LoRA.
  Scope: Mistral-7B-Instruct-v0.2, rank-16 LoRAs, 25 clusters at compression rank 16, asynchronous requests generating 10 tokens each on a Shakespeare-sonnet input set, one H100 80GB GPU capped at 40% memory.
  Evidence: Figure 1
- Compression settings for LoRA serving must be matched to collection size. Rank-16 JD-Full improves vLLM multi-LoRA throughput at 4 and 8 LoRAs but not beyond, while 25 clusters at rank 15 helps only from well above 32 LoRAs, with large gains at 1000+.
  Scope: vLLM multi-LoRA baseline given the same GPU memory footprint (max-gpu-lora set per collection size), Mistral-7B-Instruct, rank-16 LoRAs, collections between 4 and 1024.
  Evidence: Figure 4
- LoRA compression settings whose mean relative reconstruction error stays below 0.6 reliably preserve 99% or more of the uncompressed LoRAs' Rouge-L performance. Compression rank and cluster count can therefore be tuned on CPU without any LLM evaluation.
  Scope: Frobenius reconstruction error of BA measured on one LoRA module from the middle of the network; 10 in-distribution natural-instruction tasks with Mistral-7B-Instruct; adapters normalized to unit Frobenius norm before compression.
  Evidence: Section 6.5
- Minimizing reconstruction error does not maximize downstream performance for compressed LoRAs: moderate reconstruction error around 60% matches or slightly exceeds the zero-error setting on Rouge-L. At equal reconstruction error, clustering beats non-clustered joint diagonalization.
  Scope: Rouge-L relative to uncompressed LoRA on 10 in-distribution natural-instruction tasks, Mistral-7B-Instruct, rank-16 LoRAs, collections of 10 to 1000; at large reconstruction error performance falls off sharply.
  Evidence: Figure 3
- Joint diagonalization of LoRAs alone suffices up to about 100 adapters, using a rank of roughly (number of LoRAs / 2) + 7. Clustering becomes essential at 500-1000 LoRAs, where JD-Full with clustering preserves performance.
  Scope: Mistral-7B-Instruct with rank-16 LoRAs on natural-instruction tasks; JD-Full is preferred over JD-Diag, though below 100 LoRAs the difference between them is negligible.
  Evidence: Section 6.5
- For n LoRAs stacked as columns of a matrix L, JD-Full's reconstructed Frobenius energy is bounded above by the sum of L's top min(r^2, n) squared singular values. Reconstruction error is thus unavoidable unless L's spectrum concentrates in the top r^2 directions.
  Scope: JD-Full with orthogonal U, V of r columns; the bound is on Frobenius reconstruction error, not on downstream LLM accuracy, and the proof notes the Von Neumann upper bound is generous because vec(BA) has Kronecker structure.
  Evidence: Theorem 1, Section 4
- When LoRAs are mutually orthogonal and normalized to unit Frobenius norm, JD-Full's relative reconstruction error is at least 1 - min(r^2/n, 1). For r^2 much smaller than n the reconstruction therefore retains little of the original adapters.
  Scope: Idealized case of exactly orthogonal, unit-norm LoRAs; real trained LoRAs share structure and do considerably better, and clustering with k growing in n can keep error bounded at fixed r.
  Evidence: Corollary 1, Section 4
- Reconstruction error of joint diagonalization is consistently higher on random untrained LoRA matrices than on trained ones, indicating that training gives LoRAs a shared component that joint diagonalization exploits.
  Scope: JD-Full at ranks 16, 32 and 64 on collections of 10, 50, 100 and 500 rank-16 Mistral-7B-Instruct LoRAs, compared against the trained-LoRA errors in Table 14.
  Evidence: Table 15, Appendix H.11
- Under the LoRA-hub out-of-distribution protocol with 100 sampled adapters, JD-Full at rank 64 averages 47.66 versus 48.32 for uncompressed LoRAs and 32.28 for the base model. JD-Diag at rank 64 averages 47.43.
  Scope: 100 LoRAs sampled independently of the evaluation task, averaged over 10 BIG-Bench-style tasks; these runs were done without the Frobenius normalization of adapters that the paper later found beneficial.
  Evidence: Table 18
- The 1000 released LoRAs for Mistral-7B-Instruct-v0.2 raise mean Rouge-L from 20.62 to 67.80 over the base model. Mean exact match rises from 1.81 to 51.38 and mean test loss falls from 4.14 to 0.56.
  Scope: Rank-16 LoRAs on q_proj, k_proj and v_proj of a 4-bit quantized base model, one adapter per task across 1000 English natural-instruction tasks, early stopping on validation loss.
  Evidence: Table 1
- Compressing an adapter for one task jointly with an adapter for a second task gave no performance gain on the second task. This is preliminary evidence that joint compression does not leak task information between adapters.
  Scope: A single ablation on pairs of natural-instruction tasks with Mistral-7B-Instruct; a negative result at this scale, not a privacy guarantee, and the paper flags a fuller privacy study as future work.
  Evidence: Appendix H.2
- Compress then Serve frames multi-LoRA serving as a compression problem rather than only a systems problem. Scheduling and memory-management optimizations such as S-LoRA and vLLM multi-LoRA still degrade when thousands of adapters must be swapped in and out of GPU memory.
  Scope: Positioning as of the 2025 ICML publication, for LLM inference servers holding one base model plus many per-user LoRAs; complementary to rather than a replacement for kernel-level work, and the paper's own experiments use vLLM with the Punica kernel.
  Evidence: Section 2
- Compress then Serve releases a collection of over 1000 LoRA adapters trained on 1000 natural-instruction tasks for Mistral-7B-Instruct-v0.2. The collection is intended as a testbed for work on serving, merging and compressing large adapter collections.
  Scope: All tasks are English-language natural instructions with input and output in English; all adapters are rank 16 on q_proj, k_proj and v_proj of one base model, so the collection does not cover varied ranks, architectures or languages.
  Evidence: Section 1, Table 3

## Common misreadings
- Joint diagonalization is not model merging: it shares only the subspaces U and V across adapters while each adapter keeps its own Sigma, so the result is a set of per-task models rather than one general model.
- Reducing the parameter count of LoRA adapters does not by itself speed up inference; compression reduces memory load and CPU-to-GPU transfer time but leaves forward-pass latency unchanged, and the throughput gains come from fitting more adapters on the GPU.
- Lower reconstruction error is not the goal in itself: settings with near-zero Frobenius error do not give the best Rouge-L, and moderately lossy reconstruction can match or slightly beat them.
- The 1.6x throughput gain is not universal across collection sizes; an aggressive setting such as 25 clusters at rank 15 underperforms the vLLM multi-LoRA baseline when only 32 or fewer adapters are served.
- The privacy ablation showing no gain on a co-compressed adapter's task is a single preliminary experiment, not a proof that joint compression is leakage-free.

## Terminology
- JD-Full: Joint diagonalization of a collection of LoRA products BA into a shared orthogonal basis U, V of r columns plus an unconstrained r-by-r matrix per adapter.
- JD-Diag: Joint diagonalization of a collection of LoRA products BA into a shared basis U, V plus a diagonal r-parameter scaling per adapter, cheaper per adapter than a full r-by-r matrix.
- Agreement: The fraction of generations on which a compressed adapter's output exactly matches the uncompressed adapter's output, compared model-to-model rather than against ground truth.
- Performance relative to LoRA: A method's task metric divided by the uncompressed LoRA's metric on the same task, so 1.0 means parity with the original adapter.
- Total Parameter Saved Ratio: One minus the number of parameters after compression divided by the number before, computed for a system serving a large number of distinct adapters.

## Links
- arXiv: https://arxiv.org/abs/2407.00066
- PDF: https://arxiv.org/pdf/2407.00066
- HTML: https://arxiv.org/html/2407.00066
- Hugging Face: https://huggingface.co/papers/2407.00066
- alphaXiv: https://www.alphaxiv.org/abs/2407.00066
- DOI: https://doi.org/10.48550/arXiv.2407.00066
- Semantic Scholar: https://www.semanticscholar.org/paper/270870510
- Publisher: https://openreview.net/forum?id=3XMA8RDJu2

## How to cite
@inproceedings{
gabrielsson2025compress,
title={Compress then Serve: Serving Thousands of Lo{RA} Adapters with Little Overhead},
author={Rickard Br{\"u}el Gabrielsson and Jiacheng Zhu and Onkar Bhardwaj and Leshem Choshen and Kristjan Greenewald and Mikhail Yurochkin and Justin Solomon},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=3XMA8RDJu2}
}