Lossless and Near-Lossless Compression for Foundation Models

Moshik Hershcovitch; Leshem Choshen; Andrew Wood; Ilias Enmouri; Peter Chin; Swaminathan Sundararaman; Danny Harnik

doi:10.48550/ARXIV.2404.15198

Lossless and Near-Lossless Compression for Foundation Models

rearranging a model file so the first byte of every parameter is stored together, then the second byte, and so on, before running a general-purpose compressor

Moshik Hershcovitch, Leshem Choshen, Andrew Wood, Ilias Enmouri, Peter Chin, Swaminathan Sundararaman, Danny Harnik · arXiv · 2024

In one sentence

Standard lossless compressors, rearranged to group same-position bytes of every parameter together (byte grouping), cut popular Hugging Face models by 15-65% of their size, and a tunable lossy variant that trims sub-2^-b bits pushes savings further with no measurable accuracy loss.

Abstract

With the growth of model sizes and scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast literature about reducing model sizes, we investigate a more traditional type of compression -- one that compresses the model to a smaller form and is coupled with a decompression algorithm that returns it to its original size -- namely lossless compression. Somewhat surprisingly, we show that such lossless compression can gain significant network and storage reduction on popular models, at times reducing over $50\%$ of the model size. We investigate the source of model compressibility, introduce compression variants tailored for models and categorize models to compressibility groups. We also introduce a tunable lossy compression technique that can further reduce size even on the less compressible models with little to no effect on the model accuracy. We estimate that these methods could save over an ExaByte per month of network traffic downloaded from a large model hub like HuggingFace.

Questions this paper answers

how much smaller does a downloaded model file get if you just zip it?
what lossless compression ratios do FP32, FP16 and BF16 foundation model checkpoints reach with Zstd?
how do I shrink a Hugging Face checkpoint on disk without changing a single weight?
is it worth running zstd over my model weights before storing or shipping them?: Zstd with byte grouping compresses widely downloaded Hugging Face models to compression ratios of 35.7% for T5-base, 47.0% for RoBERTa, 50.1% for CLIP and 71% for Mistral. Decompression restores the weights exactly.
Holds for: Zstd at default level 3 with byte grouping, on FP32, FP16 and BF16 checkpoints downloaded from Hugging Face in August 2023 (Mistral and FP16 models in March 2024); compression run on CPU only.; Foundation models fall into 3 compressibility groups: FP32/FP16 models compressible only in the exponent byte, "clean" base models compressible in the exponent and both lower mantissa bytes, and BF16 models. The three groups reach compression ratios of ~80-85%, 35-50% and ~70% respectively.
Holds for: Categories drawn from highly downloaded Hugging Face models spanning several modalities, architectures and float formats; based on Zstd level 3 with byte grouping.
why does one model file squeeze down a lot while another barely shrinks at all?
which properties of a checkpoint's floating-point representation determine its entropy and therefore its compressibility?
how do I tell in advance whether a given checkpoint will compress well?
can I predict which of my checkpoints are worth compressing before I try?: Foundation models fall into 3 compressibility groups: FP32/FP16 models compressible only in the exponent byte, "clean" base models compressible in the exponent and both lower mantissa bytes, and BF16 models. The three groups reach compression ratios of ~80-85%, 35-50% and ~70% respectively.
Holds for: Categories drawn from highly downloaded Hugging Face models spanning several modalities, architectures and float formats; based on Zstd level 3 with byte grouping.; Fine-tuning destroys the compressibility of clean base models: RoBERTa drops from the 47.0% compression ratio of the base checkpoint to 80.7% after 1 epoch and 82.5% after 9 epochs of fine-tuning.
Holds for: RoBERTa FP32 fine-tuned on the Rotten Tomatoes dataset; the mechanism is that minuscule weight updates fill the previously zeroed lower mantissa bytes with entropy.
does reordering the bytes of the weights before zipping them make the file smaller?
how much does byte grouping of floating-point parameter bytes improve Zstd and LZ4 compression ratios on model weights?
how do I get a better compression ratio on model weights than a plain compressor gives?
should I add a byte-splitting step to my model storage pipeline, or is plain zstd enough?: Byte grouping before compression improves the compressed size of a model by 7-8.2% for exponent-only-compressible models, 19-27% for clean base models and 8.5-10% for BF16 models.
Holds for: Measured with Zstd; byte grouping requires knowing only the parameter type, not the model structure, so it works on a raw binary checkpoint.; Without byte grouping LZ4 achieves almost no compression on models, reaching only a 95% compression ratio on RoBERTa. With byte grouping the same compressor reaches 56%, still behind Zstd's 47.0%.
Holds for: RoBERTa FP32 checkpoint; LZ4 relies only on repetition removal, which fails on unstructured tensors, whereas Zstd and Zlib add entropy encoding.
does training a model further on my own data make its checkpoint harder to shrink?
how does fine-tuning affect the compressibility of a base checkpoint's mantissa bytes?
how do I keep my fine-tuned checkpoints as compressible as the base model they came from?
my fine-tuned checkpoints compress much worse than the base model I started from, is that expected?: Fine-tuning destroys the compressibility of clean base models: RoBERTa drops from the 47.0% compression ratio of the base checkpoint to 80.7% after 1 epoch and 82.5% after 9 epochs of fine-tuning.
Holds for: RoBERTa FP32 fine-tuned on the Rotten Tomatoes dataset; the mechanism is that minuscule weight updates fill the previously zeroed lower mantissa bytes with entropy.
how much precision can you throw away in model weights before the model gets worse?
at what precision factor does near-lossless compression start to degrade exact match, Rouge-L or sacreBLEU?
how do I pick a safe truncation level for lossy compression of my checkpoints?
can I use lossy weight compression on a model I serve without hurting its scores?: Tunable lossy compression at precision factor B=2^23 reduces the compressed size of FP32 models by a further 20%, taking wav2vec from a ~85% to a ~68% compression ratio. B=2^23 is the level at which FP32 arithmetic already introduces its own rounding error.
Holds for: FP32 models; B=2^23 is justified by FP32's own 2^-23 rounding error and by Adam epsilon defaults of 1e-8/1e-7, not by an accuracy sweep on wav2vec itself. Layers with parameters outside [-1,1] are left uncompressed.; Fine-tuned RoBERTa keeps accuracy near 90% down to a precision factor of B=2^6, with a slight rise just before the drop-off, while the compression ratio improves almost linearly to around 20%.
Holds for: RoBERTa fine-tuned on Rotten Tomatoes, single task and single fine-tuned model; on the clean base RoBERTa the same technique yields no benefit until B=2^18.; T5 variants fine-tuned on CNN-DM, XSUM, SQuAD, ASQA, WikiAnswers and WMT22 En-Ru show no significant change in exact match, Rouge-L or sacreBLEU at precision factors B=2^24 and B=2^19. Compression ratios there are 70% and 56%, versus 85% with lossless compression alone.
Holds for: Only 2 precision values were evaluated because of the cost of fine-tuning; T5-base FP32 checkpoints.
is it cheaper to store the difference between two training checkpoints than each one whole?
what compression ratios do deltas between consecutive fine-tuning epochs reach, lossless and at reduced precision?
how do I store a long series of training checkpoints without paying full size for each one?
should I keep every epoch's checkpoint as a delta instead of a full copy?: Compressing the delta between consecutive fine-tuning epochs of RoBERTa reaches a 55% lossless compression ratio, versus nearly 83% for the same checkpoint standalone. Taking the delta against the base model instead of the previous epoch gives 65%.
Holds for: RoBERTa fine-tuned on Rotten Tomatoes, lossless Zstd with byte grouping; delta against a base model avoids maintaining long chains of deltas but compresses less well.; Delta compression at precision factor B=2^23 reaches a 37% compression ratio between consecutive RoBERTa checkpoints and 49% against the base model, without affecting accuracy, and aggressive precision factors go below 10%.
Holds for: RoBERTa fine-tuned on Rotten Tomatoes with lossless Zstd and byte grouping underneath; accuracy checked on that one task only.
if several models were trained from the same starting point, can they be stored together more cheaply?
how well do pairwise deltas between sibling fine-tunes of one base checkpoint compress?
how do I cut storage for a family of task-specific models that all started from one base model?
I host several fine-tunes of the same base model, can I store them as differences from each other?: Three RoBERTa variants fine-tuned on tweets for irony, offensive-language and abuse detection compress to 85.7% standalone on average but to 56% on average when stored as pairwise deltas.
Holds for: cardiffnlp twitter-roberta-base irony/offensive/hate models, lossless Zstd with byte grouping; delta benefit deteriorates as models drift further apart in time.
does compressing a model file actually make downloading it faster, or does unpacking cancel it out?
what is the end-to-end effect of lossless and near-lossless weight compression on checkpoint transfer time at fixed network bandwidth?
how do I speed up pulling and pushing large checkpoints over a network?
on a fast connection, is compressing checkpoints worth the CPU time it costs me?: Lossy compression at precision factor B=2^23 cuts wav2vec download time by almost 20% on a 30 MBps network and upload time by 16% on a 20 MBps link. Lossless compression saves only 1% of upload time on the same model.
Holds for: wav2vec, the least compressible category, via torch.save/torch.load on a cloud VM in the Milan region; the lossy implementation omitted the sign-bit optimization. On cached 120-130 MBps reads the edge is slight.
is a quantized model already as small as it can possibly be?
do GPTQ and AWQ quantized checkpoints retain residual redundancy that lossless compression can still remove?
how do I shrink an already-quantized checkpoint further without touching its accuracy?
I already quantized my 7B model, is there anything left to gain from compressing the file?: Off-the-shelf GPTQ and AWQ quantized versions of CapybaraHermes-2.5-Mistral-7B still compress losslessly to ratios between 85% and 91%, with byte grouping contributing 1-2%.
Holds for: 8-bit and 4-bit GPTQ and AWQ quantizations of one Mistral-7B derivative, compressed with Zstd at default level 3.
how much bandwidth would a model-sharing site save if it compressed the files it serves?
what monthly traffic volume would lossless weight compression save across the most downloaded Hugging Face repositories?
how do I estimate the bandwidth a model hub would save by serving compressed weights?
would compressing downloads meaningfully cut my hosting bandwidth bill for model weights?: Lossless compression of the top downloaded Hugging Face models would save PetaBytes of traffic per model per month, including 11.7 PB for wav2vec and 26.1 PB for Bloom. Summed across the hub the estimate is over an ExaByte per month.
Holds for: Estimate from download counts as of August 2023 (Mistral March 2024) multiplied by measured compression ratios; assumes compression is applied at the hub and does not model cache hierarchies.; "Lossless and Near-Lossless Compression for Foundation Models" brings classical storage-and-network compression to model distribution, and argues it should be the default in communication with hubs such as Hugging Face. That direction is distinct from pruning, distillation and quantization, which shrink models irreversibly for inference speed.
Holds for: Framing as of 2024, with only one prior work found proposing compression, applied after two other model-compression steps. Covers stored and transferred model files, checkpoints, gradients and optimizer state, not inference-time speedups. The PyTorch save/load integration is intended for upstream contribution rather than merged.
what should I read about shrinking model files for storage and transfer rather than for faster inference?
which work applies classical lossless and near-lossless compression to model distribution, as opposed to pruning, distillation or quantization?
where do I start reading on compression for moving checkpoints around instead of compressing models for deployment?: "Lossless and Near-Lossless Compression for Foundation Models" brings classical storage-and-network compression to model distribution, and argues it should be the default in communication with hubs such as Hugging Face. That direction is distinct from pruning, distillation and quantization, which shrink models irreversibly for inference speed.
Holds for: Framing as of 2024, with only one prior work found proposing compression, applied after two other model-compression steps. Covers stored and transferred model files, checkpoints, gradients and optimizer state, not inference-time speedups. The PyTorch save/load integration is intended for upstream contribution rather than merged.

Claims and scope

Zstd with byte grouping compresses widely downloaded Hugging Face models to compression ratios of 35.7% for T5-base, 47.0% for RoBERTa, 50.1% for CLIP and 71% for Mistral. Decompression restores the weights exactly. (Table II)
Scope: Zstd at default level 3 with byte grouping, on FP32, FP16 and BF16 checkpoints downloaded from Hugging Face in August 2023 (Mistral and FP16 models in March 2024); compression run on CPU only.
Foundation models fall into 3 compressibility groups: FP32/FP16 models compressible only in the exponent byte, "clean" base models compressible in the exponent and both lower mantissa bytes, and BF16 models. The three groups reach compression ratios of ~80-85%, 35-50% and ~70% respectively. (Section III-B and Table II)
Scope: Categories drawn from highly downloaded Hugging Face models spanning several modalities, architectures and float formats; based on Zstd level 3 with byte grouping.
Byte grouping before compression improves the compressed size of a model by 7-8.2% for exponent-only-compressible models, 19-27% for clean base models and 8.5-10% for BF16 models. (Section III-B, "The benefit of Byte Grouping")
Scope: Measured with Zstd; byte grouping requires knowing only the parameter type, not the model structure, so it works on a raw binary checkpoint.
Without byte grouping LZ4 achieves almost no compression on models, reaching only a 95% compression ratio on RoBERTa. With byte grouping the same compressor reaches 56%, still behind Zstd's 47.0%. (Section III-A and Section III-B)
Scope: RoBERTa FP32 checkpoint; LZ4 relies only on repetition removal, which fails on unstructured tensors, whereas Zstd and Zlib add entropy encoding.
Fine-tuning destroys the compressibility of clean base models: RoBERTa drops from the 47.0% compression ratio of the base checkpoint to 80.7% after 1 epoch and 82.5% after 9 epochs of fine-tuning. (Table II)
Scope: RoBERTa FP32 fine-tuned on the Rotten Tomatoes dataset; the mechanism is that minuscule weight updates fill the previously zeroed lower mantissa bytes with entropy.
Tunable lossy compression at precision factor B=2^23 reduces the compressed size of FP32 models by a further 20%, taking wav2vec from a ~85% to a ~68% compression ratio. B=2^23 is the level at which FP32 arithmetic already introduces its own rounding error. (Section III-D)
Scope: FP32 models; B=2^23 is justified by FP32's own 2^-23 rounding error and by Adam epsilon defaults of 1e-8/1e-7, not by an accuracy sweep on wav2vec itself. Layers with parameters outside [-1,1] are left uncompressed.
Fine-tuned RoBERTa keeps accuracy near 90% down to a precision factor of B=2^6, with a slight rise just before the drop-off, while the compression ratio improves almost linearly to around 20%. (Figure 3)
Scope: RoBERTa fine-tuned on Rotten Tomatoes, single task and single fine-tuned model; on the clean base RoBERTa the same technique yields no benefit until B=2^18.
T5 variants fine-tuned on CNN-DM, XSUM, SQuAD, ASQA, WikiAnswers and WMT22 En-Ru show no significant change in exact match, Rouge-L or sacreBLEU at precision factors B=2^24 and B=2^19. Compression ratios there are 70% and 56%, versus 85% with lossless compression alone. (Section III-D)
Scope: Only 2 precision values were evaluated because of the cost of fine-tuning; T5-base FP32 checkpoints.
Compressing the delta between consecutive fine-tuning epochs of RoBERTa reaches a 55% lossless compression ratio, versus nearly 83% for the same checkpoint standalone. Taking the delta against the base model instead of the previous epoch gives 65%. (Figure 7 and Figure 8)
Scope: RoBERTa fine-tuned on Rotten Tomatoes, lossless Zstd with byte grouping; delta against a base model avoids maintaining long chains of deltas but compresses less well.
Delta compression at precision factor B=2^23 reaches a 37% compression ratio between consecutive RoBERTa checkpoints and 49% against the base model, without affecting accuracy, and aggressive precision factors go below 10%. (Figure 7 and Figure 8)
Scope: RoBERTa fine-tuned on Rotten Tomatoes with lossless Zstd and byte grouping underneath; accuracy checked on that one task only.
Three RoBERTa variants fine-tuned on tweets for irony, offensive-language and abuse detection compress to 85.7% standalone on average but to 56% on average when stored as pairwise deltas. (Section V-A)
Scope: cardiffnlp twitter-roberta-base irony/offensive/hate models, lossless Zstd with byte grouping; delta benefit deteriorates as models drift further apart in time.
Lossy compression at precision factor B=2^23 cuts wav2vec download time by almost 20% on a 30 MBps network and upload time by 16% on a 20 MBps link. Lossless compression saves only 1% of upload time on the same model. (Figure 5 and Figure 6)
Scope: wav2vec, the least compressible category, via torch.save/torch.load on a cloud VM in the Milan region; the lossy implementation omitted the sign-bit optimization. On cached 120-130 MBps reads the edge is slight.
Off-the-shelf GPTQ and AWQ quantized versions of CapybaraHermes-2.5-Mistral-7B still compress losslessly to ratios between 85% and 91%, with byte grouping contributing 1-2%. (Table III)
Scope: 8-bit and 4-bit GPTQ and AWQ quantizations of one Mistral-7B derivative, compressed with Zstd at default level 3.
Lossless compression of the top downloaded Hugging Face models would save PetaBytes of traffic per model per month, including 11.7 PB for wav2vec and 26.1 PB for Bloom. Summed across the hub the estimate is over an ExaByte per month. (Table I)
Scope: Estimate from download counts as of August 2023 (Mistral March 2024) multiplied by measured compression ratios; assumes compression is applied at the hub and does not model cache hierarchies.
"Lossless and Near-Lossless Compression for Foundation Models" brings classical storage-and-network compression to model distribution, and argues it should be the default in communication with hubs such as Hugging Face. That direction is distinct from pruning, distillation and quantization, which shrink models irreversibly for inference speed.
Scope: Framing as of 2024, with only one prior work found proposing compression, applied after two other model-compression steps. Covers stored and transferred model files, checkpoints, gradients and optimizer state, not inference-time speedups. The PyTorch save/load integration is intended for upstream contribution rather than merged.

Common misreadings

Lossless compression of model weights is not model compression in the usual sense: the decompressed file is bit-identical and the model runs at its original size and speed, so nothing about inference throughput improves.
A lower compression ratio is better, not worse: the ratio is the percentage of the original bytes that remain, so 47.0% means the file shrank by 53%.
The reported ratios are not a property of models in general. Clean base models reduce by 50-65% while fine-tuned FP32 and BF16 checkpoints only reduce by roughly 15-30%.
The slight accuracy rise observed just before the drop-off under aggressive precision trimming is reported as an unexpected phenomenon, not as a proposed way to improve models.
Tunable lossy compression is not quantization: it produces a file that decompresses back to the original format and precision layout, and it can trim to arbitrary bit levels such as 2^-23 rather than to runnable widths like 16 or 8 bits.
The claim that precision below 2^-23 can be discarded without accuracy loss was verified on RoBERTa/Rotten Tomatoes and on T5 fine-tunes at only 2 precision values, so it is a rule of thumb rather than a guarantee for arbitrary models.

Terminology in this paper

Compression ratio: The percentage of the original data that remains after compression, so lower is better: compressing 1 GB down to 0.25 GB is a compression ratio of 25%.
Byte grouping: A pre-compression transform for model files that groups together bytes from the same position across all parameters -- all first bytes, then all second bytes -- so that exponent bytes compress without interference from high-entropy mantissa bytes.
Tunable lossy compression: Multiplying each floating-point parameter by a precision factor B=2^b, rounding to an integer, then compressing losslessly; decompression divides back, discarding only quantities smaller than 2^-b.
Clean models: Base models whose two lower mantissa bytes are near-zero because training happened at lower precision, making them highly compressible; fine-tuning fills those bytes with entropy and destroys the compressibility.
Delta compression: Storing a base model plus the compressed difference (XOR or subtraction) between it and a similar model, rather than storing each similar model in full.

How to cite

@article{DBLP:journals/corr/abs-2404-15198,author       = {Moshik Hershcovitch and
                  Leshem Choshen and
                  Andrew Wood and
                  Ilias Enmouri and
                  Peter Chin and
                  Swaminathan Sundararaman and
                  Danny Harnik},
  title        = {Lossless and Near-Lossless Compression for Foundation Models},
  journal      = {CoRR},
  volume       = {abs/2404.15198},
  year         = {2024},
  url          = {https://doi.org/10.48550/arXiv.2404.15198},
  doi          = {10.48550/ARXIV.2404.15198},
  eprinttype    = {arXiv},
  eprint       = {2404.15198},
  timestamp    = {Sat, 25 May 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2404-15198.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.