# Asymmetry in Low-Rank Adapters of Foundation Models

Authors: Jiacheng Zhu, Kristjan H. Greenewald, Kimia Nadjahi, Haitz S\'aez de Oc\'ariz Borde, Rickard Br"uel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, Justin Solomon
Venue: Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024 (2024)

## Abstract
Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective. Inspired by an effort to investigate the different roles of LoRA matrices during fine-tuning, this paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. Specifically, when updating the parameter matrices of a neural network by adding a product $BA$, we observe that the $B$ and $A$ matrices have distinct functions: $A$ extracts features from the input, while $B$ uses these features to create the desired output. Based on this observation, we demonstrate that fine-tuning $B$ is inherently more effective than fine-tuning $A$, and that a random untrained $A$ should perform nearly as well as a fine-tuned one. Using an information-theoretic lens, we also bound the generalization of low-rank adapters, showing that the parameter savings of exclusively training $B$ improves the bound. We support our conclusions with experiments on RoBERTa, BART-Large, LLaMA-2, and ViTs.

## Links
- arXiv: https://arxiv.org/abs/2402.16842
- PDF: https://arxiv.org/pdf/2402.16842
- HTML: https://arxiv.org/html/2402.16842
- Hugging Face: https://huggingface.co/papers/2402.16842
- alphaXiv: https://www.alphaxiv.org/abs/2402.16842
- DOI: https://doi.org/10.48550/ARXIV.2402.16842
- Semantic Scholar: https://www.semanticscholar.org/paper/268033026
- Publisher: https://openreview.net/forum?id=txRZBD8tBV
- Code: https://github.com/Jiacheng-Zhu-AIML/AsymmetryLoRA

## How to cite
@inproceedings{DBLP:conf/icml/ZhuGNBGCGY024,
author       = {Jiacheng Zhu and
                  Kristjan H. Greenewald and
                  Kimia Nadjahi and
                  Haitz S{\'{a}}ez de Oc{\'{a}}riz Borde and
                  Rickard Br{"{u}}el Gabrielsson and
                  Leshem Choshen and
                  Marzyeh Ghassemi and
                  Mikhail Yurochkin and
                  Justin Solomon},
  title        = {Asymmetry in Low-Rank Adapters of Foundation Models},
  booktitle    = {Forty-first International Conference on Machine Learning, {ICML} 2024,
                  Vienna, Austria, July 21-27, 2024},
  publisher    = {OpenReview.net},
  year         = {2024},
  url          = {https://openreview.net/forum?id=txRZBD8tBV},
  timestamp    = {Mon, 02 Sep 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/icml/ZhuGNBGCGY024.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}