What Low-Rank Adaptation Really is and Its Applications

Ching (Chingis)
5 min readFeb 10, 2024

This blog piece was inspired by the recent video uploaded by the author of Low-Rank Adaptation, Edward Hu. Low-rank adaptation (LoRA) is one of the most popular methods for fine-tuning LLMs and more!

Edward uploaded a video where he covers

  • What is LoRA?
  • How to choose the rank r?
  • Does LoRA work for my model architecture?
  • Benefits of using LoRA
  • Engineering ideas enabled by LoRA

This blog connects his video with the paper and adds an idea proposed in academia to provide a very detailed summary of LoRA. I hope you enjoy reading this piece!


The rank of a matrix is the maximum number of linearly independent rows or columns in the matrix.



A neural network contains many dense layers which perform matrix multiplication. The weight matrices in these layers typically have full-rank. When adapting to a specific task, Aghajanyan et al. (2020) shows that the pre-trained language models have a low…



Ching (Chingis)

I am a passionate student. I enjoy studying and sharing my knowledge. Follow me/Connect with me and join my journey.