What Low-Rank Adaptation Really is and Its Applications

This blog piece was inspired by the recent video uploaded by the author of Low-Rank Adaptation, Edward Hu. Low-rank adaptation (LoRA) is one of the most popular methods for fine-tuning LLMs and more!

Edward uploaded a video where he covers

• What is LoRA?
• How to choose the rank r?
• Does LoRA work for my model architecture?
• Benefits of using LoRA
• Engineering ideas enabled by LoRA