LoRA, or Low-Rank Adaptation, is a parameter-efficient fine-tuning method for large neural networks. Instead of updating all model weights, LoRA freezes the base model and trains small low-rank matrices that modify selected layers.
In Large Language Models (LLMs), LoRA is often used to adapt a model to a task, domain, style, or product behavior with much less training memory than full fine-tuning.
The practical advantages include:
- fewer trainable model parameters;
- lower GPU memory requirements;
- smaller task-specific adapter files;
- easier switching between adaptations; and
- less operational overhead than storing a full copy of each fine-tuned model.
LoRA is not a general replacement for high-quality data or evaluation. A poor dataset can still degrade behavior, and an adapter can conflict with safety policies or domain expectations. Teams should evaluate LoRA adapters against held-out tasks, regression suites, and production constraints.
The original paper, LoRA: Low-Rank Adaptation of Large Language Models, introduced the method for adapting large language models with far fewer trainable parameters than full fine-tuning.
The LLM Knowledge Base is a collection of bite-sized explanations for commonly used terms and abbreviations related to Large Language Models and Generative AI.
It's an educational resource that helps you stay up-to-date with the latest developments in AI research and its applications.