Fine-Tuning LLMs: LoRA, QLoRA, and Practical Guide
Fine-tuning adapts a pre-trained LLM to excel at specific tasks or domains. Instead of training a model from scratch — a process that costs millions of dollars and requires thousands of GPUs — you start with a capable base model and continue training on your own data. This transfers the model’s general knowledge to your specific domain, whether that’s medical diagnosis, legal document analysis, customer support, or code generation.
The economics of fine-tuning have transformed with parameter-efficient methods. Full fine-tuning of a 70B model required 8+ A100 GPUs and cost thousands of dollars per run. With LoRA and QLoRA, you can fine-tune the same model on a single consumer GPU for tens of dollars. This democratization has made fine-tuning accessible to startups, academic labs, and individual developers — anyone with a well-curated dataset can now build a specialized model.
When to Fine-Tune vs. Use RAG vs. Prompt Engineering
Fine-tuning is one of three strategies for specializing LLMs, each with distinct trade-offs. Prompt engineering is the cheapest and fastest — you describe the task in natural language and provide few-shot examples. It works well for simple tasks but degrades for complex domain-specific requirements. RAG (retrieval-augmented generation) grounds responses in external knowledge without modifying the model — excellent for factual recall but limited for tasks requiring deep domain understanding.
Fine-tuning is appropriate when: you need consistent output in a specific format or style across all inputs (your legal documents must follow strict templates); RAG alone doesn’t capture the depth of domain knowledge needed (the model needs to internalize medical reasoning, not just retrieve facts); you want to match a specific brand voice or domain terminology precisely; or you need lower latency than prompt engineering with long context (fine-tuned models can achieve equivalent quality with shorter prompts).
A practical heuristic: start with prompt engineering for minimum viable product, add RAG for knowledge grounding, and consider fine-tuning only when prompt engineering with RAG fails consistently on 10%+ of test cases. Most production systems use a combination — fine-tuned model for core capability, RAG for up-to-date facts, and prompt engineering for task-specific instructions.
Full Fine-Tuning: When You Need Maximum Quality
Full fine-tuning updates all model parameters — typically 7 billion to 405 billion weights. This gives the model maximum flexibility to adapt to your domain but requires substantial computational resources. A full fine-tuning run for a 7B model requires 40-80 GB of GPU memory per batch, scaling linearly with model size. A 70B model requires 400-800 GB — multiple high-end GPUs or specialized hardware.
The training process uses a variant of the original pre-training objective, typically causal language modeling (predicting the next token) on your domain data. For instruction tuning, you use a supervised fine-tuning (SFT) loss on instruction-response pairs. The learning rate is typically 1-5% of the pre-training rate (around 1e-5 to 5e-5) with cosine decay and a warmup period of 100-500 steps.
Full fine-tuning risks catastrophic forgetting — the model may lose general capabilities as it specializes. Mitigation strategies include: mixing 10-20% general-domain data with your domain data during training, using lower learning rates, training for fewer epochs (1-3 rather than 5-10), and evaluating on general benchmarks alongside domain-specific metrics throughout training.
LoRA: Low-Rank Adaptation
LoRA (Low-Rank Adaptation), introduced by Hu et al. (2021) in “LoRA: Low-Rank Adaptation of Large Language Models,” revolutionized fine-tuning by making it accessible. The key insight is that weight updates during fine-tuning have low intrinsic rank — they can be represented as the product of two small matrices rather than a full-rank update to the original weight matrix.
A LoRA adapter for a single weight matrix consists of matrices A (dimensions d × r) and B (dimensions r × k), where r is the rank (typically 8-64). The original weight W (dimensions d × k) remains frozen. The forward pass computes h = Wx + BAx — the original weight plus the low-rank update. Since r is much smaller than d and k, LoRA reduces trainable parameters by 100-1000x. For a 7B model, full fine-tuning updates 7B parameters; LoRA trains only 4-40 million.
LoRA offers additional benefits beyond memory efficiency. The tiny adapter files (16-200 MB compared to 14 GB for a full 7B model checkpoint) are portable and easy to version. Multiple LoRA adapters can be swapped at inference time without restarting the model, enabling a single base model to serve multiple specialized tasks. Companies like Replicate and Together AI use LoRA adapters to offer millions of community-trained models on their platforms.
Practical LoRA configuration: rank r=16 for most tasks, r=8 for simple tasks, r=64 for complex domains with limited generalizability. Target attention projection matrices (q_proj, v_proj, k_proj, o_proj) for best results. LoRA alpha = 2× rank is a good default. Dropout of 0.05-0.1 prevents overfitting on small datasets.
QLoRA: Fine-Tuning on Consumer Hardware
QLoRA (Dettmers et al., 2023) combines LoRA with 4-bit NormalFloat quantization, enabling fine-tuning of 65B models on a single 48GB GPU. The base model is quantized to 4-bit precision (reducing memory by 4x), while the LoRA adapters remain in full precision (float32). The result: you can fine-tune a model that would normally require 300GB+ of GPU memory on a single high-end consumer card.
The QLoRA paper introduced several innovations. 4-bit NormalFloat quantizes weights using information-theoretically optimal bin boundaries for normally distributed data. Double quantization quantizes the quantization constants themselves, saving additional memory. Paged optimizers use unified memory to handle gradient checkpointing spikes that exceed GPU memory by spilling to CPU RAM.
QLoRA achieves nearly identical performance to full fine-tuning on most benchmarks. Dettmers’ team showed that a 65B Vicuna model fine-tuned with QLoRA matched the performance of the full-precision version within 0.5% on MMLU and 1% on GSM8K. For practical purposes, QLoRA is indistinguishable from full fine-tuning at a fraction of the cost.
Hardware requirements: a 7B model with QLoRA fits on 8GB VRAM (Raspberry Pi can run inference; an RTX 3070 can fine-tune). A 13B model needs 16GB (RTX 4090). A 70B model needs 48GB (A6000, A100, or dual RTX 4090s with NVLink). Apple Silicon with unified memory (M2 Ultra, M3 Max) can fine-tune 7B-13B models using MLX.
Dataset Preparation: Quality Over Quantity
Dataset quality is the single most important factor in fine-tuning success. A well-curated dataset of 1,000 examples often outperforms a noisy dataset of 100,000. Fine-tuning teaches the model patterns, not facts — a few high-quality demonstrations of the desired input-output relationship are more valuable than thousands of inconsistent examples.
Format your data as instruction-response pairs. The instruction describes the task (with optional context); the response is the desired model output. Include diverse examples covering edge cases — what should the model do when information is missing, when the user asks a question outside the domain, or when the request is ambiguous? These “corner case” examples are disproportionately valuable for robust behavior.
Data deduplication is essential. Duplicate examples overrepresent certain patterns and can cause the model to memorize rather than generalize. Use MinHash or embedding similarity to identify and remove near-duplicates. Data augmentation — creating variations of existing examples through paraphrasing, word replacement, or back-translation — can improve robustness when data is scarce.
Datasets should be split into training (80%), validation (10%), and test (10%). Use the validation set to monitor for overfitting during training — if validation loss increases while training loss decreases, stop training. The test set provides an unbiased final evaluation.
Training with Axolotl
Axolotl is the most popular open-source framework for fine-tuning, supporting Llama, Mistral, Qwen, Gemma, and dozens of other architectures. It provides a YAML-based configuration system that abstracts away the complexity of training loops, mixed precision, gradient checkpointing, and distributed training.
A minimal Axolotl configuration specifies: the base model, LoRA hyperparameters (rank, alpha, target modules), training parameters (batch size, learning rate, epochs, warmup steps), and dataset path. Axolotl handles dataset formatting (Alpaca, ShareGPT, ChatML, and custom formats), tokenization (with packing for efficiency), and checkpointing.
Best practices for Axolotl training: use sample_packing: true to pack multiple training examples into each sequence, maximizing throughput. Set flash_attention: true for 2-4x attention speedup. Use bf16: auto for mixed precision training. Monitor training with Weights & Biases integration by setting wandb: true with your project name.
Evaluation and Deployment
Evaluating fine-tuned models requires both automated metrics and human evaluation. Automated metrics include: perplexity on validation data (lower is better), format adherence (percentage of outputs matching required structure), and benchmark performance (MMLU, HumanEval, GSM8K depending on domain). Human evaluation with side-by-side comparisons against the base model provides the most meaningful quality assessment.
Deployment options depend on your infrastructure. LoRA adapters can be merged into the base model (merge_and_unload() in PEFT) for single-model serving, or loaded separately with the base model frozen. vLLM supports LoRA adapters with zero-copy switching between multiple adapters — useful for multi-tenant applications. Ollama supports Modelfiles that incorporate fine-tuned adapters.
For production, use the fine-tuned model’s performance characteristics to build confidence. Test on real user traffic gradually (5% → 25% → 100%), monitor quality metrics, and maintain a fallback path to the base model if the fine-tuned version regresses.
FAQ
How much data do I need for fine-tuning? As few as 100 high-quality examples can show measurable improvement for narrow tasks. For robust domain adaptation, target 1,000-10,000 examples covering diverse scenarios. Beyond 10,000, returns diminish — finer-grained data improvements or architectural changes become more important.
Can I fine-tune a model on proprietary data without it leaking? Fine-tuning does not guarantee privacy. Models can memorize training examples and potentially leak them through careful prompting. Use differential privacy (DP-SGD) to provide formal privacy guarantees, though this reduces effective training data by 10-50%. For sensitive data, self-host your fine-tuned model rather than using API-based fine-tuning services.
How do I choose between LoRA and full fine-tuning? Use LoRA for most applications. It achieves 95-99% of full fine-tuning quality at 90% lower cost. Reserve full fine-tuning for cases where LoRA demonstrably underperforms — typically when the domain requires broad, deep adaptation across many model layers simultaneously.
How long does fine-tuning take? For a 7B model with LoRA: 1-4 hours on an RTX 4090 for 1,000 examples over 3 epochs. For a 70B model with QLoRA: 8-24 hours on an A100 for 5,000 examples. Data preparation and evaluation typically take longer than training itself.
Should I fine-tune the base model or an instruction-tuned version? Fine-tune from the instruction-tuned version whenever possible. Instruction-tuned models have learned to follow directions and produce helpful responses. Fine-tuning from the base model requires learning both the task and helpful conversational behavior, requiring more data and longer training.