Infrastructure · AI

How to Fine-Tune a Model: LoRA, QLoRA, Datasets, and Avoiding Overfitting

Two years ago, fine-tuning a large language model meant a rack of A100s and a five-figure cloud bill. Today, thanks to LoRA and QLoRA, a single consumer GPU, a few hundred good examples, and an afternoon can specialise a capable model on your domain. But the techniques being accessible doesn’t make them foolproof — there are real decisions about whether to fine-tune at all, how to prepare data, and a handful of silent mistakes that quietly ruin a run. This guide walks the whole process, from deciding to deploying.

When should you fine-tune at all?

The first and most important question is whether you should fine-tune in the first place, because there are four ways to customise a model and three of them require no training at all. Prompt engineering, few-shot examples, and retrieval-augmented generation (RAG) all leave the model untouched, making them faster to try and far easier to update. The honest rule is to reach for those first and fine-tune only when they genuinely fall short.

Fine-tuning earns its place in a few specific situations. The clearest is behavioural consistency at scale: prompt engineering produces variable results, and when you need the model to reliably follow a particular format, tone, or task every time, fine-tuning bakes that in. It also helps when you want a smaller, cheaper, faster model to match a larger one on a narrow task. The critical distinction to internalise is that fine-tuning changes a model’s behaviour, not its knowledge — if you need the model to know fresh or proprietary facts, RAG is the right tool, because it injects information at query time. Many strong systems combine both: RAG for knowledge, a light fine-tune for behaviour. Getting this choice right saves you from training a model to solve a problem that a better prompt would have handled, much as choosing the right GPU for AI work starts with the workload.

RAG versus fine-tuning

Because the most common mistake is fine-tuning when you didn’t need to, it’s worth drawing the line between fine-tuning and retrieval-augmented generation sharply, since they solve genuinely different problems. RAG works by retrieving relevant documents at query time and feeding them to the model as context, which means it excels at giving the model access to knowledge — facts, documents, data that change over time. If your need is “the model should know our product catalogue” or “answer from this week’s policy documents,” RAG is almost always the better answer, because you update it by changing the documents rather than retraining anything.

Fine-tuning, by contrast, changes how the model behaves rather than what it knows. It’s the right tool when you need a consistent output format, a particular tone, reliable adherence to a task structure, or a smaller model that punches above its weight on a narrow job — things no amount of retrieved context reliably produces. The two aren’t rivals so much as complementary, and many of the strongest production systems use both: a fine-tune to lock in behaviour and RAG to supply current knowledge. The practical test is simple — if the gap you’re trying to close is about information, reach for RAG; if it’s about behaviour, fine-tune. Confusing the two leads to expensive training runs that never fixed the actual problem.

LoRA, QLoRA, and full fine-tuning

Once you’ve decided to fine-tune, the method you choose determines your hardware needs and cost, and in 2026 the landscape has consolidated around parameter-efficient approaches. Full fine-tuning updates every parameter in the model and gives maximum quality, but it’s expensive enough that doing it on a single GPU for anything but small models is basically obsolete. The technique that changed everything is LoRA — Low-Rank Adaptation — which freezes the original model and trains only a tiny set of low-rank adapter matrices, roughly a tenth of a percent of the parameters, dramatically cutting memory and time while landing surprisingly close to full fine-tuning quality.

QLoRA goes further by quantizing the frozen base model to 4-bit precision while training the adapters in higher precision, which compresses the memory footprint about fourfold — at a cost of perhaps one to two percent accuracy versus full fine-tuning, a rounding error for most applications. The diagram contrasts the approaches.

For most people, QLoRA with a tool like Unsloth is the right starting point — it’s the cheapest and easiest, and the accuracy gap is negligible. A newer refinement, DoRA, decomposes the adapter updates into magnitude and direction and improves convergence on harder tasks with a single flag, which is why the 2026 default recommendation has become a LoRA rank of 16 with DoRA enabled. The table maps models to hardware.

Choosing a base model and toolchain

With a method settled, you choose a base model and the software to train it, and both decisions have gotten simpler in 2026. For the base, you start from a capable open instruction-tuned model — families like Llama, Qwen, and DeepSeek are common choices — picking a size your hardware can handle and that’s strong enough at the underlying task, since fine-tuning sharpens an existing capability rather than creating one from nothing. A 7-to-8-billion-parameter model is the sweet spot for most domain adaptations on a single GPU.

The toolchain has consolidated around the Hugging Face ecosystem — transformers, datasets, peft, and trl — running on a recent Python, PyTorch, and CUDA. On top of that, the practical recommendation is to start with Unsloth, which is the fastest and easiest path on a single consumer GPU and handles most of the fiddly setup for you. Axolotl, with its YAML-driven configuration, comes into its own when you’re training across four or more GPUs, and TRL’s trainers handle more advanced objectives when you need them. The guidance is consistent: if you’re starting out, use Unsloth, and save the heavier frameworks for when scale actually demands them. This keeps your first fine-tune focused on the data and hyperparameters rather than on wrestling infrastructure.

Preparing your dataset

The dataset is where fine-tuning succeeds or fails, and the single most repeated lesson is that quality beats quantity decisively. You generally want somewhere between 500 and 10,000 examples in instruction-response format, but a clean, carefully curated set of 500 will outperform 5,000 noisy ones for most adaptation tasks. The work is in the curation: cleaning, deduplicating, and making sure each example actually demonstrates the behaviour you want, because the model will faithfully learn whatever patterns — good or bad — are present in the data.

One technical step that’s easy to get wrong is converting your data to the exact chat template your chosen model expects. Every model family has its own format for marking user and assistant turns, and the framework needs to apply that template correctly. This matters enough that when results disappoint, the published evidence is consistent: improving the size and quality of your dataset outperforms fiddling with hyperparameters almost every time. So before you reach for a parameter sweep, look hard at your data — adding cleaner examples is usually the move with the bigger payoff. Tools like Argilla for label review and synthetic-data generators help build that quality at scale.

Which hyperparameters actually matter?

Hyperparameter tuning for LoRA is far less mysterious than it’s often made to seem, and a small set of defaults works the vast majority of the time. The LoRA rank — the size of the adapter — is the main dial: 16 is plenty for simple style adaptation, 32 suits general instruction tuning, and 64 helps for complex multi-turn or coding tasks, with higher ranks costing marginal extra memory and rarely hurting. You set the LoRA alpha to twice the rank, and — importantly — you target all the linear layers (the attention and MLP projections), not just the query and value matrices, because attention-only adapters consistently underperform.

The remaining settings are similarly stable. A learning rate around 2e-4 with a cosine schedule and a short warmup works for QLoRA, while full fine-tuning wants something much lower. The setting most likely to bite you is epochs: two to three passes is right for instruction tuning on a few thousand to tens of thousands of examples, and more than that overfits fast on small datasets — which is exactly why you watch evaluation loss rather than just training loss. Effective batch sizes in the 16-to-64 range, reached via gradient accumulation, work well, and smaller batches actually tend to generalise better with LoRA than you might expect. The 2026 stack centres on the Hugging Face ecosystem — transformers, peft, and trl — with Unsloth for speed on a single GPU and Axolotl for multi-GPU YAML pipelines.

What goes wrong when fine-tuning?

Several mistakes ruin fine-tuning runs silently — the training completes, the loss looks fine, and the result is quietly broken — so knowing them in advance saves real time and money. The terminal shows a representative training setup and the checks that catch these.

# QLoRA defaults (Unsloth / TRL SFTTrainer)
model, tokenizer = FastLanguageModel.from_pretrained(“Qwen3-7B”, load_in_4bit=True)
model = FastLanguageModel.get_peft_model(model,
  r=16, lora_alpha=32, use_dora=True,
  target_modules=[“q_proj”,“k_proj”,“v_proj”,“o_proj”,“gate_proj”,“up_proj”,“down_proj”])
# learning_rate=2e-4, num_train_epochs=2, warmup_ratio=0.03, cosine schedule
# The silent killers — verify BEFORE you train:
assert tokenizer.apply_chat_template(…)  # right template for THIS model
train_on_responses_only(…)               # loss masking: assistant turns only
# keep a validation set; stop early if eval loss climbs (overfitting)
# QLoRA merge: dequantize base to bf16 BEFORE merge_and_unload()

Four mistakes recur. The first is using the wrong chat template — fine-tuning a Mistral model with Llama-style tokens, for instance, silently produces a broken adapter, so you verify the tokenizer’s template matches the model before training. The second is forgetting loss masking: if your framework computes loss over the whole input rather than just the assistant’s responses, the model learns to predict its own questions, which is useless. The third is overfitting — LoRA has few trainable parameters but still overfits small datasets, so you always keep a validation set, monitor eval loss, and stop early if it climbs. The fourth is a merge-order trap specific to QLoRA: the 4-bit base must be dequantized to a higher precision before you merge the adapter, or you lose precision. Avoiding these four removes most of the ways a run quietly fails.

How do you know it worked?

The final and most overlooked step is evaluating honestly, because a low training loss tells you almost nothing about whether the fine-tune succeeded. The blunt truth is that a fine-tune which doesn’t improve your actual target metric has failed, no matter how beautiful the loss curve looks. So you define a task-specific metric upfront — accuracy on a held-out test set, a rubric score, whatever measures the behaviour you were trying to instil — and you measure the fine-tuned model against the base model on it.

Two more checks round out a sound evaluation. You watch for capability regression by comparing a general benchmark like MMLU before and after, since aggressive fine-tuning can degrade the broad abilities you didn’t mean to touch. And you do a side-by-side qualitative comparison, reading the fine-tuned and base outputs on real prompts, because numbers miss things a human notices immediately. Once it passes, you merge the adapter into the base model and export — commonly to a format like GGUF for local inference with llama.cpp or Ollama. For the infrastructure to run all this, a GPU server with adequate VRAM is the foundation, whether your own or rented, and our GPU server setup guide covers building one — while serving the result reliably pairs naturally with containerised deployment from our Docker guide. For teams wanting that GPU capacity on dedicated hardware, our dedicated servers in Toronto give fine-tuning and inference a controllable home — though the real determinant of success remains a clean dataset and an honest evaluation, not the hardware.

Merging and deploying the result

Once a fine-tune passes evaluation, a few steps turn the trained adapter into something you can actually serve. The adapter you trained is a small separate set of weights, so you merge it into the base model to produce a single standalone model — and here a QLoRA-specific care applies: the 4-bit base must be dequantized back to a higher precision like BF16 before you merge, since merging an adapter directly onto a quantized base loses precision and degrades the result. Skipping this is one of the silent failure modes worth re-checking at deploy time.

From the merged model, you export to whatever format your serving stack expects. A very common path for local inference is exporting to GGUF, the format used by llama.cpp and Ollama, which lets you run the fine-tuned model efficiently on modest hardware, optionally quantized again for inference. Whatever the target, the deployment concerns are the ordinary ones — reproducible packaging, resource limits, and reliable serving — which is where containerising the inference service pays off, the subject of our Docker deployment guide. The fine-tuned model is only useful once it’s running somewhere dependable, so treating deployment as part of the project rather than an afterthought keeps the work from stalling at a checkpoint file on a developer’s laptop.