Fine-Tuning LLMs on Custom Data: A Developer’s Practical Guide

Fine-Tuning LLMs with Custom Data: A Practical Production Guide

Large language models are impressive out of the box, but they struggle with domain-specific terminology, company-internal knowledge, and specialized output formats. Fine-tuning LLMs with custom data bridges that gap — transforming a general-purpose model into one that speaks your domain fluently. Therefore, this guide walks you through when to fine-tune versus prompt engineer, how to prepare training data, and how to train efficiently with LoRA and QLoRA techniques that don’t require a GPU cluster.

When to Fine-Tune vs Prompt Engineering

Before investing weeks in fine-tuning, understand when it actually helps. Prompt engineering (few-shot examples, system prompts, chain-of-thought) solves a large share of customization needs. Moreover, it’s faster to iterate and doesn’t require training infrastructure. Fine-tuning is the right choice when you need consistent output formatting (structured JSON, specific templates), domain-specific language the base model gets wrong (medical codes, legal citations, internal jargon), latency reduction (a smaller fine-tuned model can outperform a larger prompted model), or cost reduction (shorter prompts because the model already knows context).

A practical decision framework: start with prompt engineering. If you’re spending more than 500 tokens on instructions and examples in every request, fine-tuning will save money and reduce latency. Additionally, if the model consistently fails on domain-specific tasks despite good prompts, fine-tuning is the answer. It is worth noting what fine-tuning does not do well — it teaches behavior and style, not facts. For injecting fresh, frequently changing knowledge, retrieval-augmented generation is the better tool, and the two techniques are complementary rather than competing.

# Decision: prompt engineering vs fine-tuning
# Rule of thumb: calculate your per-request overhead

# Prompt engineering approach — 600 token system prompt every call
system_prompt = """
You are a medical coding assistant. Use ICD-10 codes.
Format: {"code": "...", "description": "...", "confidence": 0.0-1.0}
Examples:
- "chest pain" -> {"code": "R07.9", "description": "Chest pain, unspecified", "confidence": 0.95}
- "type 2 diabetes" -> {"code": "E11.9", "description": "Type 2 diabetes mellitus", "confidence": 0.98}
... (20 more examples)
"""
# Cost: 600 tokens * 10,000 requests/day = 6M tokens/day in overhead

# Fine-tuned approach — model already knows the format and domain
# system_prompt = "Extract ICD-10 codes from clinical notes."
# Cost: 50 tokens * 10,000 requests/day = 500K tokens/day
# Savings: ~90% reduction in prompt tokens

Data Preparation: Quality Over Quantity

Training data quality determines fine-tuning success more than anything else. You need 500-5,000 high-quality examples for most use cases — not millions. Each example should follow a consistent format with clear input-output pairs. Furthermore, diverse examples covering edge cases matter more than repetitive examples of common cases.

import json
import random

# Structure: each example is a conversation with system, user, assistant
training_data = []

# Load your domain-specific examples
raw_examples = load_from_database()  # Your labeled data

for example in raw_examples:
    training_data.append({
        "messages": [
            {"role": "system", "content": "You are a medical coding assistant. Return ICD-10 codes as JSON."},
            {"role": "user", "content": example["clinical_note"]},
            {"role": "assistant", "content": json.dumps(example["expected_codes"])}
        ]
    })

# Quality checks before training
def validate_dataset(data):
    issues = []
    for i, item in enumerate(data):
        msgs = item["messages"]
        # Check structure
        if len(msgs) < 2:
            issues.append(f"Example {i}: too few messages")
        # Check assistant response is valid JSON
        assistant_msg = next((m for m in msgs if m["role"] == "assistant"), None)
        if assistant_msg:
            try:
                json.loads(assistant_msg["content"])
            except json.JSONDecodeError:
                issues.append(f"Example {i}: invalid JSON in response")
        # Check for duplicates
        user_msg = next((m for m in msgs if m["role"] == "user"), None)
        if user_msg and len(user_msg["content"].strip()) < 10:
            issues.append(f"Example {i}: user message too short")
    return issues

# Split: 90% train, 10% validation
random.shuffle(training_data)
split = int(len(training_data) * 0.9)
train_set = training_data[:split]
val_set = training_data[split:]

# Save in JSONL format (one JSON object per line)
with open("train.jsonl", "w") as f:
    for item in train_set:
        f.write(json.dumps(item) + "\n")

print(f"Training: {len(train_set)}, Validation: {len(val_set)}")

Common data preparation mistakes include inconsistent formatting across examples, training on incorrect or ambiguous labels, insufficient diversity in inputs, and forgetting to validate output format. Consequently, always have domain experts review a sample of your training data before starting a training run.

One subtle but expensive trap is data leakage between your train and validation splits. If near-duplicate records — the same clinical note with trivial whitespace differences — land on both sides of the split, your validation accuracy will look excellent while real-world performance disappoints. Therefore, deduplicate by a normalized hash of the input before splitting, and where possible split along a natural boundary such as patient or document ID so that no single source leaks across the divide.

LoRA and QLoRA: Efficient Fine-Tuning

Full fine-tuning updates all model parameters — for a 7B parameter model, that requires 28GB+ of GPU memory just for the model weights, plus optimizer states. LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices, reducing trainable parameters by 99%. QLoRA goes further by quantizing the base model to 4-bit, letting you fine-tune a 7B model on a single 24GB GPU.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

# QLoRA: 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True  # Nested quantization saves more memory
)

# Load base model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.3",
    quantization_config=bnb_config,
    device_map="auto"
)

# LoRA config: which layers to adapt
lora_config = LoraConfig(
    r=16,                    # Rank — higher = more capacity, more memory
    lora_alpha=32,           # Scaling factor (usually 2x rank)
    target_modules=[         # Which layers to add adapters to
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

# Check trainable parameters
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)")
# Output: Trainable: 13,107,200 / 7,241,748,480 (0.18%)

The key LoRA hyperparameters are rank (r) and alpha. A rank of 8-16 works for most tasks. Higher ranks capture more complexity but use more memory. Alpha should typically be 2x the rank. Additionally, targeting all attention and MLP projection layers gives the best results for instruction-following tasks. Be aware of the trade-off QLoRA makes: 4-bit quantization of the frozen base introduces a small quality cost relative to LoRA on a full-precision model, which is usually negligible for narrow domain tasks but can matter for reasoning-heavy work where every percentage point counts.

Training Strategies and Hyperparameters

Training a fine-tuned model isn't like training from scratch. You're adjusting an already-capable model, so aggressive learning rates destroy existing capabilities. Start with a learning rate of 1e-4 to 2e-5, train for 2-5 epochs, and use a cosine learning rate scheduler. More importantly, monitor validation loss — if it starts increasing while training loss decreases, you're overfitting.

# Training configuration for QLoRA fine-tuning
training_config = SFTConfig(
    output_dir="./fine-tuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,     # Effective batch size: 16
    learning_rate=2e-4,
    weight_decay=0.01,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    eval_strategy="epoch",
    fp16=True,
    max_seq_length=2048,
    dataset_text_field="text",
    packing=True,                      # Pack short examples together
)

trainer = SFTTrainer(
    model=model,
    args=training_config,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    tokenizer=tokenizer,
)

trainer.train()

# Save the LoRA adapter (small — typically 50-200MB)
trainer.save_model("./fine-tuned-adapter")
# The base model stays unchanged — you only deploy the adapter

A frequently underestimated risk is catastrophic forgetting: aggressive or over-long training can erase general capabilities the base model had, so a model fine-tuned to extract ICD-10 codes suddenly answers ordinary questions poorly. Mixing in a small fraction of general instruction data, keeping epochs modest, and evaluating against off-domain prompts all help preserve breadth. The goal is a specialist that still remembers how to be a generalist.

Deploying and Serving the Adapter

One of the most practical advantages of LoRA is that the adapter is tiny and decoupled from the base weights. Because the base model is unchanged, a single GPU can host one set of base weights and hot-swap multiple adapters for different tasks or tenants, which dramatically lowers serving cost compared to deploying several full models. Inference servers such as vLLM support exactly this multi-adapter pattern.

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Option A: keep base + adapter separate (multi-tenant, hot-swappable)
base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.3")
model = PeftModel.from_pretrained(base, "./fine-tuned-adapter")

# Option B: merge adapter into base weights for single-purpose, lowest-latency serving
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")
# Merged model has no PEFT overhead at inference time, but loses the
# ability to swap adapters — choose per deployment, not by default

Merging trades flexibility for a small latency gain; keeping the adapter separate trades that latency for the ability to serve many specializations from one base. Choose deliberately, version the adapter alongside the exact base checkpoint and tokenizer it was trained against, and never assume an adapter is portable across base model versions — it is not.

Evaluation: Measuring Fine-Tuning Success

Loss curves alone don't tell you if fine-tuning worked. You need task-specific evaluation metrics. For classification tasks, measure accuracy and F1 score. For generation tasks, use human evaluation alongside automated metrics like ROUGE or BERTScore. Furthermore, always compare against the base model with your best prompt to quantify the actual improvement from fine-tuning.

# Evaluation framework for fine-tuned models
def evaluate_model(model, tokenizer, test_cases):
    results = {"correct": 0, "total": 0, "errors": []}

    for case in test_cases:
        input_text = case["input"]
        expected = case["expected_output"]

        # Generate prediction
        inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
        outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1)
        prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)

        # Compare (task-specific comparison)
        if task_match(prediction, expected):
            results["correct"] += 1
        else:
            results["errors"].append({
                "input": input_text[:100],
                "expected": expected,
                "got": prediction
            })
        results["total"] += 1

    accuracy = results["correct"] / results["total"]
    print(f"Accuracy: {accuracy:.2%}")
    print(f"Errors: {len(results['errors'])} / {results['total']}")
    return results

When NOT to Fine-Tune: Trade-offs and Costs

Fine-tuning carries ongoing costs that a quick proof of concept hides. Every time the base model is upgraded, you must retrain and re-evaluate to inherit the improvements, which turns the adapter into a maintenance commitment rather than a one-time task. Moreover, the training data itself becomes a governance asset: if it contains PII or regulated content, the resulting weights may be considered to embed that data, with corresponding privacy obligations.

Avoid fine-tuning when your knowledge changes daily, when you have fewer than a few hundred clean examples, or when a well-engineered prompt already meets your quality bar. In those situations, retrieval, better prompting, or simply a larger base model deliver more value for less operational burden. Reach for fine-tuning when behavior and format must be reliably reproduced, when prompt overhead dominates your token bill, or when latency targets demand a smaller specialized model.

Related Reading:

• RAG Architecture Patterns for Production
• AI Agents and Tool Use Patterns
• Fine-Tuning LLMs with LoRA and QLoRA

Resources:

• Hugging Face PEFT Documentation
• LoRA: Low-Rank Adaptation of Large Language Models (Paper)

In conclusion, fine-tuning LLMs with custom data is a powerful technique when prompt engineering hits its limits. Start with high-quality, leak-free data preparation, use QLoRA for cost-effective training on consumer hardware, guard against catastrophic forgetting, and always measure improvement against a prompted baseline. The key insight is that 1,000 carefully curated examples often outperform 100,000 noisy ones.