Fine-Tuning vs. Prompt Engineering

Welcome to the Arena: Prompting, RAG, or Fine-Tuning?

Imagine you just hired a genius intern. They have read the entire internet, but they know absolutely nothing about your company’s proprietary code, internal slang, or how you like your API responses formatted. What do you do?

• Prompt Engineering: You stand over their shoulder and scream instructions at them every time they write a line of code. (Fast, but expensive and highly annoying).
• Retrieval-Augmented Generation (RAG): You hand them a search engine linked to your internal Wiki. Before they answer any question, they search the wiki. (Great for facts, but doesn’t change how they think or write).
• Fine-Tuning: You put them through a 2-week intensive boot camp to rewrite their neural pathways so they adopt your exact coding style, structure, and persona. (Slow and expensive, but they operate autonomously exactly how you want).

Today, we’re doing brain surgery. We are going to fine-tune an open-weight LLM using Parameter-Efficient Fine-Tuning (PEFT) and LoRA. Strap in.

SFT vs. RLHF: The Two Main Flavors

Fine-tuning generally falls into two phases:

1. Supervised Fine-Tuning (SFT): You feed the model prompt-response pairs. It learns to mimic the answers. This is what we will do today.
2. Reinforcement Learning from Human Feedback (RLHF): You have the model output multiple answers, and humans (or another model) rank them. This aligns the model’s behavior with human values (safety, helpfulness).

Data is King: Formatting Your Dataset

An LLM doesn’t learn from random CSVs. It needs structured dialog. The two most common formats are Alpaca (simple instruction-input-output) and ShareGPT (conversational style).

The Alpaca Format (JSON)

[
  {
    "instruction": "Convert the following temperature to Celsius.",
    "input": "77°F",
    "output": "25°C"
  }
]

The ShareGPT / ChatML Format

Modern models (like Llama 3 or Mistral) use conversational roles:

[
  {
    "conversations": [
      {"from": "system", "value": "You are a sarcastic code assistant."},
      {"from": "human", "value": "How do I reverse a list in Python?"},
      {"from": "gpt", "value": "Use `my_list.reverse()`. Was that so hard?"}
    ]
  }
]

LoRA & QLoRA: The Sticky Note Hack

If you wanted to fine-tune Llama 3 8B fully, you would need to update all 8 billion parameters. This requires 80GB A100 GPUs and a small loan. Enter LoRA (Low-Rank Adaptation).

Instead of updating the massive weight matrices $W_0$ directly, we freeze them. We then train two tiny matrices $A$ and $B$ next to them. When input passes through, we multiply it by the frozen base weights and add the output of our small trained matrices. It’s like leaving a sticky note on a textbook rather than rewriting the whole book.

QLoRA goes one step further: it quantizes the base model down to 4-bit precision (squeezing it), allowing you to run feature extraction on a consumer GPU like an RTX 3090 or RTX 4090, while keeping the LoRA adapter in 16-bit float for training.

Show Me the Code: Python Training Script

Here is a complete, production-grade PyTorch script using Hugging Face transformers, peft, and trl to fine-tune a model using QLoRA.

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 1. Setup Configs & Quantization (4-bit Double Quantization)
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# 2. Load Model and Tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    quantization_config=bnb_config, 
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# Prepare model for gradient checkpointing (saves VRAM)
model = prepare_model_for_kbit_training(model)

# 3. Define LoRA Configuration
peft_config = LoraConfig(
    r=16,                  # Rank of the update matrices (smaller = less VRAM, larger = more capacity)
    lora_alpha=32,         # Scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"], # Target self-attention modules
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, peft_config)

# 4. Load Dataset
dataset = load_dataset("json", data_files="my_dataset.json", split="train")

# 5. Define Training Arguments
training_args = TrainingArguments(
    output_dir="./llama3-custom-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    logging_steps=10,
    max_steps=100,
    fp16=False,
    bf16=True, # Set to True if using Ampere/Ada GPU
    optim="paged_adamw_8bit"
)

# 6. Initialize Trainer & Train
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_args
)

trainer.train()
# Save adapter weights only
trainer.model.save_pretrained("./best_adapter")

Merge & Deploy: Rejoining the Adapter

Once training is done, you have a small folder of adapter weights (~100MB). To deploy this in production with low latency, you want to merge it back into the base 15GB model.

from peft import PeftModel

# Load the clean base model (unquantized)
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct", 
    torch_dtype=torch.float16, 
    device_map="cpu"
)

# Load the trained adapter
model = PeftModel.from_pretrained(base_model, "./best_adapter")

# Merge weights
merged_model = model.merge_and_unload()

# Save the unified model
merged_model.save_pretrained("./my-fine-tuned-llama-3")
tokenizer.save_pretrained("./my-fine-tuned-llama-3")

Boom. You now have a custom model optimized to your specific formatting needs. Load it into vLLM, Ollama, or Hugging Face TGI and ship it to production!