AI Learner #10: Fine-Tuning & Alignment — Teaching Models How to Behave

A freshly pretrained model is a genius with no manners. It has read the entire internet, but has no clue it’s supposed to be helpful. Ask it a question and it might just predict more questions. Turning that raw brain into a useful assistant takes two more steps: fine-tuning, then alignment.

From pretrained genius to helpful, aligned assistant

Supervised Fine-Tuning: Imitation Class

First, supervised fine-tuning (SFT). Show the model thousands of examples of good question-and-answer pairs, and it learns to imitate the format. Basically, manners class. This is what turns a text-predictor into something that answers you.

Alignment: The Helpful, Honest, Harmless Triad

But imitation isn’t enough. You want helpful, honest, and harmless — and those are brutally hard to write down as rules. So instead, we teach by preference.

RLHF

The classic recipe is RLHF (reinforcement learning from human feedback). The model writes a couple of answers, a human picks the better one, and the model gets nudged toward that taste. Repeat thousands of times.

RLHF: the model proposes, a human picks the better answer, the model tunes toward it

DPO

There’s a simpler, newer cousin called DPO (direct preference optimization) that skips the separate reward model and learns straight from the preferred answers. Same goal, less machinery — which is why it’s everywhere now.

LoRA and QLoRA: Fine-Tuning for Everyone

Fine-tuning a whole giant model is wildly expensive. So most people use LoRA: freeze the billions of original weights and train just a tiny adapter on top. Cheap, fast, and it runs on one GPU.

LoRA freezes the giant model and trains a tiny adapter

QLoRA goes one further: quantize the frozen model first, so you can fine-tune a serious model on a single consumer GPU. Democratization, basically.

Full fine-tuning vs. LoRA vs. QLoRA memory savings

The Alignment Tax

There’s a tax, though. Push alignment too hard and the model gets timid, or forgets skills it used to have — catastrophic forgetting is a real and slightly tragic thing. Aligning a model is a balancing act between being safe and staying capable.

The Whole Picture

And that’s the whole series. Tokens become numbers, embeddings give them meaning, attention decides what matters, weights store everything learned, decoding turns probability into words, context is the working memory, the KV cache keeps it fast, quantization makes it small, RAG keeps it current, and fine-tuning finally gives it manners. That’s a language model, end to end.

Thanks for learning with us. Now go build something ridiculous.

Quick Quiz 🧠

1. What’s the difference between fine-tuning and alignment?

Answer: Fine-tuning (SFT) teaches a model to follow a format by imitating examples. Alignment (RLHF/DPO) shapes its behavior toward human preferences — helpful, honest, harmless.

2. What does LoRA do?

Answer: It freezes the model’s original weights and trains a small number of new “adapter” parameters, making fine-tuning far cheaper and faster.

3. What is catastrophic forgetting?

Answer: When fine-tuning or heavy alignment causes a model to lose capabilities it previously had, as new training overwrites old knowledge.

Source: RLHF Explained (Ouyang et al.), DPO (Rafailov et al.), LoRA & QLoRA, Guide to Fine-Tuning with LoRA/QLoRA