AI Learner #8: Quantization — Making Models Smaller Without Losing Their Minds

A frontier model can be hundreds of gigabytes of numbers. Your laptop has maybe eight. So how does a giant model ever run on normal hardware? You shrink it.

The Precision Ladder

Every weight is stored at some precision — how many bits each number gets. Full precision is 32 bits. Drop to 16, then 8, then 4, and the model gets dramatically smaller at each rung. A 70B model in FP16 is ~140 GB; in 4-bit, closer to ~35 GB.

Rounding each weight to one of a few levels makes the model far smaller

How Quantization Actually Works

The trick is quantization: instead of storing each weight as a fussy high-precision decimal, you round it to one of just a handful of levels. It sounds reckless — rounding off billions of numbers — but neural nets are surprisingly forgiving. Go to 4 bits and the model is ~4× smaller, and barely any worse.

The trade: much smaller, a bit faster, only a tiny dip in quality

Two Approaches: PTQ vs. QAT

There are two ways to do it. Post-training quantization (PTQ) just compresses a finished model — quick and easy, with a little quality loss. Quantization-aware training (QAT) bakes the squeeze in during training — more effort, but it holds quality better.

PTQ compresses after training; QAT trains aware of the compression

The Methods You’ll See

In the wild you’ll see names like GGUF (the format behind llama.cpp and most local models), GPTQ, and AWQ. Different recipes, exact same goal: fit a big brain into a small box. You’ll also see labels like Q4_K_M — that’s a 4-bit GGUF variant balancing size and quality.

Why It Matters

Quantization is why a model that needed a data center last year runs on your gaming laptop today — same intelligence, a quarter of the weight. It’s also applied to the KV cache itself, shrinking memory even further.

What Comes Next

A model’s knowledge is frozen the day training ends. So how do you give it fresh, private, up-to-the-minute information it was never trained on?

Coming up: RAG — retrieval-augmented generation, or how to give a model an open-book exam.

Quick Quiz 🧠

1. What does quantization reduce?

Answer: The numerical precision (bits per weight) used to store the model’s parameters — e.g., from 16-bit floats down to 4-bit integers — shrinking memory and speeding up inference.

2. What’s the difference between PTQ and QAT?

Answer: Post-training quantization compresses an already-trained model (fast, slight quality loss). Quantization-aware training accounts for the compression during training (more work, better quality).

3. Roughly how much smaller is a 4-bit model than a 16-bit one?

Answer: About 4× smaller, since each weight uses 4 bits instead of 16 — with only a small drop in quality for most uses.

Source: FP8 Quantization Explained, Quantized Neural Networks, Choosing a GGUF Model