AI Learner #4: Weights & Parameters — What’s Actually Inside the Model

We’ve covered attention (Part 3), but attention is just plumbing — it organizes information, it doesn’t store any. So where does a model’s actual knowledge live? Not a database. Not a knowledge graph. In the weights — billions of plain numbers that decide how every signal flows.

The Knobs-and-Levers Analogy

Picture a machine with millions of tiny dials. Each one sets how strongly one neuron’s output pushes on the next. Big dial, strong influence. Tiny or negative, basically ignored. And nobody turns those dials by hand — they’re learned, nudged automatically by an algorithm keeping score of how wrong the model is.

What Is a “Parameter”?

Quick vocabulary check. The weights (and biases) the model learns are its parameters. The stuff engineers pick beforehand — number of layers, learning rate, batch size — are hyperparameters. Learned versus chosen. That distinction matters more than it sounds.

The Scale: How Many Numbers?

This is where it gets a little silly. A “small” model like Llama 3.1 8B has 8 billion of these numbers. Llama 3.3 70B carries 70 billion. GPT-4-class flagships are estimated in the trillions.

To put 70 billion in perspective: printed out, that’s about 1,600 books filled with nothing but decimals. And that’s a model that fits on a single GPU.

Where Do the Weights Live?

In every layer. A chunk sits in the attention projections (Q, K, V), a sliver in the output, but a full two-thirds hide in the humble feed-forward network.

How Training Sets the Weights

So how do billions of numbers find the right values? A loop called backpropagation: predict the next token, measure how wrong (the loss), then nudge every single weight a hair toward less-wrong. Repeat a few billion times.

The gradients say which direction to nudge. The learning rate says how far. Too far and you overshoot; too timid and you’re training until the heat death of the universe.

What Do the Weights Actually “Know”?

Here’s the genuinely weird part. The weights don’t store facts. There is no single number that says “Paris is the capital of France.” The knowledge is distributed — smeared across millions of weights, each contributing a sliver to the overall probability. Which is why you can’t just edit one number to make a model forget something. Pull one thread and the whole pattern shifts.

Bigger Isn’t Always the Answer

Training also walks a tightrope called overfitting: memorize the examples too well, and the model aces its homework but flunks the real test. The fix is mostly to show it more, and more varied, data. And while more parameters usually means smarter, it also means slower, pricier, and hungrier — so the field keeps hunting for the sweet spot (mixture-of-experts, quantization, and friends).

What Comes Next

We know what’s inside now: billions of learned numbers wearing a transformer coat. But having the knowledge isn’t the same as using it to write.

Coming up: how models write text — decoding strategies and the art of prediction.

---

Quick Quiz 🧠

1. What’s the difference between weights and hyperparameters?

Answer: Weights are learned during training to minimize prediction error. Hyperparameters (learning rate, number of layers, batch size) are chosen by engineers before training and aren’t themselves learned.

2. Why do feed-forward layers hold two-thirds of the parameters?

Answer: They project the hidden state up ~4× and back, and those projection matrices scale with the square of the hidden dimension — far more weights than the attention layers.

3. Why can’t you edit one weight to change what a model “knows”?

Answer: Knowledge is distributed across all the weights — no single weight stores a fact. Changing one shifts many outputs slightly rather than toggling a fact on or off.

---

Source: IBM — LLM Parameters, How LLMs Work (Sean Goedecke), Anatomy of an LLM (Webopedia)