How LLMs Actually Work

The Fundamental Problem: Computers Can't Read

Here's something that sounds obvious but has enormous consequences: computers work with numbers. Not letters, not meaning, not context — just numbers. So before a machine can do anything with language, it needs to answer a deceptively hard question:

How do you turn a word into a number — in a way that preserves what the word means?

The naive approach is to assign each word an ID. "Cat" = 1, "dog" = 2, "house" = 3. But this is terrible — it implies that "cat" and "dog" (IDs 1 and 2) are more similar than "cat" and "house" (IDs 1 and 3), which is sometimes true and sometimes not. The numbering is arbitrary. It captures nothing about meaning.

The breakthrough came from a beautifully simple idea.

The Personality Test Insight

Imagine you take a personality test — something like the Big Five, which scores you on five traits: how introverted vs. extraverted you are, how agreeable, how open to new experiences, and so on. After the test, you're represented as five numbers. Maybe you're [0.8, 0.3, 0.6, 0.9, 0.4].

Those five numbers capture something real about you. And here's the key insight: if someone else has similar numbers, they're probably a similar person. You can literally measure how similar two people are by comparing their number lists (using a technique called cosine similarity — it measures the angle between two vectors, where a smaller angle means more similar meaning).

In 2013, researchers applied this exact idea to words. Instead of five personality dimensions, they used 50 to 300 dimensions. And instead of a personality quiz, they used something far more clever — they let the computer figure out the dimensions by reading billions of words of text.

The result was Word2Vec: a method that turns every word into a list of numbers (called a vector or embedding) where similar words end up with similar numbers.

This is extraordinary. The machine has never been told what "king" or "queen" means. It learned these relationships purely by observing which words appear near each other in text. The underlying principle — called the distributional hypothesis — is beautifully expressed as: "You shall know a word by the company it keeps."

Companies like Airbnb, Spotify, and Alibaba took this same idea and applied it beyond words — embedding listings, songs, and products into vector spaces to power recommendation engines. The idea is that universal.

But these embeddings have a critical flaw — one that would drive the next major breakthrough.

The Flaw That Sparked a Revolution

Consider the word "bank." In Word2Vec, it gets one vector — the same whether you're talking about a river bank or a savings bank. The context is completely lost. Every word is frozen in a single meaning, regardless of the sentence it appears in.

This matters because language is deeply contextual. "I went to the bank to deposit money" and "I sat on the bank of the river" use the same word with completely different meanings. The first generation of embeddings couldn't tell them apart.

Solving this would require a fundamentally new idea. And that idea — attention — would change everything.

The Bottleneck Problem

Before attention, the best language models worked like this: read the input sentence one word at a time (left to right), compress everything you've seen into a single summary vector, then use that vector to generate the output.

Think of it like an exam where you read a 30-page textbook, close it, and then answer questions from a one-paragraph summary you wrote. For short passages, this works fine. But for long texts? Critical details get crushed into that tiny summary.

This was the context bottleneck — the Achilles' heel of early sequence-to-sequence models. In translation systems, longer sentences produced significantly worse results because the model had to squeeze "European Economic Area" and everything else into a single vector of 256 or 512 numbers.

Attention: Looking Back at Your Notes

The solution, proposed in 2014, was elegant: what if the decoder could look back at the entire input, not just a summary?

Instead of compressing everything into one vector, the encoder now keeps all its intermediate states — one for each input word. When the decoder needs to generate the next output word, it scores every input word's state for relevance, creates a weighted combination, and uses that as context.

Going back to the exam analogy: instead of writing one summary and closing the book, you now get to keep the textbook open and flip to the relevant page for each question. The model literally learns where to look.

In a French-to-English translation system, researchers showed that the model correctly learned that "European Economic Area" maps to "zone économique européenne" — reversed word order! The attention mechanism figured this out purely from data, with no grammar rules programmed in.

Attention solved the bottleneck. But there was still a fundamental limitation: these models processed words one at a time, sequentially. Reading a sentence was like reading through a straw — one word, then the next, then the next. For long documents, this was painfully slow and still lost information.

What if every word could look at every other word — all at once?

That question led to the most important architecture in modern AI. And it has a name you've probably heard.

The Problem with Taking Turns

Remember the attention mechanism from Part 1? It was a huge step forward — the model could finally "look back at its notes" instead of compressing everything into one tiny vector. But there was still a fundamental bottleneck:

RNNs process words one at a time, in order.

Think about reading a book by looking through a keyhole — you can only see one word at a time, and you have to read left to right. Even with attention helping you flip back to earlier pages, you're still stuck processing sequentially. Word 50 has to wait for words 1 through 49 to finish processing.

This created two problems. First, it was painfully slow — you couldn't parallelize the computation because each step depended on the previous one. Second, even with attention, information from early words could degrade by the time you reached word 100.

In 2017, a team at Google published a paper with a deceptively simple title: "Attention Is All You Need." Their argument was radical: throw away the RNNs entirely. Use only attention.

The model they introduced — the Transformer — is the single most important architecture in modern AI. Every model you've heard of — GPT, BERT, Claude, Gemini, LLaMA — is a Transformer or a direct descendant of one.

The Group Discussion Analogy

The Transformer's key innovation is called self-attention, and the best way to understand it is to compare it with what came before.

An RNN processes language like a game of telephone: person 1 whispers to person 2, who whispers to person 3, and so on. By the time the message reaches person 20, it might be garbled. Each person only ever talks to their immediate neighbor.

Self-attention works like a group discussion: everyone in the room can talk to everyone else simultaneously. Person 20 can directly ask person 1 a question without going through 18 intermediaries. And critically — all these conversations happen at the same time.

Here's a concrete example. Consider the sentence:

"The animal didn't cross the street because it was too tired."

What does "it" refer to? You instantly know it means "the animal" — but how would a machine figure that out? With self-attention, when the model processes the word "it," it simultaneously looks at every other word in the sentence and calculates how relevant each one is. It discovers that "animal" is highly relevant and "street" is not, so it bakes the meaning of "animal" into its understanding of "it."

The Library Search: Q, K, V

The mechanism behind self-attention is elegant, and it revolves around three concepts with intimidating names but a simple intuition. Think of it like searching a library:

Let's make this concrete with actual numbers. Here's a simplified example of how self-attention works for a 3-word sentence. (Real models use 768 dimensions — we'll use 4 to keep it readable.)

The beautiful thing about this mechanism is that nothing is sequential. Every word performs its library search at the same time. The computation can be massively parallelized on GPUs, which is why Transformers train orders of magnitude faster than RNNs.

Multi-Head Attention: Multiple Perspectives at Once

There's one more ingredient that makes the Transformer truly powerful. Consider the sentence: "The animal didn't cross the street because it was too tired."

When processing the word "it," there are multiple things worth paying attention to simultaneously:

• "animal" — because that's what "it" refers to
• "tired" — because that's the state being described
• "didn't cross" — because that's the action context

A single attention operation might focus on one of these, but miss the others. The solution? Run multiple attention operations in parallel — each with its own set of Q, K, V weight matrices. Each "head" learns to focus on different types of relationships:

After all 8 heads produce their results, the outputs are concatenated (joined end-to-end) and then projected through a final learned matrix. Here's exactly how that works:

This multi-head attention is what gives the Transformer its remarkable ability to capture rich, nuanced relationships between words.

How a Prompt Becomes a Response

We've seen the individual pieces — embeddings, self-attention, multi-head attention, layer stacks. But how does it all fit together? What actually happens when you type "The cat sat on the" and the model predicts the next word?

Here's the complete journey, from raw text to prediction:

This is why autoregressive generation (predicting one token at a time, then feeding it back in) is both powerful and expensive. The model doesn't "see" the whole answer at once — it builds it word by word, and each word requires a complete pass through billions of parameters. When you see ChatGPT streaming text, you're watching this pipeline execute in real time.

The Real Revolution: Transfer Learning

The Transformer architecture was important. But the idea it unlocked was transformative — and I'm not just making a pun.

Before 2018, if you wanted to build a sentiment analysis model, you'd train a model from scratch on your sentiment data. Want a spam detector? Train from scratch again. Medical text classifier? Start over. Each task required its own expensive training process, its own large labeled dataset, its own compute budget.

Then something clicked — an insight borrowed from computer vision, where it had been working for years:

What if you could train a model to deeply understand language first, and then quickly adapt it to any specific task?

This is transfer learning, and it works exactly like medical specialization:

This was the breakthrough moment. 2018 became NLP's "ImageNet moment" — the year transfer learning finally came to language. And it arrived through two competing approaches.

Two Philosophies: BERT vs. GPT

When researchers applied the Transformer to transfer learning, they faced a design choice: which half of the Transformer should we use? Two camps emerged, each with a different philosophy:

BERT (by Google, 2018) uses the encoder side of the Transformer. It reads text bidirectionally — every word can see every other word in both directions. This makes it excellent at understanding text. BERT is pre-trained with two clever tricks: first, randomly mask 15% of words in a sentence and train the model to predict the missing words (Masked Language Modeling — think of it as a fill-in-the-blank exam). Second, given two sentences, predict whether the second one actually follows the first (Next Sentence Prediction — teaching the model to understand relationships between sentences). A special [CLS] token is prepended to every input, and its output vector becomes the sentence-level representation used for downstream tasks like classification.

GPT (by OpenAI, 2018) uses the decoder side. It reads text left-to-right only — each word can only see what came before it. This makes it a natural language generator. GPT is pre-trained by simply predicting the next word, over and over, on billions of words of text.

The irony? Both approaches are trained on unlabeled data — just raw text from the internet, books, and Wikipedia. No human had to manually label millions of examples. The models learn the structure of language itself, and that knowledge transfers to virtually any downstream task.

from transformers import DistilBertModel, DistilBertTokenizer
from sklearn.linear_model import LogisticRegression

# 1. Load pre-trained model (the "med school graduate")
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased')

# 2. Convert sentences to embeddings (768-dim vectors)
inputs = tokenizer("A visually stunning film", return_tensors="pt")
outputs = model(**inputs)
sentence_embedding = outputs.last_hidden_state[:, 0, :]  # [CLS] token

# 3. Train a simple classifier (the "specialization")
clf = LogisticRegression()
clf.fit(all_embeddings, labels)  # ~81% accuracy with zero fine-tuning!

We've now seen how the Transformer architecture works and how transfer learning made powerful NLP accessible to everyone. But there's a burning question: if these models learn from raw text, what happens when you make them bigger — and feed them more text? The answer launched the era of Large Language Models.

What Happens When You Scale Up?

By 2019, researchers had a powerful architecture (the Transformer) and a powerful idea (transfer learning). The natural question was: what happens if we make these models much, much bigger?

The answer turned out to be one of the most surprising findings in AI research. Bigger models didn't just get incrementally better — they unlocked entirely new capabilities that smaller models simply didn't have. This phenomenon is called emergent capabilities:

But "bigger" isn't just about cramming more parameters into the model. That's where the scaling laws come in.

The Chinchilla Insight: Data Matters as Much as Size

In the early days of LLMs, the strategy was simple: more parameters = better performance. GPT-3 had 175 billion parameters and was trained on 300 billion tokens. The assumption was that if you wanted a better model, you should make it even bigger.

Then in 2022, DeepMind dropped a bombshell called the Chinchilla paper. Their finding was elegantly simple:

To put this in concrete terms: GPT-3 had 175 billion parameters but was trained on only 300 billion tokens. According to the Chinchilla rule, it should have been trained on 3.5 trillion tokens — more than 10x what it actually saw. The model was essentially a student who enrolled in a PhD program but only attended the first month of classes.

DeepMind proved this by training Chinchilla — a model with only 70 billion parameters (less than half of GPT-3) but trained on 1.4 trillion tokens (4.7x more data). The result? Chinchilla outperformed GPT-3 on virtually every benchmark, despite being much smaller.

This finding reshaped the entire industry. The trend since Chinchilla is unmistakable: newer models like Llama 2 (70B parameters, 2 trillion tokens) prioritize data quantity and quality over raw parameter count. But quantity alone isn't enough — data quality matters enormously. Most training data comes from web crawls (Common Crawl, C4), supplemented by books, academic papers, and code repositories. The difference between a mediocre model and a great one often comes down to aggressive data filtering: removing duplicates, near-duplicates, low-quality pages, and toxic content. Meta's Llama papers specifically credit data curation as a key factor in their models' performance. The lesson is clear — a well-fed smaller model beats a starving giant, but only if the food is nutritious.

From Pre-trained to Actually Useful

Here's an uncomfortable truth: a raw pre-trained model is not very useful. It's like a student who read every book in the library but never learned how to have a conversation.

Ask a raw pre-trained model "How do I make pizza?" and compare the responses:

The raw model is completing text, not answering your question. It predicts what words are most likely to come next in its training data — and in its training data, questions are often followed by more questions, not answers.

Making a pre-trained model actually useful requires post-training — and this happens in two stages:

Open vs. Closed: The Great Divide

The LLM landscape is split into two camps, and the choice between them is one of the most consequential decisions in any AI project. But first, a clarification: "open source" in the LLM world doesn't mean what it means in traditional software. A fully open LLM would release three things: model weights (the trained parameters), model code (training scripts, hyperparameters), and training data (the actual corpus). In practice, most "open" models — including Meta's Llama — release weights and code but not their training data, which limits reproducibility and makes it harder to check for data contamination in benchmarks.

In practice, this isn't an either/or decision. Many organizations use a hybrid approach: prototype with a closed-source API (fast to start, great performance), then migrate to an open-source model for production (lower cost at scale, full control over data). The key criteria are: performance requirements, data sensitivity, in-house ML talent, cost structure, and licensing needs.

We now understand how LLMs are built: pre-trained on massive data, then aligned through post-training. But there's one more crucial piece — how do you actually talk to them? It turns out that the way you frame your question changes the answer dramatically.

The Creativity Dial: Temperature

When an LLM generates text, it doesn't simply "know" the right answer. At each step, it produces a probability distribution over its entire vocabulary — tens of thousands of possible next words, each with a probability. The word "Paris" might have a 40% chance, "London" a 15% chance, "the" a 3% chance, and so on.

The question is: which word do you pick?

This is where temperature comes in — and it's one of the most powerful controls you have over an LLM's behavior. Think of it as a creativity dial:

Under the hood, temperature works by dividing the model's raw scores (logits) by T before applying softmax. Let's see exactly what this does to the probability distribution. Imagine 5 candidate words with raw logits:

The Prompting Revolution

Temperature controls how the model picks words. But prompting controls what the model thinks about in the first place — and it turns out to be far more powerful than anyone expected.

The simplest form of prompting is zero-shot — just ask. But when that falls short, few-shot prompting changes the game:

The remarkable thing: the model figures out the pattern from the examples in the prompt alone — no weight updates, no retraining. It just "gets it."

But the real breakthrough came from a deceptively simple idea: what if you just ask the model to "show its work"?

Chain-of-Thought: "Show Your Work"

In 2022, researchers discovered that adding four words — "Let's think step by step" — to a prompt could dramatically improve a model's performance on reasoning tasks. On a math benchmark, this simple addition improved accuracy from 17% to 58%.

This technique is called Chain-of-Thought (CoT) prompting. If you've read Daniel Kahneman's Thinking, Fast and Slow, the intuition snaps into place: LLMs default to System 1 thinking — fast, automatic, pattern-matching. CoT forces them into System 2 — slow, deliberate, step-by-step reasoning. When you ask the model to explain its reasoning before giving the answer, each step of reasoning becomes additional context that helps the model arrive at a better final answer. It's the same reason math teachers tell students to "show your work" — the process of writing out steps helps you catch errors and think more clearly.

For complex real-world tasks, these techniques combine into prompt chains — breaking a task into sequential steps, where each prompt's output feeds into the next. Need a marketing campaign? Step 1: generate product names. Step 2: create slogans for the winning name. Step 3: write the sales pitch. Each step is a focused, manageable prompt. This is the design pattern behind most production LLM applications.

But there's a catch with all these prompting techniques: the model returns free text. In production systems, you usually need structured output — valid JSON, specific fields, predictable formats that your code can parse. This is where structured outputs come in: you define a schema (like a JSON template), and the model is constrained to only generate text that matches that schema. Think of it as giving the model a form to fill out instead of a blank page. OpenAI's JSON Mode, function calling, and open-source tools like Outlines and Guidance all solve this problem. Without structured outputs, you'd spend half your engineering time writing fragile parsing code to extract data from free text — and it would still break regularly.

One more practical insight: as models have evolved, their context windows have grown dramatically — from 1K tokens to over 2M tokens in just five years. But longer context doesn't mean perfect recall. Research shows models understand information at the beginning and end of the context much better than the middle (the "lost in the middle" problem). When constructing prompts, put the most important information at the start or end — not buried in the middle.

The Elephant in the Room: Prompt Security

There's a darker side to the power of prompting. If the right prompt can make a model helpful and accurate, the wrong prompt can make it dangerous. This is the prompt security problem, and it's one of the biggest unsolved challenges in AI.

Three main attack categories threaten every LLM application:

We've now journeyed from individual words as numbers to the full modern LLM stack — architecture, training, alignment, and the art of communicating with these models. It's time to put it all together.