From Prompt to GPU

The Phrasebook Problem

Imagine you're traveling to Japan with a phrasebook. You could look up every single letter individually — "h," "e," "l," "l," "o" — and spell everything character by character. That would work, but it would take forever. A better phrasebook groups common sequences: "hello" is one entry, "thank you" is another. Common phrases get a single lookup; rare words get spelled out letter by letter.

That's exactly how a tokenizer works. It's the very first thing that processes your prompt — before any neural network, any attention mechanism, any GPU computation. The tokenizer converts your text into a sequence of numbers (called tokens) that the model can process. And the quality of this conversion has a direct impact on cost, speed, and even the model's ability to understand your input.

A token is roughly three-quarters of a word in English, or about 4 characters. "Hello, how are you?" becomes about 5 tokens. But this ratio changes dramatically across languages, and that has real financial consequences.

BPE: How the Phrasebook Is Built

Byte Pair Encoding (BPE) is the algorithm that builds this phrasebook. Think of it like a language teacher who watches millions of sentences and notices which character pairs appear together most frequently. The teacher then creates a shorthand for the most common pairs, then the most common pairs of those pairs, and so on — building up from individual characters to common words and even multi-word fragments.

The process starts with every character as its own token. Then it repeatedly merges the most frequent adjacent pair into a single new token. After thousands of merge operations, you get a vocabulary of 32,000 to 128,000 tokens that efficiently encodes common text.

SentencePiece is the tool that actually builds these phrasebooks. Developed by Google, it's the standard tokenizer builder used by Llama, Mistral, and most modern open-source models. SentencePiece takes a corpus of text and runs BPE (or a variant called Unigram) to produce the final vocabulary file that the model uses forever after.

The Arabic Tokenizer Tax

Here's where tokenization has real business impact. The fertility score measures how many tokens a language needs per word. English has a fertility of about 1.0 — one word, one token (roughly). Arabic has a fertility of 2.19. That means every Arabic word uses more than twice as many tokens as an English word.

Why? Because BPE vocabularies are trained primarily on English text. Common English words like "the," "is," and "database" each get their own token. But Arabic words — even extremely common ones — often get split into 3, 4, or even 5 subword pieces because the tokenizer never learned to merge those character sequences.

The financial impact is direct: if every API call costs per token, Arabic users are paying 2-5x more for the same conversation. A chatbot that costs $500/month in English costs $1,000-$2,500/month in Arabic — for the same number of users, the same conversations, the same functionality.

The Desk Size Problem

Imagine your GPU is a desk. The model's parameters — all 70 billion of them — are textbooks that need to be open on that desk. The bigger the model, the more textbooks. The desk has a fixed size, and if the books don't fit, you either need a bigger desk or you need to squeeze the books.

That desk is VRAM — Video RAM, the memory physically located on the GPU chip. Unlike regular system RAM (which sits on your motherboard and communicates with the CPU over a relatively slow bus), VRAM sits right next to the GPU's compute cores with a massively wide memory bus. An NVIDIA A100 GPU has 80GB of VRAM. An H100 has 80GB. A consumer RTX 4090 has 24GB.

FP16: Squeezing the Books

Every parameter in a model is a number — a weight that was learned during training. In full precision (FP32), each number takes 4 bytes of storage. A 70-billion parameter model in FP32 needs 70B x 4 bytes = 280GB. That doesn't fit on any single GPU.

The solution: FP16 (half precision). Instead of using 32 bits per number, you use 16 bits — cutting storage exactly in half. A 70B model in FP16 needs 70B x 2 bytes = 140GB. Still doesn't fit on one 80GB GPU, but it fits on two.

The remarkable thing is that half precision barely affects output quality. The difference between 3.14159265 and 3.14160 matters in scientific computing but is irrelevant for language model weights. Most modern inference runs in FP16 or BF16 (brain float 16, a variant that handles extreme values better) by default.

TFLOPS vs Memory Bandwidth: The Real Bottleneck

GPUs have two speed metrics, and understanding which one matters is the key to understanding inference performance.

TFLOPS (Tera Floating-point Operations Per Second) measures how fast the GPU can do math. The A100 does 312 TFLOPS in FP16. That's 312 trillion multiply-add operations every second. It sounds like the GPU should be blindingly fast.

Memory bandwidth measures how fast the GPU can move data from VRAM to the compute cores. The A100 has 2 TB/s of memory bandwidth. That's 2 terabytes per second — fast, but not fast enough.

Think of it as a factory. TFLOPS is how fast the workers can assemble products. Memory bandwidth is how fast the conveyor belt delivers raw materials. Even if you have 1,000 workers, if the conveyor belt can only deliver enough materials for 100 workers at a time, 900 workers sit idle. That's exactly what happens during most of LLM inference: the GPU compute cores are waiting for data to arrive from memory.

Why the First Token Takes So Long

You've probably noticed this when using ChatGPT or Claude: you hit send, there's a noticeable pause, and then tokens start appearing rapidly one after another. That initial pause and the subsequent fast streaming aren't just an interface choice — they reflect two fundamentally different computational phases happening on the GPU.

Phase 1: Prefill — Reading Your Prompt

Imagine a speed reader who can read an entire book in parallel — all pages at once. That's the prefill phase. The GPU takes your entire prompt (all tokens at once) and processes them simultaneously through the model. This is matrix-matrix multiplication: large blocks of data being multiplied together.

During prefill, the GPU's compute cores are actually busy. The operation has high arithmetic intensity — the ratio of computation to memory access. For every byte loaded from memory, the GPU performs many operations. This is compute-bound: the speed limit is how fast the GPU can do math, not how fast it can fetch data.

Phase 2: Decode — Generating One Token at a Time

Now imagine the same speed reader has to write a novel, but they can only write one word at a time, and before writing each word, they need to re-read everything they've written so far. That's the decode phase. The GPU generates tokens one by one, and each token requires accessing the entire model's weights.

During decode, the operation is matrix-vector multiplication: one tiny vector (the current token) multiplied against the enormous weight matrices. The arithmetic intensity drops dramatically — you load a huge amount of data for a tiny amount of computation. This is memory-bound: the speed limit is how fast data can move from VRAM to the compute cores.

Arithmetic intensity is the ratio that determines which phase you're in. High arithmetic intensity (many operations per byte of data moved) means compute-bound. Low arithmetic intensity (few operations per byte) means memory-bound. Prefill has high intensity because you're multiplying large matrices together. Decode has low intensity because you're multiplying one tiny vector against large matrices.

The Re-Reading Problem

Imagine you're writing an essay, and every time you write a new sentence, you need to re-read the entire essay from the beginning to remember what you've said. Sentence 1 takes 1 second. Sentence 2 takes 2 seconds (re-read sentence 1, then write). Sentence 100 takes 100 seconds. The time to write each new sentence grows linearly with the length of what you've already written.

This is exactly what would happen during the decode phase without a cache. For each new token, the model needs to "attend to" (look at) every previous token. The attention mechanism computes Key and Value vectors for each token — these are the compressed representations that allow the model to "remember" what each token means in context.

The KV cache stores these Key and Value vectors so the model doesn't have to recompute them for every new token. When generating token 100, instead of recomputing Keys and Values for tokens 1-99, the model just looks up the cached values and only computes the Key/Value for the new token.

PagedAttention: The Filing Cabinet Fix

Without PagedAttention, the KV cache works like a bookshelf with fixed-size shelves. You allocate a contiguous block of VRAM for each request based on the maximum possible sequence length. A request that might use 4K tokens gets a 4K-token allocation, even if it only ends up using 200 tokens. The wasted space fragments memory and limits how many requests can run concurrently.

PagedAttention (introduced by vLLM) works like a filing cabinet with movable folders. Instead of allocating one big continuous block per request, it breaks the KV cache into small, fixed-size pages (typically 16 tokens each). Pages are allocated on demand as the sequence grows, and freed pages can be reused by other requests immediately.

Think of the difference between renting an entire floor of an office building (even if you only use 3 rooms) versus renting individual rooms as you need them. PagedAttention reduces VRAM waste from 60-80% down to less than 4%, which means you can serve 2-4x more concurrent requests with the same GPU.

The Empty Factory Problem

Imagine a factory with 1,000 workers. One customer walks in with a single widget to assemble. All 1,000 workers crowd around that one widget. 999 of them stand idle while one does the work. That's what happens when a GPU with thousands of compute cores processes a single request during the decode phase — the memory bandwidth bottleneck means most cores have nothing to do.

The fix? Batching — processing multiple requests at the same time. Instead of one customer's widget, you have 32 customers' widgets on the assembly line simultaneously. Now more workers have something to do, and the factory is actually efficient.

Continuous Batching: No Waiting in Line

Static batching is like a bus: it waits until all seats are full, drives the route, and nobody can get on or off until the trip is complete. If one request finishes early (a short response), its "seat" sits empty until the longest request in the batch completes.

Continuous batching (also called "inflight batching") is like a conveyor belt at a buffet. Requests enter and exit freely. When one request finishes, a new one immediately takes its place. There's no wasted time waiting for the entire batch to complete.

This seemingly simple change has an enormous impact: continuous batching improves GPU utilization by 10-20x compared to static batching in real workloads with mixed-length requests.

The Batch Size Trap

"If batching helps, why not make the batch huge?" Because of a trap: throughput scales sub-linearly with batch size. Doubling the batch from 8 to 16 might increase throughput by 70%. Doubling from 32 to 64 might only increase it by 20%. And at some point, you run out of VRAM for the KV caches of all those concurrent requests.

The metric that captures this is MBU (Model Bandwidth Utilization) — what percentage of the GPU's memory bandwidth is actually being used for useful computation. A well-optimized serving system targets 80-90% MBU. Below 50% means you're leaving money on the table. Above 95% means you're probably trading latency for throughput.

Stretching the Window

A model trained with a 4K context window can't magically handle 32K tokens. The model's position encoding — how it understands "this token is in position 500" vs "this token is in position 5000" — breaks when you exceed the training length. The model has never seen positions beyond 4K during training, so those positions produce garbage.

But what if you need longer context? Retraining the entire model on longer sequences is expensive (millions of dollars). Researchers found shortcuts.

RoPE: How Models Know Position

Think of a clock. The hour hand rotates at one speed, the minute hand at another. By combining these two rotations, you can encode any time of day. RoPE (Rotary Position Embedding) works similarly — it encodes each token's position by rotating its vector representation at specific frequencies. Position 1 gets a small rotation. Position 1000 gets a large rotation. The model learns to interpret these rotations during training.

The problem: the model was trained with rotations up to, say, position 4096. Position 5000 requires a rotation the model has never seen. It's like a clock trying to display 25 o'clock — the numbers don't wrap around properly.

NTK-Aware Scaling: Changing the Base Frequency

NTK-Aware Scaling modifies the base frequency of RoPE's rotational encoding, effectively "stretching" the position space. Instead of positions 0-4096, the same rotation angles now span 0-32768. It's like switching from a 12-hour clock to a 24-hour clock — you cover more positions without changing the model's ability to distinguish nearby positions.

The beauty is that NTK-Aware Scaling requires zero additional training. You modify a single hyperparameter (the RoPE base frequency) at inference time. The tradeoff is that the model's ability to distinguish very close positions decreases slightly — position 100 and position 101 become slightly harder to tell apart — but for most applications, this is negligible.

YaRN: The Full Solution

YaRN (Yet another RoPE extensioN) takes NTK scaling further. Instead of uniformly stretching all frequencies, YaRN applies different scaling factors to different frequency bands. High-frequency components (which encode local relationships like "these two words are adjacent") get less stretching. Low-frequency components (which encode global relationships like "this paragraph is about the same topic") get more stretching.

This preserves local precision while extending global range. YaRN can extend context from 4K to 128K with only a small amount of fine-tuning data (a few hundred examples), compared to the full retraining that would otherwise be needed.

The Creativity Dial

After the model processes your prompt through all its layers, the final step isn't just "pick the best word." The model outputs a probability distribution over its entire vocabulary — a score for every possible next token. "The" might have a 15% probability, "A" might have 8%, "In" might have 5%, and so on for all 128,000 tokens. How you sample from this distribution determines the character of the output.

Temperature: How Much Randomness

Think of a radio dial. On the left (temperature = 0), you get crystal-clear reception of a single station — the model always picks the highest-probability token. The output is deterministic and repeatable. On the right (temperature = 1.0+), you get static mixed with signal — lower-probability tokens get a real chance of being selected, creating more diverse, creative, and sometimes surprising output.

Temperature = 0: "The cat sat on the mat." Every time, identical output. Perfect for factual tasks, code generation, data extraction.

Temperature = 0.7: "The cat settled on the mat." Slight variation, still coherent. Good for conversational AI, writing assistance.

Temperature = 1.2: "The feline lounged across the weathered doormat." More creative, sometimes surprising. Good for brainstorming, creative writing.

Temperature = 2.0: "The quantum velvet whispered pancake orbits." Chaotic. Rarely useful except for generating random inspiration seeds.

Logprobs: The Confidence Score

Logprobs (log-probabilities) reveal the model's confidence in each token it generates. A logprob of 0 means 100% confident. A logprob of -1 means ~37% confident. A logprob of -5 means ~0.7% confident.

This is incredibly useful for detecting hallucination. When the model confidently states a fact, the logprob for that token should be high (close to 0). When the logprob drops below -2.0, the model is essentially guessing. In production systems, you can flag low-confidence claims for human review.

Stop Sequences: Knowing When to Stop

Stop sequences tell the model when to stop generating. Without them, the model might keep generating until it hits the maximum token limit — producing rambling, repetitive, or off-topic content. Common stop sequences include newlines (for single-line answers), closing brackets (for JSON), or specific delimiter strings.

For structured output (JSON, XML, SQL), stop sequences are essential. You tell the model to stop after the closing } of a JSON object, preventing it from generating garbage after the valid output.

The Build vs Buy Question

Every team deploying LLMs faces this decision: use a hosted API (OpenAI, Anthropic, Google) or run your own model on your own GPUs. The answer depends on your volume, your latency requirements, your data privacy needs, and — critically — your team's operational capacity.

The Serving Frameworks

If you self-host, you need a serving framework. Here are the four that matter:

The Cost Comparison

The math changes dramatically based on your scale:

Metric	API (GPT-4o)	Self-Host (vLLM + Llama-70B)
Setup time	Minutes	Days to weeks
Cost at 10K requests/day	~$300/month	~$3,000/month (GPU lease)
Cost at 1M requests/day	~$30,000/month	~$8,000/month (GPU cluster)
Latency (P50)	500-1500ms TTFT	100-300ms TTFT
Data privacy	Data leaves your network	Data stays in your infrastructure
Ops burden	Zero (managed)	High (GPU monitoring, scaling, updates)
Model flexibility	Vendor's models only	Any open-source model, fine-tuned