Memory for LLMs:
From Stateless to Production-Ready

The Uncomfortable Truth: LLMs Have Amnesia

Every time you call an LLM API, you're talking to a stranger. The model doesn't know you messaged it 5 seconds ago. It doesn't know you messaged it yesterday. Each API call exists in complete isolation.

When you use ChatGPT or Claude, it seems like they remember. But that's not the LLM—that's an entire memory system built on top. The LLM is the engine; memory is a separate module that most people never see.

The Obvious Fix: Just Send Everything

The simplest solution? Send the entire conversation every time. The LLM doesn't remember, so you remind it — over and over.

from openai import OpenAI
client = OpenAI()

# This is "memory" — a list you manage yourself
messages = [
    {"role": "system", "content": "You are a coding assistant."},
]

def chat(user_message):
    messages.append({"role": "user", "content": user_message})

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages  # Send EVERYTHING every time
    )

    assistant_msg = response.choices[0].message.content
    messages.append({"role": "assistant", "content": assistant_msg})
    return assistant_msg

# Each call sends the full history:
chat("I'm building a REST API in Python")  # 3 messages sent
chat("Let's use FastAPI")                   # 5 messages sent
chat("Add authentication to it")             # 7 messages sent
chat("Now add rate limiting")                # 9 messages sent... growing!

This works beautifully for short conversations. The LLM sees the full context, so its responses are coherent and relevant. But it can't last.

This works beautifully... until it doesn't.

The Wall: Context Windows Have Limits

LLMs can only process so much text at once. This limit is called the context window—and when you hit it, everything breaks.

But bigger context windows aren't free. More tokens means:

• Higher costs — you pay per token, input and output
• Slower responses — more to process, more latency
• Worse focus — models struggle to attend to everything equally

Strategy 1: Trimming (Drop the Old Stuff)

The bluntest solution: when the conversation gets too long, cut the oldest messages. First in, first out.

Trimming is fast and cheap. But critical information often lives in those early messages — user preferences, project context, key decisions. Once it's gone, it's gone.

import tiktoken

def trim_messages(messages, max_tokens=4000, model="gpt-4o"):
    """Keep the system prompt + most recent messages that fit."""
    enc = tiktoken.encoding_for_model(model)

    # Always keep the system prompt
    system_msg = messages[0]
    system_tokens = len(enc.encode(system_msg["content"]))

    # Add messages from newest to oldest until we hit the limit
    trimmed = []
    token_count = system_tokens

    for msg in reversed(messages[1:]):
        msg_tokens = len(enc.encode(msg["content"]))
        if token_count + msg_tokens > max_tokens:
            break
        trimmed.insert(0, msg)
        token_count += msg_tokens

    return [system_msg] + trimmed

# Usage: trim before every API call
trimmed = trim_messages(messages, max_tokens=4000)
response = client.chat.completions.create(
    model="gpt-4o", messages=trimmed
)

The Moment Trimming Betrays You

Trimming works until it silently deletes the one message that mattered. Here is a real conversation where trimming creates a dangerous failure. Watch what happens when message 3 gets dropped:

# Message 1 (user):
"I'm building a medical triage chatbot for an ER."

# Message 2 (assistant):
"I can help. What symptoms should it handle?"

# Message 3 (user):  ← THIS IS CRITICAL
"Important constraint: NEVER suggest the patient is fine
 to go home. Always recommend they wait for a doctor."

# Message 4 (assistant):
"Understood. I'll always err on the side of caution."

# Messages 5-20: Building out triage logic, testing scenarios...

# Message 21 (user):
"Patient reports mild headache, no other symptoms. What should
 the bot say?"

This is why naive trimming is dangerous for any use case where early messages contain constraints, preferences, or safety rules. You have two options: pin critical messages so they never get trimmed, or graduate to summarization.

import tiktoken

# Different models use different tokenizers
enc_gpt4 = tiktoken.encoding_for_model("gpt-4o")
enc_gpt35 = tiktoken.encoding_for_model("gpt-3.5-turbo")

text = "Refactor the authentication module to use JWT tokens"

# Same text, different token counts
gpt4_tokens = len(enc_gpt4.encode(text))    # 9 tokens
gpt35_tokens = len(enc_gpt35.encode(text))  # 10 tokens

# For a full messages array (OpenAI format), count overhead too
def count_messages_tokens(messages, model="gpt-4o"):
    """Count tokens including per-message overhead."""
    enc = tiktoken.encoding_for_model(model)
    tokens = 0
    for msg in messages:
        tokens += 4  # every message has overhead: role, content markers
        for key, value in msg.items():
            tokens += len(enc.encode(value))
    tokens += 2  # priming overhead for assistant reply
    return tokens

# A 20-message conversation: ~3,200 tokens
# A 50-message conversation: ~15,000 tokens
# A 100-message coding session: ~45,000 tokens

Summarization: Compress, Don't Delete

Instead of throwing away old messages, compress them. Use an LLM to create a summary of what happened.

def summarize_and_compress(messages, max_tokens=4000):
    """Summarize old messages, keep recent ones verbatim."""
    enc = tiktoken.encoding_for_model("gpt-4o")
    total = sum(len(enc.encode(m["content"])) for m in messages)

    if total <= max_tokens:
        return messages  # No need to summarize yet

    # Keep system prompt + last 6 messages verbatim
    system = messages[0]
    recent = messages[-6:]
    old = messages[1:-6]

    # Summarize the old messages
    summary = client.chat.completions.create(
        model="gpt-4o-mini",  # Cheaper model for summarization
        messages=[{
            "role": "system",
            "content": """Summarize this conversation. Preserve:
- User's name, preferences, and stated requirements
- Key decisions made and their reasoning
- Technical context (language, framework, architecture)
- Any constraints or deadlines mentioned
Be concise but don't lose critical facts."""
        }] + old
    ).choices[0].message.content

    # Return: system + summary + recent messages
    return [
        system,
        {"role": "system", "content": f"Previous conversation summary: {summary}"},
        *recent
    ]

"""You are a conversation summarizer for a coding assistant.
Produce a structured summary preserving EXACTLY these fields:

1. USER IDENTITY: Name, role, timezone (if mentioned)
2. PROJECT CONTEXT: Language, framework, architecture decisions
3. CONSTRAINTS: Any "must", "never", "always" rules the user stated
4. DECISIONS MADE: What was decided, with reasoning
5. CURRENT STATE: What has been built so far
6. OPEN QUESTIONS: What is unresolved

Format as a flat list. Be specific. Use the user's exact words
for constraints. Do not editorialize or add your own opinions.
Maximum 400 tokens."""

USER IDENTITY: Backend developer, building for a startup
PROJECT CONTEXT: Python 3.12, FastAPI, PostgreSQL, deployed on Railway
CONSTRAINTS:
  - All endpoints MUST return JSON:API format
  - No ORM - raw SQL with asyncpg
  - Auth tokens expire after 15 minutes
DECISIONS MADE:
  - PostgreSQL over MongoDB (need relational joins for reporting)
  - JWT with refresh tokens (not session-based)
  - CORS allows only app.example.com
CURRENT STATE:
  - /auth/login and /auth/refresh endpoints working
  - User model with bcrypt password hashing
  - CORS middleware configured
OPEN QUESTIONS:
  - Rate limiting strategy not yet decided
  - File upload endpoint design pending

External Memory: Store It Somewhere Else

Here is the key insight: memory does not have to live in the prompt. You can store information externally and pull it in only when relevant.

This is the shift from "remember everything" to "remember what matters right now."

A Framework for Thinking About Memory

Memory in LLMs maps to 5 categories. The first two (parametric and procedural) are baked into the model — you can't change them at runtime. The three that matter for developers are: working memory (context window), episodic memory (conversation history), and semantic memory (knowledge bases/RAG).

RAG: The Industry Standard

Retrieval-Augmented Generation is how most production systems handle long-term memory. Instead of stuffing everything in the prompt, you store documents externally and retrieve only what's relevant.

If context window management is "remembering what happened in this conversation," RAG is "knowing things that were never part of this conversation." Your company's documentation, your codebase, your customer's order history — these all live outside the conversation and need to be retrieved on demand.

from openai import OpenAI
import numpy as np

client = OpenAI()

# Step 1: INGEST - Chunk your documents and embed them
def embed_text(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def chunk_document(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

# Step 2: RETRIEVE - Find the most relevant chunks
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, knowledge_base, top_k=3):
    query_embedding = embed_text(query)
    scored = [
        (chunk, cosine_similarity(query_embedding, emb))
        for chunk, emb in knowledge_base
    ]
    scored.sort(key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in scored[:top_k]]

# Step 3: GENERATE - Ask the LLM with retrieved context
def rag_query(question, knowledge_base):
    relevant_chunks = retrieve(question, knowledge_base)
    context = "\n\n".join(relevant_chunks)

    return client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"""Answer based on this context:
{context}

If the context doesn't contain the answer, say so."""},
            {"role": "user", "content": question}
        ]
    ).choices[0].message.content

Step 1: Chunking — How You Split Documents Changes Everything

Before you can search your documents, you need to split them into chunks small enough for embedding. This sounds trivial — just split every 500 words, right? But chunking strategy is the single biggest lever you have for RAG quality. Bad chunking produces bad retrieval, and no amount of fancy reranking fixes it.

def fixed_chunk(text, chunk_size=500, overlap=50):
    """Split text into fixed-size chunks with overlap."""
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start = end - overlap  # overlap prevents cutting mid-thought
    return chunks

# A 10,000-word doc with 500-word chunks + 50 overlap = ~22 chunks
# Pro: Fast, predictable chunk sizes
# Con: Splits mid-sentence, mid-paragraph, mid-thought

import re

def semantic_chunk(text, max_tokens=800):
    """Split at paragraph boundaries, merge small paragraphs."""
    paragraphs = re.split(r'\n\n+', text)
    chunks = []
    current = []
    current_size = 0

    for para in paragraphs:
        para_tokens = len(para.split()) * 1.3  # rough token estimate
        if current_size + para_tokens > max_tokens and current:
            chunks.append("\n\n".join(current))
            current = [para]
            current_size = para_tokens
        else:
            current.append(para)
            current_size += para_tokens

    if current:
        chunks.append("\n\n".join(current))
    return chunks

# Pro: Each chunk is a complete thought
# Con: Chunk sizes vary (100-800 tokens), harder to predict costs

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,         # characters, not tokens
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""]  # try these in order
)

chunks = splitter.split_text(document_text)
# Result: chunks that are ~1000 chars each, split at natural boundaries
# This is the best default for most use cases

Step 2: Embedding — Turning Text into Searchable Vectors

Once you have chunks, each one gets converted into a vector — a list of numbers that captures its meaning. Two pieces of text about the same topic will have vectors that are close together in this space. Two unrelated texts will be far apart. This is how "search by meaning" works instead of "search by keyword."

Step 3: Vector Storage — Where Your Embeddings Live

Once you have vectors, you need somewhere to store and search them. This is where vector databases come in. The choice matters less than you think for getting started — but matters a lot at scale.

Step 4: Reranking — The Quality Multiplier

Here is the thing most teams miss: vector similarity is not the same as relevance. Two texts can be "semantically similar" without one being a useful answer to the other. Reranking is the fix — and it is the single biggest quality improvement you can make after basic RAG is working.

import cohere

co = cohere.Client("your-api-key")

def retrieve_and_rerank(query, knowledge_base, initial_k=20, final_k=3):
    # Step 1: Fast vector search for initial candidates
    candidates = vector_search(query, knowledge_base, top_k=initial_k)

    # Step 2: Rerank with cross-encoder for true relevance
    results = co.rerank(
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=final_k,
        model="rerank-english-v3.0"
    )

    # Return only the truly relevant chunks
    return [candidates[r.index] for r in results.results]

# Typical improvement: recall@3 goes from 65% to 88%
# Cost: ~$0.001 per rerank call (20 documents)
# Latency: adds ~150-250ms

Measuring Retrieval Quality

You cannot improve what you do not measure. Most teams build RAG, test it manually with 5 queries, say "looks good," and ship. Then users report wrong answers and nobody knows if it is a retrieval problem, a chunking problem, or an LLM problem.

def build_context(user_query, conversation, knowledge_base):
    # Layer 1: System prompt (procedural memory)
    system = {"role": "system", "content": SYSTEM_PROMPT}

    # Layer 2: Summary of old conversation (compressed short-term)
    summary = summarize_if_needed(conversation[:-10])

    # Layer 3: RAG context (long-term/semantic memory)
    relevant_docs = retrieve(user_query, knowledge_base, top_k=3)
    rag_context = "\n".join(relevant_docs)

    # Layer 4: Recent conversation (verbatim short-term)
    recent = conversation[-10:]

    return [
        system,
        {"role": "system", "content": f"Session summary: {summary}"},
        {"role": "system", "content": f"Relevant docs:\n{rag_context}"},
        *recent
    ]

Testing Memory Systems

Memory bugs are invisible at demo time. They emerge after 50+ messages, after documents change, or after users push edge cases you never imagined. Here is how to catch them before production does.

import json

def test_retrieval_quality(retriever, test_cases):
    """
    test_cases = [
        {"query": "How do I reset my password?",
         "expected_doc_id": "auth-guide-v3",
         "expected_in_top_k": 3},
        ...
    ]
    """
    hits = 0
    reciprocal_ranks = []

    for case in test_cases:
        results = retriever.search(case["query"], top_k=5)
        result_ids = [r.doc_id for r in results]

        if case["expected_doc_id"] in result_ids[:case["expected_in_top_k"]]:
            hits += 1
            rank = result_ids.index(case["expected_doc_id"]) + 1
            reciprocal_ranks.append(1.0 / rank)
        else:
            reciprocal_ranks.append(0.0)

    recall = hits / len(test_cases)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)

    assert recall >= 0.85, f"Recall dropped to {recall:.2f} (threshold: 0.85)"
    assert mrr >= 0.70, f"MRR dropped to {mrr:.2f} (threshold: 0.70)"

    return {"recall": recall, "mrr": mrr, "total": len(test_cases)}

def test_context_window_overflow():
    """Simulate a 200-message conversation and verify
    the system handles it gracefully (summarize/trim)."""
    messages = []
    for i in range(200):
        messages.append({"role": "user", "content": f"Message {i}: " + "x" * 500})
        messages.append({"role": "assistant", "content": f"Reply {i}"})

    # This should NOT throw a token limit error
    context = build_context(messages)
    total_tokens = count_tokens(context)
    assert total_tokens <= MODEL_MAX_TOKENS * 0.9

def test_early_info_retained():
    """User mentions their name in message 1.
    After 50 messages, the system should still know it."""
    messages = [
        {"role": "user", "content": "My name is Ahmed and I work at Acme Corp"},
        # ... 50 filler messages ...
        {"role": "user", "content": "What's my name?"}
    ]
    response = get_completion(build_context(messages))
    assert "Ahmed" in response